Point cloud data transmission system

Abstract

An object is to provide a point cloud data transmission system, a controller, a data transmission method, and a program that compress point cloud data while balancing low latency and point cloud quality.SOLUTION: A point cloud data transmission system 301 according to the present invention is a point cloud data transmission system that transmits three dimensional point cloud data 60 via a network 50, and includes a transmitter 10 that has two types of encoding means (11, 12) of coarse encoding for reducing the number of points and fine encoding for performing compression while maintaining quality, and performs two stage encoding of the three dimensional point cloud data 60 with the two types of encoding means (11, 12), and a controller 30 that determines encoding parameters (M, C) of the two types of encoding means (11, 12) based on conditions (N, P, R, D), and causes the transmitter 10 to perform two stage encoding of the three dimensional point cloud data 60 with the encoding parameters (M, C).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to data compression in a point cloud data transmission system that converts real-space data acquired by a depth camera or the like into three-dimensional point cloud data and transmits the data. [Background technology]

[0002] In recent years, use cases have been considered in which real-world space is converted into 3D point cloud data using a depth camera, etc., and then transmitted to share that space in remote locations. A 3D point cloud represents the 3D structure of an object using irregularly sampled 3D points, and each point holds 3D coordinate information (x, y, z) and attribute information (e.g., color information).

[0003] When transmitting 3D point cloud data to a remote location, a very large bandwidth is required for the network. For example, if the amount of data per point is 4 bytes for 3D coordinate information and 1 byte for color information, the amount of data per point cloud frame is as follows: 860K points x 15 bytes x 30fps ≒ 3.09Gbps This becomes:

[0004] Therefore, in order to reduce the load on the network, a bandwidth reduction technique has been disclosed that compresses and transmits 3D point cloud data. (1) Geometry-based Point Cloud Compression (G-PCC) (see Non-Patent Document 1) G-PCC achieves bandwidth reduction through the following two mechanisms: - Approximate the surface of the point cloud using an octree structure and multiple triangles to encode coordinate information. · Predictive coding is performed using attribute information belonging to nearby points. (2) Video-based PCC (V-PCC) (see Non-Patent Document 1) Three types of images (occupancy, depth, color maps) are generated from 3D point clouds, and bandwidth reduction is achieved by using video coding. (3) Point Cloud Library (PCL) (see Non-Patent Document 2) Bandwidth reduction is achieved by aggregating or thinning out points using an octree structure. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] D. Graziosi D et al., “An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)”, APSIPA Transactions on Signal and Information Processing, 2020. [Non-patent document 2] Y. Huang et al., “ISCom: Interest-Aware Semantic Communication Scheme for Point Cloud Video Streaming on Metaverse XR Devices”, in IEEE Journal on Selected Areas in Communications, vol. 42, no. 4, pp. 1003-1021, April 2024. Summary of the Invention [Problem to be solved by the invention]

[0006] G-PCC and V-PCC achieve bandwidth reduction while maintaining point cloud quality, but have the characteristic of large encoding processing delay when the number of points is large. PCL achieves bandwidth reduction while maintaining low latency, but has the characteristic of quality degradation due to point reduction processing.

[0007] Therefore, in a point cloud data transmission system, when point cloud data is compressed to the limited bandwidth available for transmission, a trade-off occurs: increased delay due to the compression process, or reduced quality due to point reduction. In other words, the disclosed bandwidth reduction technology has the problem of difficulty in balancing low latency and point cloud quality within a limited bandwidth.

[0008] Therefore, in order to solve the above problem, the present invention aims to provide a point cloud data transmission system, controller, data transmission method, and program that compress point cloud data by balancing low latency and point cloud quality. [Means for solving the problem]

[0009] In order to achieve the above object, the point cloud data transmission system of the present invention has the transmitter perform a combination of two-stage encoding: coarse encoding, which reduces the number of points that make up the point cloud, and fine encoding, which compresses while maintaining quality, and adjusts the encoding parameters of both based on conditions.

[0010] The point cloud data transmission system according to the present invention is a point cloud data transmission system for transmitting three-dimensional point cloud data via a network, a transmitter having two types of encoding means, namely, coarse encoding for reducing the number of points and fine encoding for compressing while maintaining quality, and performing two-stage encoding of the 3D point cloud data using the two types of encoding means; a controller that determines encoding parameters for each of the two types of encoding means based on a condition and causes the transmitter to perform two-stage encoding of the 3D point cloud data using the encoding parameters; Equipped with.

[0011] Furthermore, a controller according to the present invention has two types of encoding means, namely, Coarse encoding which reduces the number of points and Fine encoding which performs compression while maintaining quality, and controls a transmitter which performs two-stage encoding of 3D point cloud data using the two types of encoding means and transmits the data to a network, a coding parameter determination table that describes the correspondence between the coding parameters and conditions of the coarse coding and the fine coding; a control unit that checks the input conditions against the encoding parameter determination table, extracts the encoding parameters that match the conditions, and causes the transmitter to perform the two-stage encoding using the extracted encoding parameters; Equipped with.

[0012] Furthermore, the data transmission method according to the present invention comprises: Coarse encoding, which reduces the number of points, and Fine encoding, which compresses while maintaining quality, are determined based on conditions. performing two-stage encoding of the three-dimensional point cloud data using the encoding parameters by the two types of encoding means; and Transmitting the two-stage encoded three-dimensional point cloud data to a network. It is characterized by:

[0013] The point cloud data transmission system according to the present invention performs two-stage encoding of three-dimensional point cloud data. At this time, input conditions (point number information N, bandwidth information D, transmission quality requirement R, server processing capacity P, etc.) are checked against an encoding parameter determination table, and encoding parameters (post-reduced point number M, encoding parameter C) for two-stage encoding that match the conditions are extracted and processed by the transmitter. This makes it possible to achieve optimal encoding according to each condition, such as when low latency is important or when point cloud quality is important.

[0014] Therefore, the present invention can provide a point cloud data transmission system, a controller, and a data transmission method that compress point cloud data while balancing low latency and point cloud quality.

[0015] The present invention also provides a program that causes a computer to function as the controller. The controller of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0016] The above inventions can be combined as much as possible. [Effects of the Invention]

[0017] The present invention can provide a point cloud data transmission system, a controller, a data transmission method, and a program that compress point cloud data while balancing low latency and point cloud quality. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram illustrating a configuration of a point cloud data transmission system according to the present invention. [Figure 2] FIG. 4 is a diagram illustrating an encoding parameter determination table provided in the controller according to the present invention. [Figure 3] 1 is a diagram illustrating a configuration of a point cloud data transmission system according to the present invention. [Figure 4] FIG. 4 is a diagram illustrating an encoding parameter determination table provided in the controller according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] The following description of the preferred embodiments of the present invention will be given with reference to the accompanying drawings. The preferred embodiments described below are examples of the present invention, and the present invention is not limited to the preferred embodiments. In this specification and the drawings, components having the same reference numerals are intended to represent the same components.

[0020] (Embodiment 1) FIG. 1 is a diagram illustrating a point cloud data transmission system 301 according to this embodiment. The point cloud data transmission system 301 uses two-stage encoding, which performs coarse encoding to reduce the number of points that make up a point cloud, and fine encoding to perform compression while maintaining quality. In detail, the point cloud data transmission system 301: A point cloud data transmission system for transmitting three-dimensional point cloud data 60 via a network 50, a transmitter 10 having two types of encoding means (11, 12) for performing two-stage encoding of three-dimensional point cloud data 60 using the two types of encoding means (11, 12), namely, coarse encoding for reducing the number of points and fine encoding for compressing while maintaining quality; a controller (30) that determines encoding parameters (M, C) for two types of encoding means (11, 12) based on conditions (N, P, R, D) and causes a transmitter (10) to perform two-stage encoding of the three-dimensional point cloud data (60) using the encoding parameters (M, C); Equipped with.

[0021] The transmitter 10 includes a coarse coding unit 11 that reduces the number of points in the 3D point cloud data 60, and a fine coding unit 12. Note that fine coding refers to coding other than point reduction of point cloud data. Examples of fine coding include G-PCC, V-PCC, and PCL. The transmitter 10 encodes and compresses the 3D point cloud data 60 in two stages using the coarse coding unit 11 and the fine coding unit 12, and outputs the compressed data to the network 50. The transmitter 10 receives instructions for each encoding parameter from the controller 30.

[0022] Specifically, the transmitter 10 operates as follows. Step S1: The point group 60 having N points is input to the coarse encoding unit 11. Step S2: In accordance with the encoding parameter M from the controller 30, the coarse encoding unit 11 reduces the number of points N included in the point group 60 to the number of points M. The coarse encoding unit 11 performs point reduction in the same manner as in general PCL. Step S3: The coarse encoder 11 inputs the group of points having M points to the fine encoder 12. Step S4: According to the encoding parameter set C from the controller 30, the Fine encoder 12 encodes the point cloud using an existing point cloud encoding technique (for example, G-PCC). Step S5: The encoded point cloud is transmitted to the receiver 20 via the network 50 in band D. Step S6: The decoding unit 21 of the receiver 20 decodes the transmitted point group in accordance with the encoding parameter group C from the controller 30. Step S7: The rendering unit 22 of the receiver 20 renders the decoded point cloud and displays it on the display unit 40, such as a headset or a display.

[0023] The controller 30 is a controller that controls the transmitter 10, an encoding parameter determination table 32 that describes the correspondence between the encoding parameters (M, C) and the conditions (N, P, R, D) for Coarse encoding and Fine encoding; a control unit (31) that checks input conditions (N, P, R, D) against an encoding parameter determination table (32), extracts encoding parameters (M, C) that match the conditions, and causes the transmitter (10) to perform the two-stage encoding using the extracted encoding parameters; Equipped with.

[0024] The controller 30 receives as input a transmission quality requirement R consisting of at least one of a communication bandwidth, a required delay, and a required quality, network bandwidth information D, score information N, and server processing capacity information P. It is not necessary for all of the conditions (N, P, R, D) to be present; at least one of them is sufficient. The controller 30 has an encoding parameter determination table 32 that lists an encoding parameter (score reduction amount) M for coarse encoding and an encoding parameter C for fine encoding for each of the conditions (N, P, R, D). FIG. 2 is a diagram illustrating an example of the encoding parameter determination table 32. The controller 30 determines the reduced score amount M for coarse encoding and the encoding parameter set C for fine encoding based on the collected conditions and the parameter determination table 32.

[0025] The encoding parameter determination table 32 in FIG. 2 will be described. The leftmost column is the type number of the two-stage encoding. In the input field, for each type number, score information N, network bandwidth information D, transmission quality requirement R, and server processing capacity information P are entered. Q is a requirement for the quality of the point cloud. For example, R Q is the Chamfer distance. This refers to the average position error of the transmitted point cloud relative to the original point cloud, and the smaller the distance, the higher the quality. Also, among the transmission quality requirements R, R D is the latency requirement. This is simply the latency, the time it takes to encode the point cloud. In the output column, for each type number, the number of points after reduction M and the encoding parameter group C are entered. The encoding parameter group C will be explained below. "geometry coding configuration" is a parameter for geometry coding. G-PCC converts point clouds into geometric information for compression, and this parameter determines whether the data is represented as a set of triangle meshes or a set of cubes. "Attribute coding configuration" is a parameter for attribute coding, which indicates how the colors of the point cloud are compressed. "Position Quantization Scale" is the quantization scale for the position. If it is "1", no quantization is performed, and if it is "less than 1", the position is quantized and then encoded. "Quantization Parameter" is the quantization scale of the attribute. If it is "0", no quantization is performed. If it is a number greater than 0, the attribute (=color) is quantized before encoding. Note that the above are all examples of parameters, and the encoding parameter set C is not limited to these.

[0026] The purpose of encoding type numbers in two-stage encoding is as follows: Type number 1: Default setting. Classification number 2: Since the network bandwidth is low, increase the compression ratio of encoding without reducing the score. Classification number 3: The network bandwidth is low and there is a delay. Therefore, increase the compression ratio of encoding after reducing the score (reduce the number of points M after reduction and the Position Quantization Scale). This encoding reduces the delay while degrading the quality. Classification number 4: The network bandwidth is low and there is a more severe delay. Therefore, further increase the compression ratio of encoding after reducing the score (reduce the number of points M after reduction more than that of classification number 3). This encoding also reduces the delay while degrading the quality. Classification number 5: The network bandwidth is low and there are quality requirements. Under this condition, improve the encoding quality without reducing the score (make the Position Quantization Scale closer to 1 and increase the Quantization Parameter). This encoding improves the quality while increasing the delay. Classification number 6: The network bandwidth is low and there are more severe quality requirements. Under this condition, also improve the encoding quality without reducing the score (use trisoup / RAHT). This encoding also improves the quality while increasing the delay. Classification number 7: The network bandwidth is low and there are quality requirements, but the server processing capacity is high. Under this condition, improve the encoding quality without reducing the score (adopt Inter octree). This encoding also improves the quality while increasing the delay.

[0027] In the table of FIG. 2, the input conditions are a single value such as N = 850000 or D = 100, but a range such as 600000 < N < 900000 or 50 < D < 150 may be used.

[0028] That is, the point cloud data transmission system 301 performs the following data transmission method. The data transmission method is Determine the encoding parameters of two types of encoding means (11, 12), namely Coarse encoding for reducing the number of points and Fine encoding for compression while maintaining the quality, based on the conditions (N, P, R, D). Two types of encoding means (11, 12) perform two-stage encoding of the three-dimensional point cloud data 60 using the encoding parameters; and Transmitting the two-stage encoded three-dimensional point cloud data to a network 50. Do the following.

[0029] In the point cloud data transmission system 301, the controller 30 determines the coarse encoding parameters and the fine encoding parameters based on the conditions (N, P, R, D) and controls the two-stage encoding, thereby enabling the transmission of 3D point clouds while balancing low latency and point cloud quality over a limited network bandwidth. For example, the following case can be considered. Case 1: When low latency is required, reduce the number of points using coarse encoding and then perform fine encoding. Case 2: When point cloud quality is required, the number of points is not reduced by coarse encoding, but high-quality parameters are given by fine encoding. Case 3: When the bandwidth is limited, parameters are given that reduce the number of points using coarse encoding and achieve a high compression rate using fine encoding.

[0030] (Embodiment 2) 3 is a diagram illustrating a point cloud data transmission system 302 according to this embodiment. The point cloud data transmission system 302 includes a repeater 15, and differs from the point cloud data transmission system 301 in FIG. 1 in that the coarse encoding unit 11 is located in the transmitter 10 and the fine encoding unit 12 is located in the repeater 15.

[0031] In a configuration in which the network is divided into two sections, network 51 and network 52, two-stage encoding can be divided into each network. Examples of cases where a network may be divided into two sections include the following: Case 1: A configuration in which the network 51 is wired and the network 52 is wireless. Case 2: A configuration in which the network 51 is wireless and the network 52 is wired. Case 3: A configuration in which a relay device 15 with a large processing volume is located in a cloud data center. Specifically, a configuration in which the network 51 is a closed network and the network 52 is the Internet, a configuration in which the network 51 is the Internet and the network 52 is a closed network, or a configuration in which both networks (51, 52) are the Internet.

[0032] The operations of the point cloud data transmission system 302 and the controller 30 are the same as those of the point cloud data transmission system 301 and the controller 30 described in Fig. 1, but because there are two networks, one more condition is input to the controller 30, resulting in condition (N, P, R, D1, D2). D1 is the bandwidth information of the network 51, and D2 is the bandwidth information of the network 52.

[0033] The encoding parameter determination table 32 provided in the controller 30 is a table as shown in Fig. 4. The encoding parameter determination table 32 in Fig. 4 will be described. The leftmost column is the type number of the two-stage encoding. In the input fields, score information N, network bandwidth information D1 and D2, transmission quality requirement R, and server processing capacity information P are entered for each type number. In the output column, the number of points after reduction M and the encoding parameter group C are entered for each type number.

[0034] The purpose of encoding type numbers in two-stage encoding is as follows: Type number 11: Default setting. Type number 12: The network bandwidth D2 of the network 52 is small. Since the network bandwidth D1 of the network 51 remains unchanged, the compression rate of encoding is increased without reducing the number of points, thereby reducing the bandwidth of the point cloud transmitted through the network 52. Type number 13: The network bandwidth D1 of the network 51 is small. The number of points is reduced to reduce the bandwidth of the point cloud transmitted over the network 51. On the other hand, the network bandwidth D2 of the network 52 remains unchanged, so the encoding compression rate is set to the same as the default.

[0035] In the table of FIG. 4, the input conditions are single numerical values such as N = 850000 and D2 = 100, but they may have a range such as 600000 < N < 900000 and 50 < D2 < 150. In addition, the controller 30 may output the encoding parameters using not only the encoding parameter determination table 32 of FIG. 4 for the case of two networks, but also the encoding parameter determination table 32 of FIG. 2 if effective.

[0036] Similar to the point cloud data transmission system 301 in FIG. 1, the point cloud data transmission system 302 can transmit a three-dimensional point cloud while balancing low latency and point cloud quality in a limited network bandwidth by the controller 30 determining the parameters of Coarse encoding and Fine encoding based on the conditions (N, P, R, D) and performing two-stage encoding control.

[0037] Furthermore, the point cloud data transmission system 302 with multiple networks can also handle cases where the network bottleneck varies by section or the processing capacity varies by location by separating the implementation locations of Coarse encoding and Fine encoding. For example, the following cases can be considered. Case 1: When the network 51 is wired and the network 52 is wireless, the controller 30 does not perform point reduction by Coarse encoding at the transmitter 10 and gives parameters with a higher compression rate to Fine encoding at the relay 15. Case 2: When the network 51 is wireless and the network 52 is wired, the controller 30 performs point reduction by Coarse encoding at the transmitter 10 and gives parameters with higher quality to Fine encoding at the relay 15. Case 3: In the configuration where the relay 15 with a large processing volume is in the cloud data center, the controller 30 does not perform point reduction by Coarse encoding at the transmitter 10 and gives parameters with a large computational processing volume to Fine encoding at the relay 15.

[0038] (Embodiment 3) By loading the two-stage encoding program into a computer, the computer can be made to function as the controller 30. The program can be recorded on a recording medium or provided via a network. [Explanation of symbols]

[0039] 10:Transmitter 11:Coarse encoding part 12:Fine encoding section 15: Repeater 20: Receiver 21: Decryption unit 22: Rendering section 30: Controller 31: Control unit 32: Encoding parameter determination table 40: Display device 50, 51, 52: Network 60: 3D point cloud data 61: Requirement information input machine 62: Bandwidth information input machine 301, 302: Point cloud data transmission system

Claims

1. A point cloud data transmission system for transmitting three-dimensional point cloud data via a network, a transmitter having two types of encoding means, namely, coarse encoding for reducing the number of points and fine encoding for compressing while maintaining quality, and for performing two-stage encoding of the 3D point cloud data using the two types of encoding means; a controller that determines encoding parameters for each of the two types of encoding means based on a condition and causes the transmitter to perform two-stage encoding of the 3D point cloud data using the determined encoding parameters; A point cloud data transmission system comprising:

2. A controller that controls a transmitter that has two types of encoding means, namely, coarse encoding that reduces the number of points and fine encoding that compresses while maintaining quality, and that performs two-stage encoding of three-dimensional point cloud data using the two types of encoding means and transmits the data to a network, a coding parameter determination table that describes the correspondence between the coding parameters and conditions for the coarse coding and the fine coding; a control unit that checks the input conditions against the encoding parameter determination table, extracts the encoding parameters that match the conditions, and causes the transmitter to perform the two-stage encoding using the extracted encoding parameters; A controller comprising:

3. Coarse encoding, which reduces the number of points, and fine encoding, which compresses while maintaining quality, are used to determine the encoding parameters of each of the two encoding methods based on conditions. performing two-stage encoding of the three-dimensional point cloud data using the encoding parameters by the two types of encoding means; and Transmitting the two-stage encoded three-dimensional point cloud data to a network. A data transmission method comprising:

4. A program that causes a computer to function as the controller according to claim 2.

Images