Point cloud decoding device, point cloud decoding method, and program

Abstract

To provide a method for decoding a point group and a program which can improve the compression performance of coding.SOLUTION: In a point group processing system having a point group coder and a point group decoder, a point group decoder 200 has a tree synthesizing unit 2020 for performing processing differently according to the type of a sensor or an operation mode in an Angular mode of Predictive coding, and responds to processing to Spinning LiDAR data and to processing to Non-Spinning LiDAR data by switching the type of a sensor or the value of the operation mode.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a point cloud decoding apparatus, a point cloud decoding method, and a program.

Background Art

[0007] Therefore, the present invention has been made in view of the above-mentioned problems, and aims to provide a point cloud decoding device, a point cloud decoding method, and a program that can improve the compression performance of encoding. [Means for solving the problem]

[0008] The first feature of the present invention is a point cloud decoding device that includes a tree synthesis unit that performs different processing depending on the sensor type or operating mode in the Angular mode of predictive coding.

[0009] The second feature of the present invention is a point cloud decoding method that includes a step of performing different processing depending on the sensor type or operating mode in the Angular mode of predictive coding.

[0010] A third feature of the present invention is a program that causes a computer to function as a point cloud decoder, wherein the point cloud decoder includes a tree synthesis unit that performs different processing depending on the sensor type or operating mode in the Angular mode of predictive coding. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a point cloud decoding device, a point cloud decoding method, and a program that can improve the compression performance of encoding. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 shows an example of the configuration of a point cloud processing system 10 according to one embodiment. [Figure 2]Figure 2 shows an example of the functional block of a point cloud decoding device 200 according to one embodiment. [Figure 3] Figure 3 shows an example of the configuration of encoded data (bitstream) received by the geometric information decoding unit 2010 of a point cloud decoding device 200 according to one embodiment. [Figure 4] Figure 4 shows an example of the syntax configuration of GPS2011. [Figure 5] Figure 5 is a flowchart showing an example of processing in the tree synthesis unit 2020 of the point cloud decoding device 200 according to one embodiment. [Figure 6] Figure 6 is a flowchart showing an example of the slice data decoding process in step S505. [Figure 7] Figure 7 illustrates an example of a coordinate transformation process, specifically a transformation to a polar coordinate system. [Figure 8A] Figure 8A illustrates an example of processing when the flag controlling the sensor type indicates Spinning LiDAR data. [Figure 8B] Figure 8B illustrates an example of processing when the flag controlling the sensor type indicates non-spinning LiDAR data. [Figure 9] Figure 9 is a diagram illustrating an example of the prediction method in step S604. [Figure 10] Figure 10 is a flowchart showing an example of the coordinate prediction process in step S604. [Figure 11] Figure 11 illustrates an example of the process of selecting a predictor from a reference frame in step S1004, when the sensor type is Non-Spinning LiDAR or when elevation angle component prediction and residual decoding are enabled in Angular mode. [Figure 12]FIG. 12 is a diagram for explaining an example of a process of selecting a predictor from a reference frame when the sensor type indicates Non-Spinning LiDAR or when prediction of an elevation angle component and residual decoding are effective in the Angular mode in step S1004. [Figure 13] FIG. 13 is a diagram showing an example of functional blocks of the point cloud encoding device 100 according to the present embodiment.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0014] (First Embodiment) Hereinafter, the point cloud processing system 10 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 13. FIG. 1 is a diagram showing the point cloud processing system 10 according to the embodiment of the present invention.

[0015] As shown in FIG. 1, the point cloud processing system 10 includes a point cloud encoding device 100 and a point cloud decoding device 200.

[0016] The point cloud encoding device 100 is configured to generate encoded data (bitstream) by encoding an input point cloud signal. The point cloud decoding device 200 is configured to generate an output point cloud signal by decoding the bitstream.

[0017] Note that the input point cloud signal and the output point cloud signal are composed of position information and attribute information of each point in the point cloud. The attribute information is, for example, color information or reflectivity of each point.

[0018] Here, such a bitstream may be transmitted from the point cloud encoding device 100 to the point cloud decoding device 200 via a transmission line. Alternatively, the bitstream may be stored in a storage medium and then provided from the point cloud encoding device 100 to the point cloud decoding device 200.

[0019] (Point cloud decoder 200) The point cloud decoder 200 according to this embodiment will be described below with reference to Figure 2. Figure 2 is a diagram showing an example of the functional block of the point cloud decoder 200 according to this embodiment.

[0020] As shown in Figure 2, the point cloud decoding device 200 includes a geometric information decoding unit 2010, a tree synthesis unit 2020, an approximate surface synthesis unit 2030, a geometric information reconstruction unit 2040, an inverse coordinate transformation unit 2050, an attribute information decoding unit 2060, an inverse quantization unit 2070, an RAHT unit 2080, an LoD calculation unit 2090, an inverse lifting unit 2100, an inverse color transformation unit 2110, and a frame buffer 2120.

[0021] The geometric information decoding unit 2010 is configured to take the bitstream related to geometric information (geometric information bitstream) from the bitstream output from the point cloud coding device 100 as input and decode the syntax.

[0022] The decoding process is, for example, a context-adaptive binary arithmetic decoding process. Here, for example, the syntax includes control data (flags and parameters) to control the decoding process of the location information.

[0023] The tree synthesis unit 2020 is configured to generate tree information indicating which regions within the decoding target space contain points, by taking as input the control data decoded by the geometric information decoding unit 2010 and the occupancy code, which indicates which node in the tree (described later) contains the point cloud.

[0024] The system may also be configured to perform the decoding of the occupancy code within the tree synthesis unit 2020.

[0025] This process divides the space to be decoded into rectangular prisms, determines whether a point exists within each rectangular prism by referring to the occupancy code, divides the rectangular prism containing a point into multiple rectangular prisms, and recursively repeats the process of referring to the occupancy code, thereby generating tree information.

[0026] Here, interpretation, as described later, may be used when decoding such occupancy code.

[0027] In this embodiment, a method called "Octree" can be used, which recursively performs octree partitioning by always treating the cuboid as a cube, and a method called "QtBt" can be used, which performs quadtree partitioning and binary partitioning in addition to octree partitioning. Whether or not to use "QtBt" is transmitted as control data from the point cloud encoding device 100.

[0028] Alternatively, if the control data specifies that predictive geometry coding should be used, the tree synthesis unit 2020 is configured to decode the coordinates of each point based on an arbitrary tree configuration determined by the point cloud coding device 100.

[0029] The approximate surface synthesis unit 2030 is configured to generate approximate surface information using tree information generated by the tree synthesis unit 2020, and to decode the point cloud based on this approximate surface information.

[0030] Approximate surface information is used, for example, when decoding 3D point cloud data of an object, in cases where the point cloud is densely distributed on the object's surface. Instead of decoding each individual point cloud, the region where the point cloud exists is approximated and represented by a small plane.

[0031] Specifically, the approximate surface synthesis unit 2030 can generate approximate surface information and decode point clouds using a method called "Trisoup," for example. A specific example of the "Trisoup" process will be described later. Furthermore, this process can be omitted when decoding sparse point clouds acquired by Lidar or the like.

[0032] The geometric information reconstruction unit 2040 is configured to reconstruct the geometric information (position information in the coordinate system assumed by the decoding process) of each point in the point cloud data to be decoded, based on the tree information generated by the tree synthesis unit 2020 and the approximate surface information generated by the approximate surface synthesis unit 2030.

[0033] The inverse coordinate transformation unit 2050 is configured to take the geometric information reconstructed by the geometric information reconstruction unit 2040 as input, transform it from the coordinate system assumed by the decoding process to the coordinate system of the output point cloud signal, and output position information.

[0034] The frame buffer 2120 is configured to take the geometric information reconstructed by the geometric information reconstruction unit 2040 as input and store it as a reference frame. The stored reference frame is read from the frame buffer 2130 and used as a reference frame when the tree synthesis unit 2020 performs interpretation of frames that are different in time.

[0035] Here, the choice of which time reference frame to use for each frame may be determined, for example, based on control data transmitted as a bitstream from the point cloud encoding device 100.

[0036] The attribute information decoding unit 2060 is configured to take the bitstream related to attribute information (attribute information bitstream) from the bitstream output from the point cloud coding device 100 as input and decode the syntax.

[0037] The decoding process is, for example, a context-adaptive binary arithmetic decoding process. Here, for example, the syntax includes control data (flags and parameters) to control the decoding process of attribute information.

[0038] Furthermore, the attribute information decoding unit 2060 is configured to decode quantized residual information from the decoded syntax.

[0039] The inverse quantization unit 2070 is configured to perform inverse quantization processing based on the quantized residual information decoded by the attribute information decoding unit 2060 and the quantization parameter, which is one of the control data decoded by the attribute information decoding unit 2060, in order to generate inverse quantized residual information.

[0040] The inversely quantized residual information is output to either the RAHT unit 2080 or the LoD calculation unit 2090, depending on the characteristics of the point cloud to be decoded. Which unit it is output to is specified by the control data decoded by the attribute information decoding unit 2060.

[0041] The RAHT unit 2080 is configured to take the inversely quantized residual information generated by the inverse quantization unit 2070 and the geometric information reconstruction unit 2040 as input, and decode the attribute information of each point using a type of Haar transform called RAHT (Region Adaptive Hierarchical Transform) (inverse Haar transform in the decoding process). As a specific example of the RAHT processing, the method described in Non-Patent Document 1 can be used.

[0042] The LoD calculation unit 2090 is configured to take geometric information generated by the geometric information reconstruction unit 2040 as input and generate LoD (Level of Detail).

[0043] LoD (Level of Data) is information used to define reference relationships (referring points and referenced points) for implementing predictive coding, which involves predicting the attribute information of one point from the attribute information of another point and then encoding or decoding the prediction residual.

[0044] In other words, LoD is information that defines a hierarchical structure in which each point included in geometric information is classified into multiple levels, and the attributes of points belonging to lower levels are encoded or decoded using the attribute information of points belonging to higher levels.

[0045] As for a specific method for determining the LoD, for example, the method described in Non-Patent Document 1 above may be used.

[0046] The inverse lifting unit 2100 is configured to decode the attribute information of each point based on the hierarchical structure defined by the LoD, using the LoD generated by the LoD calculation unit 2090 and the inversely quantized residual information generated by the inverse quantization unit 2070. As a specific processing method for inverse lifting, for example, the method described in Non-Patent Document 1 above can be used.

[0047] The reverse color conversion unit 2110 is configured to perform reverse color conversion on the attribute information output from the RAHT unit 2080 or the reverse lifting unit 2100 if the attribute information to be decoded is color information and color conversion has been performed on the point cloud encoding device 100 side. Whether or not such reverse color conversion processing is performed is determined by the control data decoded by the attribute information decoding unit 2060.

[0048] The point cloud decoder 200 is configured to decode and output attribute information for each point in the point cloud through the above processing.

[0049] (Geometric information decoding unit 2010) The control data decoded by the geometric information decoding unit 2010 will be explained below using Figures 3 and 4.

[0050] Figure 3 shows an example of the configuration of encoded data (bitstream) received by the geometric information decoding unit 2010.

[0051] Firstly, the bitstream may include GPS2011. GPS2011, also known as the geometry parameter set, is a set of control data related to the decoding of geometric information. Specific examples will be discussed later. Each GPS2011 includes at least GPS ID information to identify each individual GPS2011 if multiple GPS2011s exist.

[0052] Secondly, the bitstream may include GSH2012A / 2012B. GSH2012A / 2012B, also called geometry slice headers or geometry data unit headers, are sets of control data corresponding to slices, which will be described later. Hereafter, we will use the term "slice," but you can also read "slice" as "data unit." Specific examples will be given later. Each GSH2012A / 2012B includes at least GPS ID information to specify the GPS2011 corresponding to each GSH2012A / 2012B.

[0053] Thirdly, the bitstream may include slice data 2013A / 2013B after GSH2012A / 2012B. Slice data 2013A / 2013B contains data that encodes geometric information. An example of slice data 2013A / 2013B is encoded data using occupancy code or predicative coding, which will be discussed later.

[0054] As described above, the bitstream is configured so that each slice data 2013A / 2013B corresponds to one GSH2012A / 2012B and one GPS2011.

[0055] As described above, GSH2012A / 2012B allows you to specify which GPS2011 to refer to using GPS ID information, so a common GPS2011 can be used for multiple slice data 2013A / 2013B.

[0056] In other words, GPS2011 does not necessarily need to be transmitted for each slice. For example, as shown in Figure 3, the bitstream can be configured so that GPS2011 is not encoded immediately before GSH2012B and slice data 2013B.

[0057] Note that the configuration in Figure 3 is merely an example. As long as each slice data 2013A / 2013B corresponds to GSH2012A / 2012B and GPS2011, other elements besides those mentioned above may be added as components of the bitstream.

[0058] For example, as shown in Figure 3, the bitstream may include a sequence parameter set (SPS) 2001. Similarly, it may be formatted to a different configuration than that shown in Figure 3 during transmission. Furthermore, it may be combined with the bitstream decoded by the attribute information decoding unit 2060 described later and transmitted as a single bitstream.

[0059] Figure 4 shows an example of the GPS2011 syntax configuration.

[0060] Please note that the syntax names described below are merely examples. If the functionality of the syntax described below is the same, the syntax names may differ.

[0061] GPS2011 may include GPS ID information (gps_geom_parameter_set_id) to identify each GPS2011.

[0062] In Figure 4, the Descriptor column indicates how each syntax is encoded. ue(v) means it is an unsigned zero-order exponential Golomb code, and u(1) means it is a 1-bit flag.

[0063] GPS2011 may include a flag (interprediction_enabled_flag) that controls whether or not interprediction is performed in the tree synthesis unit 2020.

[0064] For example, you could define that if the value of `interprediction_enabled_flag` is "0", then interprediction will not be performed, and if the value of `interprediction_enabled_flag` is "1", then interprediction will be performed.

[0065] Note that the `interprediction_enabled_flag` may be included in SPS2001 instead of GPS2011.

[0066] GPS2011 may include a flag (geom_tree_type) for controlling the tree type in the tree synthesis unit 2020. For example, a value of "1" for geom_tree_type may define the use of predicative coding, while a value of "0" for geom_tree_type may define the disuse of predicative coding.

[0067] Note that geom_tree_type may be included in SPS2001 instead of GPS2011.

[0068] GPS2011 may include a flag (geom_angular_enabled) to control whether or not to process in Angular mode in the tree synthesis unit 2020.

[0069] For example, if the value of geom_angular_enabled is "1", it may be defined that predictive coding will be performed in Angular mode, and if the value of geom_angular_enabled is "0", it may be defined that predictive coding will not be performed in Angular mode.

[0070] Note that geom_angular_enabled may be included in SPS2001 instead of GPS2011.

[0071] GPS2011 may include a flag (sensor_type) in the tree composition unit 2020 for controlling the sensor type in Angular mode.

[0072] For example, it might be defined that if the value of sensor_type is "1", it will be processed as spinning LiDAR data, and if the value of sensor_type is "0", it will be processed as non-spinning LiDAR data.

[0073] Note that `sensor_type` may be included in `SPS2001` instead of `GPS2011`. Alternatively, instead of `sensor_type`, a flag used to control the switching of operating modes in Angular mode, as described later, may be used.

[0074] GPS2011 may include a flag (global_motion_enabled_flag) that controls whether or not to perform global motion compensation for interpretation in the tree synthesis unit 2020.

[0075] For example, you could define that global motion compensation is disabled if the value of global_motion_enabled_flag is "0", and that global motion compensation is enabled if the value of global_motion_enabled_flag is "1".

[0076] When performing global motion compensation, each slice data may include a global motion vector.

[0077] Note that global_motion_enabled_flag may be included in SPS2001 instead of GPS2011.

[0078] (Tree Synthesis Department 2020) The processing of the tree synthesis unit 2020 will be explained below using Figures 5 to 11. Figure 5 is a flowchart showing an example of the processing in the tree synthesis unit 2020. The following explanation will describe an example of synthesizing a tree using "Predictive geometry coding".

[0079] Note that terms such as "predictive geometry," "predictive geometry coding," and "predictive tree" are sometimes used instead of "predictive coding."

[0080] As shown in Figure 5, in step S501, the tree synthesis unit 2020 determines whether to use interprediction based on the value of interprediction_enabled_flag.

[0081] If the tree synthesis unit 2020 determines that interpretation should be used, it proceeds to step S502; otherwise, it proceeds to step S505.

[0082] In step S502, the tree composition unit 2020 obtains a reference frame from the frame buffer 2120. The frame buffer 2120 may already store one previously decoded frame, and the addition of decoded frames to the frame buffer 2020 may be performed one frame at a time or a predetermined number of frames after the decoding is completed. After obtaining the reference frame, the tree composition unit 2020 proceeds to step S503.

[0083] In step S503, the tree synthesis unit 2020 determines whether to perform global motion compensation based on the global_motion_enabled_flag.

[0084] If the tree synthesis unit 2020 determines that global motion compensation should be performed, it proceeds to step S504; if it determines that global motion compensation should not be performed, it proceeds to step S505.

[0085] In step S504, the tree synthesis unit 2020 performs global motion compensation on the reference frame acquired in step S502.

[0086] Here, global motion compensation is a process that corrects the global positional shift for each frame, and applies rotation and translation based on the global motion vector decoded by the geometric information decoding unit 2010 to all or a specified range of points in the reference frame.

[0087] After performing global motion compensation, the tree synthesis unit 2020 proceeds to step S505.

[0088] In step S505, the tree synthesis unit 2020 decodes the slice data. The specific processing in step S505 will be described later. After decoding the slice data, the tree synthesis unit 2020 proceeds to step S506.

[0089] In step S506, the tree synthesis unit 2020 terminates its processing. Note that the processing in steps S503 and S504, i.e., the determination and execution of global motion compensation, may be performed during the slice data decoding process in step S505.

[0090] Figure 6 is a flowchart showing an example of the slice data decoding process in step S505.

[0091] As shown in Figure 6, in step S601, the tree synthesis unit 2020 constructs a prediction tree corresponding to the slice data.

[0092] The slice data may contain a list of the number of child nodes of each node in the prediction tree, sorted in depth-first order. One way to construct the prediction tree is to start from the root node and add the number of child nodes specified in the above list to each node, in depth-first order.

[0093] After completing the construction of the prediction tree, the tree synthesis unit 2020 proceeds to step S602.

[0094] In step S602, the tree synthesis unit 2020 determines whether processing of all nodes in the prediction tree has been completed.

[0095] If the tree synthesis unit 2020 determines that processing of all nodes in the prediction tree is complete, it proceeds to step S607; otherwise, it proceeds to step S603.

[0096] In step S603, the tree synthesis unit 2020 selects the node to be processed from the prediction tree.

[0097] The tree synthesis unit 2020 may select the node that follows the previously processed node in depth-first order as the node to be processed.

[0098] After the tree synthesis unit 2020 has finished selecting the nodes to be processed, it proceeds to step S604.

[0099] In step S604, the tree synthesis unit 2020 predicts the coordinates of the points corresponding to the nodes to be processed. The specific method for predicting these coordinates will be described later.

[0100] After completing the prediction, the tree synthesis unit 2020 proceeds to step S605.

[0101] In step S605, the tree synthesis unit 2020 decodes the predicted residuals of the coordinates of the points corresponding to the nodes to be processed. The slice data may include the predicted residuals of the coordinates of the points corresponding to each node.

[0102] After the tree synthesis unit 2020 has finished decoding the predicted residuals of the target node, it proceeds to step S606.

[0103] In step S606, the tree synthesis unit 2020 reconstructs the coordinates of the points corresponding to the nodes to be processed. The tree synthesis unit 2020 may determine the coordinates of the points by summing the coordinates predicted in step S604 and the residuals decoded in step S605.

[0104] After the coordinate reconstruction is complete, the tree synthesis unit 2020 returns to step 602.

[0105] In step S607, the tree synthesis unit 2020 terminates the process in step S505.

[0106] Here, the order of steps S604 and S605 may be reversed.

[0107] If Angular mode is used, the tree composition unit 2020 may take into consideration that the coordinate values ​​handled in steps S604 to S606 are the values ​​after coordinate transformation, and step S606 may include the process of the inverse transformation.

[0108] Figure 7 illustrates an example of a coordinate transformation process, specifically a transformation to a polar coordinate system. As shown in Figure 7, the coordinate values ​​of each point are expressed as radius r, elevation angle θ, and azimuth angle φ.

[0109] Furthermore, the elevation angle θ and azimuth angle φ among the coordinate values ​​after polar coordinate transformation may be subjected to further appropriate transformations. For example, the two values ​​of elevation angle θ and azimuth angle φ may be treated as two-dimensional Cartesian coordinates, and a further two-dimensional polar coordinate transformation may be applied.

[0110] Figure 8A illustrates an example of processing when the flag controlling the sensor type indicates Spinning LiDAR data, and Figure 8B illustrates an example of processing when the flag controlling the sensor type indicates Non-spinning LiDAR data.

[0111] As shown in Figure 8A, when the sensor type is Spinning LiDAR, the tree synthesis unit 2020 processes only the radius r and azimuth angle φ from the coordinate values ​​in steps S604 and S605, and in step S606, it reconstructs the radius r and azimuth angle φ, while setting the elevation angle θ from a predefined fixed value.

[0112] For example, the tree synthesis unit 2020 may set the elevation angle θ to the elevation angle of one of the multiple lasers in the LiDAR, and the laser ID indicating which laser it is may be included in the slice data.

[0113] On the other hand, if the sensor type is Non-Spinning LiDAR, the tree synthesis unit 2020 may perform prediction, residual decoding, and coordinate reconstruction processing for all of the coordinate values ​​r, θ, and φ in steps S604 to S606.

[0114] The parameters used for predicting r, θ, and φ, decoding residuals, and reconstructing coordinates may be set individually or as a common set. These parameters include, for example, the scale values ​​used when scaling up to represent the angles (radians) of θ and φ as integer values, and the quantization step size used to quantize the values ​​of θ and φ. These parameters may be included in the GPS2011 or SPS2001 header information.

[0115] Furthermore, the tree synthesis unit 2020 may switch between the two operations described above based on the value of a flag that controls whether or not to enable elevation angle component prediction and residual decoding, rather than a flag that controls the sensor type.

[0116] Figure 9 is a diagram illustrating an example of the prediction method in step S604.

[0117] In step S604, the tree synthesis unit 2020 may perform a prediction using only one of the consecutive N points, as shown in Figure 9. The tree synthesis unit 2020 may then share the predicted value of that one point as the predicted value for the remaining N-1 points.

[0118] Figure 10 is a flowchart showing an example of the coordinate prediction process in step S604.

[0119] As shown in Figure 10, in step S1001, the tree synthesis unit 2020 decodes the predictor flag.

[0120] Here, the slice data may include flags indicating the predictor to be used for each node. For example, the slice data may include flags similar to those described in Non-Patent Documents 1 and 2, such as a flag indicating whether it is an inter-predictor or an intra-predictor, or an index for the inter-predictor. The slice data may also include other flags described later.

[0121] After decoding the predictor flag, the tree synthesis unit 2020 proceeds to step S1002.

[0122] In step S1002, the tree synthesis unit 2020 determines whether to use the interpreter based on the predictor flag decoded in step S1001.

[0123] If the tree synthesis unit 2020 determines that the inter predictor should be used, it proceeds to step S1004; otherwise, it proceeds to step S1003.

[0124] In step S1003, the tree synthesis unit 2020 performs intra-prediction of the coordinates of the nodes to be processed.

[0125] When performing intra-prediction, the tree synthesis unit 2020 configures a predictor based on the coordinates of the parent node or ancestor node (for example, the parent node's parent node) of the node to be processed, and predicts the coordinates of the node to be processed.

[0126] The tree synthesis unit 2020 may use the methods described in Non-Patent Documents 1 and 2 for configuring the intra predictor, and may use the predictor indicated by the predictor flag decoded in step S1001 from among the multiple intra predictors.

[0127] After completing intra-prediction, the tree synthesis unit 2020 proceeds to step S1005.

[0128] In step S1004, the tree synthesis unit 2020 performs inter-coordinate prediction for the nodes to be processed.

[0129] When performing interpretation, the tree synthesis unit 2020 selects a node corresponding to the node to be processed from the reference frame as a predictor, and uses the coordinates of the selected predictor as the predicted value of the coordinates of the node to be processed.

[0130] In Angular mode, if the sensor type is Spinning LiDAR or if elevation angle component prediction and residual decoding are disabled, the tree synthesis unit 2020 may select a predictor from the reference frame using the same method as in Non-Patent Document 2.

[0131] Furthermore, in Angular mode, if the sensor type is Non-Spinning LiDAR or if elevation angle component prediction and residual decoding are enabled, the tree synthesis unit 2020 selects a predictor from the reference frame, as described later.

[0132] After the inter-prediction is complete, the tree synthesis unit 2020 proceeds to step S1005.

[0133] In step S1005, the tree synthesis unit 2020 terminates the process in step S604.

[0134] Figures 11 and 12 illustrate an example of the process of selecting a predictor from a reference frame in step S1004 described above, when the sensor type is Non-Spinning LiDAR or when elevation angle component prediction and residual decoding are enabled in Angular mode.

[0135] Figure 11 shows an example where, for the parent node of the node to be processed, the node with the closest coordinate values ​​θ and φ among the nodes in the reference frame is searched for, and a predictor is selected from its child nodes and grandchild nodes.

[0136] Figure 12 shows an example where each node has a unique node ID within the frame, and a predictor is selected from the reference frame based on the node ID of the node to be processed.

[0137] In Angular mode, the tree synthesis unit 2020 may configure the prediction tree so that, when the sensor type is Non-Spinning LiDAR or when elevation angle component prediction and residual decoding are enabled, the prediction tree consists of N branches connected in series from the root, and each node may hold a branch ID indicating the branch to which it belongs.

[0138] The tree synthesis unit 2020 may limit the range in which it searches for a predictor from the reference frame based on the branch ID of the parent node of the node to be processed.

[0139] As described above, in this embodiment, the tree synthesis unit 2020 is configured to perform different processing depending on the sensor type or operating mode in the Angular mode of predictive coding.

[0140] Furthermore, in Angular mode, the tree synthesis unit 2020 may be configured to perform prediction, residual decoding, and coordinate reconstruction processing for each of the coordinate values ​​r, θ, and φ after polar coordinate transformation, when the sensor type indicates Non-Spinning LiDAR data or when the operating mode indicates the activation of elevation angle component prediction and residual decoding.

[0141] Furthermore, the tree synthesis unit 2020 may be configured to perform prediction, residual decoding, and coordinate reconstruction processing for each of the coordinate values ​​r, θ, and φ after applying a coordinate transformation to θ and φ among the coordinate values ​​r, θ, and φ after polar coordinate transformation, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding.

[0142] Furthermore, in Angular mode, the tree synthesis unit 2020 may be configured to perform a prediction at one of multiple points and treat it as a common predicted value with the other points when the sensor type indicates Non-Spinning LiDAR data or when the operating mode indicates the activation of elevation angle component prediction and residual decoding.

[0143] Furthermore, in Angular mode, the tree synthesis unit 2020 may be configured to select a predictor in interpretation based on the coordinate values ​​θ and φ after polar coordinate transformation from the reference frame when the sensor type indicates Non-Spinning LiDAR data or when the operating mode indicates the activation of elevation component prediction and residual decoding.

[0144] Furthermore, in Angular mode, the tree synthesis unit 2020 may be configured to select a predictor based on the node ID from the reference frame during interpretation when the sensor type indicates Non-Spinning LiDAR data or when the operating mode indicates the activation of elevation component prediction and residual decoding.

[0145] Furthermore, in Angular mode, the tree synthesis unit 2020 may be configured to select a predictor based on the branch ID of the prediction tree of the reference frame during interpretation when the sensor type indicates Non-Spinning LiDAR data or when the operating mode indicates the activation of elevation component prediction and residual decoding.

[0146] (Point cloud encoding device 100) The point cloud coding device 100 according to this embodiment will be described below with reference to Figure 13. Figure 13 is a diagram showing an example of the functional blocks of the point cloud coding device 100 according to this embodiment.

[0147] As shown in Figure 13, the point cloud coding device 100 includes a coordinate transformation unit 1010, a geometric information quantization unit 1020, a tree analysis unit 1030, an approximate surface analysis unit 1040, a geometric information coding unit 1050, a geometric information reconstruction unit 1060, a color conversion unit 1070, an attribute transfer unit 1080, a RAHT unit 1090, a LoD calculation unit 1100, a lifting unit 1110, an attribute information quantization unit 1120, an attribute information coding unit 1130, and a frame buffer 1140.

[0148] The coordinate transformation unit 1010 is configured to perform a transformation process from the 3D coordinate system of the input point cloud to any different coordinate system. The coordinate transformation may be performed, for example, by rotating the input point cloud to transform the x, y, and z coordinates of the input point cloud into arbitrary s, t, and u coordinates. Alternatively, as one variation of the transformation, the coordinate system of the input point cloud may be used as is.

[0149] The geometric information quantization unit 1020 is configured to quantize the position information of the input point cloud after coordinate transformation and to remove points with overlapping coordinates. When the quantization step size is 1, the position information of the input point cloud and the position information after quantization coincide. In other words, when the quantization step size is 1, it is equivalent to not performing quantization.

[0150] The tree analysis unit 1030 is configured to take the position information of the quantized point cloud as input and generate an occupancy code that indicates which node in the encoding target space a point is located at, based on the tree structure described later.

[0151] The tree analysis unit 1030 is configured to generate a tree structure in this process by recursively dividing the space to be encoded into rectangular parallelepipeds.

[0152] Here, if a point exists within a given rectangular prism, a tree structure can be generated by recursively dividing that rectangular prism into multiple rectangular prisms until the rectangular prism reaches a predetermined size. Each of these rectangular prisms is called a node. Each rectangular prism generated by dividing a node is called a child node, and the occupancy code is a representation of whether or not a point is contained within a child node, expressed as 0 or 1.

[0153] As described above, the tree analysis unit 1030 is configured to generate occupancy code by recursively dividing the nodes until they reach a predetermined size.

[0154] In this embodiment, a method called "Octree" can be used, which recursively performs octree partitioning by always treating the cuboid as a cube, and a method called "QtBt" can be used, which performs quadtree partitioning and binary tree partitioning in addition to octree partitioning.

[0155] Whether or not to use "QtBt" is transmitted to the point cloud decoder 200 as control data.

[0156] Alternatively, it may be specified to use predictive coding with an arbitrary tree structure. In this case, the tree analysis unit 1030 determines the tree structure, and the determined tree structure is transmitted to the point cloud decoder 200 as control data.

[0157] For example, the control data in a tree structure may be configured to be decryptable using the procedure described in Figures 5 to 12.

[0158] The approximate surface analysis unit 1040 is configured to generate approximate surface information using the tree information generated by the tree analysis unit 1030.

[0159] Approximate surface information is used, for example, when decoding 3D point cloud data of an object, in cases where the point cloud is densely distributed on the object's surface. Instead of decoding each individual point cloud, the region where the point cloud exists is approximated and represented by a small plane.

[0160] Specifically, the approximate surface analysis unit 1040 may be configured to generate approximate surface information using a method called "Trisoup," for example. Furthermore, this process can be omitted when decoding sparse point clouds acquired by Lidar or the like.

[0161] The geometric information encoding unit 1050 is configured to encode the syntax of the occupancy code generated by the tree analysis unit 1030 and the approximate surface information generated by the approximate surface analysis unit 1040, and generate a bitstream (geometric information bitstream). Here, the bitstream may include, for example, the syntax described in Figure 4.

[0162] The encoding process is, for example, context-adaptive binary arithmetic encoding. Here, for example, the syntax includes control data (flags and parameters) to control the decoding process of the location information.

[0163] The geometric information reconstruction unit 1060 is configured to reconstruct the geometric information of each point in the point cloud data to be encoded (the coordinate system assumed by the encoding process, i.e., the position information after the coordinate transformation in the coordinate transformation unit 1010) based on the tree information generated by the tree analysis unit 1030 and the approximate surface information generated by the approximate surface analysis unit 1040.

[0164] The frame buffer 1140 is configured to take geometric information reconstructed by the geometric information reconstruction unit 1060 as input and store it as a reference frame.

[0165] The saved reference frames are read from the frame buffer 1140 and used as reference frames when the tree analysis unit 1030 performs interpretation.

[0166] The color conversion unit 1070 is configured to perform color conversion if the input attribute information is color information. Color conversion is not always necessary; whether or not the color conversion process is performed is encoded as part of the control data and transmitted to the point cloud decoder 200.

[0167] The attribute transfer unit 1080 is configured to correct attribute values ​​so as to minimize distortion of attribute information, based on the position information of the input point cloud, the position information of the point cloud after reconstruction by the geometric information reconstruction unit 1060, and the attribute information after color change by the color conversion unit 1070. For example, the method described in Non-Patent Document 2 can be applied as a specific correction method.

[0168] The RAHT unit 1090 is configured to take the attribute information after the attribute transfer by the attribute transfer unit 1080 and the geometric information reconstruction unit 1060 as input, and generate residual information for each point using a type of Haar transform called RAHT (Region Adaptive Hierarchical Transform). As for the specific processing of RAHT, for example, the method described in Non-Patent Document 2 above can be used.

[0169] The LoD calculation unit 1100 is configured to take geometric information generated by the geometric information reconstruction unit 1060 as input and generate LoD (Level of Detail).

[0170] LoD (Level of Data) is information used to define reference relationships (referring points and referenced points) for implementing predictive coding, which involves predicting the attribute information of one point from the attribute information of another point and then encoding or decoding the prediction residual.

[0171] In other words, LoD is information that defines a hierarchical structure in which each point included in geometric information is classified into multiple levels, and the attributes of points belonging to lower levels are encoded or decoded using the attribute information of points belonging to higher levels.

[0172] As for a specific method for determining the LoD, for example, the method described in Non-Patent Document 2 above may be used.

[0173] The lifting unit 1110 is configured to generate residual information through a lifting process using the LoD generated by the LoD calculation unit 1100 and the attribute information after attribute transfer in the attribute transfer unit 1080.

[0174] As for the specific lifting process, for example, the method described in Non-Patent Document 2 above may be used.

[0175] The attribute information quantization unit 1120 is configured to quantize the residual information output from the RAHT unit 1090 or the lifting unit 1110. Here, when the quantization step size is 1, it is equivalent to not performing quantization.

[0176] The attribute information encoding unit 1130 is configured to encode the quantized residual information output from the attribute information quantization unit 1120 as syntax, and to generate a bitstream related to attribute information (attribute information bitstream).

[0177] The encoding process is, for example, context-adaptive binary arithmetic encoding. Here, for example, the syntax includes control data (flags and parameters) to control the decoding process of attribute information.

[0178] The point cloud encoding device 100 is configured to perform encoding processing on the positional information and attribute information of each point in the point cloud as input, and to output a geometric information bitstream and an attribute information bitstream.

[0179] Furthermore, the point cloud coding device 100 and point cloud decoding device 200 described above may be implemented as programs that cause a computer to execute each function (each process).

[0180] In the above embodiments, the present invention was described using the application to a point cloud coding device 100 and a point cloud decoding device 200 as an example. However, the present invention is not limited to such examples and can be similarly applied to a point cloud coding / decoding system equipped with the functions of the point cloud coding device 100 and the point cloud decoding device 200. [Industrial applicability]

[0181] Furthermore, according to this embodiment, for example, it is possible to achieve an overall improvement in service quality in video communication, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of Symbols]

[0182] 10…Point cloud processing system 100...Point cloud encoding device 1010... Coordinate transformation section 1020...Geometric information quantization section 1030...Tree Analysis Unit 1040…Approximate surface analysis section 1050...Geometric information encoding unit 1060...Geometric information reconstruction unit 1070...Color conversion unit 1080... Attribute Transfer Section 1090...RAHT Department 1100...LoD calculation unit 1110... Lifting Section 1120...Attribute information quantization section 1130...Attribute information encoding unit 1140... Frame buffer 200... Point cloud decoder 2010...Geometric Information Decoding Unit 2020... Tree Composition Section 2030…Approximate surface synthesis part 2040...Geometric information reconstruction unit 2050... Inverse coordinate transformation section 2060... Attribute Information Decoding Unit 2070...Inverse quantization section 2080…RAHT Department 2090...LoD calculation unit 2100... Reverse lifting section 2110... Reverse color conversion unit 2120... Frame buffer

Claims

1. A point cloud decoder, In the Angular mode of Predictive coding, a tree synthesis unit is provided that performs different processing depending on the sensor type or operating mode. The tree synthesis unit is characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding, it performs prediction, residual decoding, and coordinate reconstruction processing for each of the coordinate values ​​r, θ, and φ after polar coordinate transformation.

2. A point cloud decoding device, In the Angular mode of Predictive coding, a tree synthesis unit is provided that performs different processing depending on the sensor type or operating mode. The tree synthesis unit is characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding, it performs prediction, residual decoding, and coordinate reconstruction processing for each of the coordinate values ​​r, θ, and φ after applying a coordinate transformation to θ and φ among the coordinate values ​​r, θ, and φ after polar coordinate transformation.

3. A point cloud decoding device, In the Angular mode of Predictive coding, a tree synthesis unit is provided that performs different processing depending on the sensor type or operating mode. The tree synthesis unit is characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding, it performs a prediction at one of the multiple points and treats it as a common predicted value with the other points.

4. A point cloud decoding device, In the Angular mode of Predictive coding, a tree synthesis unit is provided that performs different processing depending on the sensor type or operating mode. The tree synthesis unit is characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding, it selects a predictor in interpretation based on the coordinate values ​​θ and φ after polar coordinate transformation from the reference frame.

5. A point cloud decoding device, In the Angular mode of Predictive coding, a tree synthesis unit is provided that performs different processing depending on the sensor type or operating mode. The tree synthesis unit is characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operating mode indicates the activation of elevation angle component prediction and residual decoding, it selects a predictor from the reference frame based on the node ID in interpretation.

6. The point cloud decoding apparatus according to claim 4 or 5, wherein the tree synthesis unit, in the Angular mode, selects a predictor based on the branch ID of the prediction tree of the reference frame in interpretation when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding.

7. A point cloud decoding method, In the Angular mode of predictive coding, there is a process that performs different processing depending on the sensor type or operating mode. A point cloud decoding method characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding, prediction, residual decoding, and coordinate reconstruction processing are performed for each of the coordinate values ​​r, θ, and φ after polar coordinate transformation.

8. A program that makes a computer function as a point cloud decoder, The point cloud decoder includes a tree synthesis unit that performs different processing depending on the sensor type or operating mode in the Angular mode of predictive coding. The tree synthesis unit is a program characterized in that, in the Angular mode, when the sensor type indicates Non-Spinning LiDAR data or when the operation mode indicates the activation of elevation angle component prediction and residual decoding, it performs prediction, residual decoding, and coordinate reconstruction processing for each of the coordinate values ​​r, θ, and φ after polar coordinate transformation.

Images