Decoding method, decoding apparatus, and computer-readable recording medium

Abstract

The reference determination section determines a reference value corresponding to each of a plurality of points of the point cloud data. The residual decoding section decodes attribute residuals of the plurality of points from encoded residual data of the attribute residuals of the plurality of points with respect to the reference value. The attribute value calculation section calculates an attribute value of each of the plurality of points from the attribute residual and the reference value.

Description

TECHNICAL FIELD

[0001] The present application relates to a decoding method, a decoding apparatus, a decoding program, and a data structure of encoded point cloud data. BACKGROUND

[0002] As a method of encoding a moving image represented by point cloud data, there is proposed a combine frame coding (refer to Non-Patent Literature 1). The combine frame coding is a method of encoding after merging point clouds belonging to a plurality of frames. As an encoding method of point cloud data, there is known an octree. Since point clouds represented in consecutive frames in a moving image have high correlation with each other with respect to their positions, they have the same structure in upper levels (levels close to a path) of the octree. Therefore, by using the combine frame coding, it is possible to improve the encoding efficiency. Further, by assigning a frame index to each of the plurality of points merged, it is possible to extract point clouds by frame at the time of decoding.

[0003] PRIOR ART DOCUMENTS

[0004] NON-PATENT LITERATURE

[0005] Non-Patent Literature 1: N19525, ISO / IEC JTC 1 / SC 29 / WG 11 “Coding of moving pictures and audio”, October 2, 2020. SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] Further, attributes other than positions of points do not necessarily have high correlation between frames. Therefore, in the case of using the combine frame coding, it is possible to improve the encoding efficiency at least with respect to the positions of points, and on the other hand, it is possible that the encoding efficiency cannot be improved with respect to other attributes.

[0008] An object of the present application is to provide a decoding method, a decoding apparatus, a decoding program, and a data structure of encoded point cloud data, which are capable of decoding point clouds from data in which the encoding efficiency is improved with respect to attributes of point clouds.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] One embodiment of the present application is a decoding method of decoding point cloud data, the decoding method including: a step of determining a reference value corresponding to each of a plurality of points; a step of decoding attribute residuals of the plurality of points from encoded residual data that encodes attribute residuals of attribute values of the plurality of points with respect to the reference value; and a step of calculating, for each of the plurality of points, an attribute value of the point from the attribute residual of the point and the reference value corresponding to the point.

[0011] One embodiment of the present application is a decoding method including: a step of decoding, for each of a plurality of points included in merged point cloud data in which point clouds belonging to different frames are merged, a frame index and an attribute residual of an attribute value of each of the plurality of points from encoded attribute data that encodes attribute data including the frame index showing the frame to which the point belongs and the attribute residual of the attribute value with respect to a reference value; a step of acquiring reference data showing a correspondence relationship between the frame index and the reference value; a step of determining, for each of the plurality of points, the reference value corresponding to the decoded frame index from the reference data; and a step of calculating, for each of the plurality of points, the attribute value from the decoded attribute residual and the determined reference value.

[0012] One embodiment of the present application is a decoding device of decoding point cloud data, the decoding device including: a reference determination section that determines a reference value corresponding to each of a plurality of points; a residual decoding section that decodes attribute residuals of the plurality of points from encoded residual data that encodes attribute residuals of attribute values of the plurality of points with respect to the reference value; and an attribute value calculation section 56 that calculates, for each of the plurality of points, an attribute value of the point from the attribute residual of the point and the reference value corresponding to the point.

[0013] One embodiment of the present application is a decoding program for causing a computer to execute: a step of determining a reference value corresponding to each of a plurality of points from reference data showing a correspondence relationship between the plurality of points and the reference value; a step of decoding attribute residuals of the plurality of points from encoded residual data that encodes attribute residuals of attribute values of the plurality of points with respect to the reference value; and a step of calculating, for each of the plurality of points, an attribute value of the point from the attribute residual of the point and the reference value corresponding to the point.

[0014] One embodiment of the present application is a data structure that codes point group data, including: encoded residual data that is data that codes a residual of an attribute value of a plurality of points with respect to a reference value, the encoded residual data being used for processing by the computer to decode the attribute residual; and reference data that is data that shows a correspondence relationship of the plurality of points with the reference value, the reference data being used for processing by the computer to calculate the attribute value for each of the plurality of points from the attribute residual and the reference value.

[0015] Effects of Invention

[0016] According to the above-described embodiment, a point group can be decoded from data in which coding efficiency is improved with respect to attributes of the point group. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a diagram that shows the structure of an activity image processing system of a first embodiment;

[0018] Figure 2 is a diagram that shows the structure of an activity image processing system of a first embodiment;

[0019] Figure 3 is a diagram that shows an example of a data structure that codes point group data of the first embodiment;

[0020] Figure 4 is a diagram that shows an example of a data structure that codes point group data of the first embodiment;

[0021] Figure 5 is a diagram that shows the structure of an activity image processing system of a first embodiment;

[0022] Figure 6 is a diagram that shows an example of a data structure that codes point group data of the first embodiment;

[0023] Figure 7 is a diagram that shows the structure of an activity image processing system of a first embodiment; DETAILED DESCRIPTION

[0024] <First Embodiment>

[0025] Structure of Activity Image Processing System 1

[0026] Hereinafter, the embodiments will be described in detail with reference to the drawings. Figure 1

[0027] Figure 1 ​is a schematic diagram showing the structure of an active image processing system according to the first embodiment. The active image processing system 1 is provided with a point cloud generation device 10, an encoding device 30, and a decoding device 50.

[0028] The point cloud generation device 10 generates point cloud data representing the three-dimensional shape of an object present in an object space. The point cloud generation device 10 generates dynamic point cloud data by generating point cloud data every prescribed frame time. As an example of the point cloud generation device 10, a LiDAR (Light Detection and Ranging) device can be given. The point cloud generation device 10 determines the global position of each point measured using a global position determined by a GNSS (Global Navigation Satellite System) and a posture determined by an inertial sensor. In the point cloud data, data showing the determined position (Geometry data), color, time of measurement, attribute data (Attribute data) of various measurement values, and the like are included.

[0029] The encoding device 30 generates encoded point cloud data in which the dynamic point cloud data generated by the point cloud generation device 10 is encoded. In the dynamic point cloud data, point clouds represented in consecutive frames have high correlation with each other with respect to their positions. On the other hand, attributes other than the positions of points do not necessarily have high correlation between points at close positions between frames. Depending on the attributes, there are cases where close values or values having high correlation are taken at points at different positions in the same frame, points at different positions in different frames, and the like. For example, with respect to the time of measurement exemplified above, the likelihood that close values are taken between points at different positions in the same frame is higher than between points at close positions in different frames. Further, in a case where the point cloud generation device 10 periodically acquires points at the same position in a real space, it is possible that close values are taken between points at close positions in different frames, between points at different positions in different frames. The encoding device 30 of the first embodiment utilizes this property to improve the encoding efficiency of the dynamic point cloud data.

[0030] The decoding device 50 decodes the encoded point cloud data and reproduces the dynamic point cloud data. The encoding device 30 outputs the encoded point cloud data to the decoding device 50, for example, via a network.

[0031] Structure of the encoding device 30

[0032] Figure 2 is a schematic block diagram showing the structure of the encoding device 30 according to the first embodiment. The encoding device 30 is provided with a point cloud acquisition section 31, a merging section 32, a geometry encoding section 33, a reference determination section 34, a residual calculation section 35, an attribute encoding section 36, and an output section 37.

[0033] The point cloud acquisition unit 31 acquires dynamic point cloud data from the point cloud generation device 10. The merging unit 32 merges a plurality of frames that constitute the dynamic point cloud data. The geometry encoding unit 33 generates encoded geometry data by encoding the positions of a plurality of points included in the merged dynamic point cloud data. As an encoding method of point cloud data, octree encoding can be given.

[0034] The reference determination unit 34 determines a reference value involved in an attribute item for which an attribute residual should be encoded, among a plurality of attribute items of the merged dynamic point cloud data. The attribute item for which the attribute residual should be encoded is determined in advance. At least a frame index is not included in the attribute item for which the attribute residual should be encoded. If an attribute item whose value changes according to time, such as a distance from a laser irradiation point, a light receiving angle, and the like, is selected as the attribute item for which the attribute residual should be encoded, the encoding efficiency can be improved.

[0035] As examples of the attribute item for which the attribute residual should be encoded, for example, thermal image data, a three-dimensional position and posture of the point cloud generation device 10 at the time of scanning, a segmentation label, a parameter of particle simulation (for example, a velocity vector), a normal, a reflection intensity of laser, transparency, a material ID, and the like can be given. The thermal image data is affected by an external air temperature, and the value changes according to time. In a case where the point cloud generation device 10 is provided to a moving body such as a vehicle, the three-dimensional position and posture of the point cloud generation device 10 change according to time. The segmentation label is a label attached by semantic segmentation or the like, for example, and shows a category to which a point belongs. In a case where semantic segmentation is performed per frame, the segmentation label attached between frames can change.

[0036] The reference determination unit 34 determines the reference value in such a manner that a divergence of attribute values between frames becomes small. The reference determination unit 34, for example, calculates an average value of the attribute values of each frame as the reference value. The reference determination unit 34 generates reference data that associates a frame index of a frame, an attribute item, and a reference value. The reference determination unit 34 can also encode the reference data.

[0037] The residual calculation unit 35 calculates, for each of a plurality of points included in the merged dynamic point cloud data, a residual of an attribute value of an attribute item for which the reference value is determined from the reference value, that is, an attribute residual. The attribute encoding unit 36 generates encoded residual data by encoding the attribute residuals of a plurality of points included in the merged dynamic point cloud data. The attribute encoding unit 36 generates encoded attribute data by encoding attribute values involved in an attribute item for which the attribute residual of a plurality of points included in the merged dynamic point cloud data is not encoded. The attribute encoding unit 36 transformally encodes the attribute residual of each point. On the other hand, the attribute encoding unit 36 losslessly encodes the frame index of each point. The encoded attribute data involved in the frame index is also referred to as encoded index data.

[0038] The output unit 37 outputs the encoded point cloud data including the encoded geometry data, the encoded attribute data, the encoded residual data, and the reference data to the decoding apparatus 50. Figure 3 is a diagram illustrating an example of a data structure of the encoded point cloud data of the first embodiment. In Figure 3 In the example illustrated, the encoded point cloud data includes the reference data in the header and includes the encoded geometry data, the encoded attribute data, and the encoded residual data in the body. The encoded geometry data, the encoded attribute data, and the encoded residual data can also be respectively divided into a plurality of pieces to be stored. Further, in other embodiments, the reference data can also be output as an additional message.

[0039] Action of the encoding apparatus 30

[0040] Figure 4 is a flowchart illustrating an encoding method of dynamic point cloud data performed by the encoding apparatus 30 of the first embodiment. The point cloud acquisition unit 31 of the encoding apparatus 30 acquires the dynamic point cloud data from the point cloud generation apparatus 10 (step S1). The dynamic point cloud data acquired by the point cloud acquisition unit 31 can also constitute a part of the dynamic point cloud data having a large number of frames.

[0041] The merging unit 32 merges a plurality of frames constituting the dynamic point cloud data acquired in step S1 (step S2). The geometry encoding unit 33 generates the encoded geometry data with respect to positions of a plurality of points included in the merged dynamic point cloud data (step S3).

[0042] The reference determination unit 34 determines a reference value for each frame with respect to an attribute item for which an attribute residual is to be encoded, generates the reference data (step S4). The residual calculation unit 35 calculates the attribute residual from an attribute value of the attribute item for which the attribute residual is to be encoded based on the reference value determined in step S4 (step S5). The attribute encoding unit 36 generates the encoded residual data by encoding the attribute residual calculated in step S5 (step S6). In addition, the attribute encoding unit 36 generates the encoded attribute data by encoding the attribute value of the attribute item for which the attribute residual is not calculated (step S7). The attribute item for which the attribute residual is not calculated includes the frame index.

[0043] The output unit 37 outputs the encoded point cloud data including the encoded geometry data, the encoded attribute data, the encoded residual data, and the reference data to the decoding apparatus 50 (step S8).

[0044] As described above, the attribute items regarding the distance from the laser irradiation point, the light receiving angle, and the like vary according to time, and the values change greatly between frames. On the other hand, since the objects indicated by the points do not change between frames, the relative relationship between the points in the attribute items does not change greatly between frames. Therefore, the encoding device 30 can generate encoded point group data with good encoding efficiency by encoding the residual of the reference value and the attribute value for each frame.

[0045] Structure of Decoding Device 50

[0046] Figure 5 is a schematic block diagram illustrating the structure of the decoding device 50 of the first embodiment. The decoding device 50 includes an input section 51, a geometry decoding section 52, an attribute decoding section 53, a residual decoding section 54, a reference determination section 55, an attribute value calculation section 56, a decomposition section 57, and a display section 58.

[0047] The input section 51 receives input of the encoded point group data from the decoding device 50. The geometry decoding section 52 decodes the encoded geometry data included in the encoded point group data. The attribute decoding section 53 decodes the encoded attribute data included in the encoded point group data. Thereby, the attribute decoding section 53 acquires the attribute values of the points including the frame index.

[0048] The residual decoding section 54 decodes the encoded residual data included in the encoded point group data. Thereby, the residual decoding section 54 acquires the attribute residual of each point. The reference determination section 55 determines the reference value for each frame index from the reference data included in the encoded point group data. The attribute value calculation section 56 calculates the attribute of each point by adding the attribute residual of each point to the reference value. The decomposition section 57 reproduces the dynamic point group data by decomposing the plurality of points for each frame. The display section 58 displays the moving image in the display device based on the dynamic point group data.

[0049] Action of Decoding Device 50

[0050] Figure 6 is a flowchart illustrating a decoding method of the encoded point group data performed by the decoding device 50 of the first embodiment. The input section 51 receives input of the encoded point group data from the decoding device 50 (step S31). The geometry decoding section 52 decodes the encoded geometry data included in the encoded point group data input in step S31 (step S32). Thereby, the geometry decoding section 52 determines the positions of the plurality of points.

[0051] The attribute decoding section 53 decodes the encoded attribute data included in the encoded point group data input in step S31 (step S33). Thereby, the attribute decoding section 53 acquires the attribute values of the points including the frame index. The residual decoding section 54 decodes the encoded residual data included in the encoded point group data (step S34). Thereby, the residual decoding section 54 acquires the attribute residual of each point.

[0052] The decoding apparatus 50 selects the points represented by the encoded point group one by one (step S35), and performs the following processes of steps S36 to S38 for the selected points. The reference determining section 55 determines the frame index of the point determined in step S35, based on the attribute values of the plurality of points acquired in step S33 (step S36). The reference determining section 55 determines the reference value corresponding to the frame index determined in step S36, based on the reference data included in the encoded point group data input in step S31 (step S37). The attribute value calculating section 56 calculates the attribute value by adding the reference value determined in step S37 to the attribute residual of the point determined in step S35 (step S38).

[0053] When the decoding apparatus 50 calculates the attribute values of all the points, the decomposing section 57 reproduces the dynamic point group data by decomposing the plurality of points on a frame-by-frame basis (step S39). The display section 58 displays the moving image in the display apparatus based on the dynamic point group data (step S40). That is, the display section 58 outputs the reproduced dynamic point group data to the display apparatus.

[0054] Thus, the decoding apparatus 50 of the first embodiment determines the reference value corresponding to each of the plurality of points based on the reference data, and calculates the attribute value of each of the plurality of points based on the attribute residual of the plurality of points decoded from the encoded residual data and the reference value. Thereby, the decoding apparatus 50 can decode the encoded geometry data in which the attribute of the point group is improved in coding efficiency, and obtain the dynamic point group data.

[0055] <Other Embodiments>

[0056] The above-described embodiment has been described in detail with reference to the drawings, but the specific structure is not limited to the above-described structure, and various design changes and the like can be made. That is, in other embodiments, the order of the above-described processes can be appropriately changed. In addition, a part of the processes can be executed in parallel.

[0057] The encoding apparatus 30 and the decoding apparatus 50 of the above-described embodiment can each be constituted by a single computer, or the structure of the encoding apparatus 30 or the decoding apparatus 50 can be divided into a plurality of computers to be configured, and the plurality of computers can cooperate with each other to function as the encoding apparatus 30 or the decoding apparatus 50. In addition, in other embodiments, the encoding apparatus 30 and the decoding apparatus 50 can be constituted by the same computer.

[0058] The encoding point cloud data of the above-described embodiment has the reference data showing the relationship between the frame index and the reference value, but is not limited thereto. For example, the encoding point cloud data of another embodiment can have the reference value for a group different from the frame. For example, in another embodiment, the encoding point cloud data can have the reference value corresponding to the point cloud generation apparatus 10 for data obtained by merging the point cloud data generated by different point cloud generation apparatuses 10.

[0059] In addition, for example, the GPS time in the attribute of the point is sometimes reset in the scanning with the laser of the LiDAR, and sometimes has no correlation in the frame unit. For example, the reference data of the encoding point cloud data of another embodiment can represent a lookup table associating the group index with the reference value. In this case, the attribute residual represented by the encoding residual data represents the residual of the reference value and the attribute value corresponding to the group. In addition, the encoding attribute data represents the attribute value including the group index in addition to the frame index. Thereby, it is possible to have different reference values for the values within the frame.

[0060] In the case of the above-described modification, the encoded point cloud data is not necessarily dynamic point cloud data.

[0061] Furthermore, the encoding point cloud data of another embodiment can represent a pointer showing the point of the reference destination instead of the frame index, using the same reference value as the point shown by the pointer.

[0062] The point cloud generation apparatus 10 of the above-described embodiment is a LiDAR, but is not limited thereto in another embodiment. For example, the point cloud generation apparatus 10 of another embodiment can be another apparatus such as a stereo camera apparatus.

[0063] 〈Computer Structure〉

[0064] Figure 7 is a schematic block diagram showing the structure of a computer of at least one embodiment.

[0065] The computer 90 has a processor 91, a main memory 93, a storage device 95, and an interface 97.

[0066] The above-described encoding apparatus 30 and the decoding apparatus 50 are installed in the computer 90. Also, the operation of each processing section described above is stored in the storage device 95 in the form of a program. The processor 91 reads out the program from the storage device 95 and expands it in the main memory 93, and executes the above-described processing according to the program. In addition, the processor 91 secures a storage area corresponding to each storage section described above in the main memory 93 according to the program. As an example of the processor 91, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a microprocessor, or the like can be given.

[0067] A program can also be used to implement part of the functions exerted by the computer 90. For example, the program can exert the functions in combination with other programs already stored in the storage device, or in combination with other programs installed in other devices. Furthermore, in other embodiments, the computer 90 can be provided with a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above-described structure. As examples of the PLD, there are a PAL (Programmable Array Logic), a GAL (Generic Array Logic), a CPLD (Complex Programmable Logic Device), and an FPGA (Field Programmable Gate Array). In this case, part or all of the functions implemented by the processor 91 can be implemented by the integrated circuit. Such an integrated circuit is also included in an example of the processor.

[0068] As examples of the storage device 95, there are a magnetic disk, a magneto-optical disk, an optical disk, a semiconductor memory, and the like. The storage device 95 can be an internal medium directly connected to the bus of the computer 90, or an external medium connected to the computer 90 via the interface 97 or a communication line. Furthermore, in the case where the program is distributed to the computer 90 via a communication line, the computer 90 that has received the distribution can also expand the program into the main storage 93 and execute the above-described processing. In at least one embodiment, the storage device 95 is a non-transitory tangible storage medium.

[0069] Furthermore, the program can also be used to implement part of the above-described functions. Further, the program can also be a so-called differential file (differential program) that implements the above-described functions in combination with other programs already stored in the storage device 95.

[0070] Explanation of Symbols

[0071] 1 … moving image processing system 10 … point cloud generating device 30 … encoding device 31 … point cloud acquisition section 32 … merging section 33 … geometry encoding section 34 … reference deciding section 35 … residual calculation section 36 … attribute encoding section 37 … output section 50 … decoding device 51 … input section 52 … geometry decoding section 53 … attribute decoding section 54 … residual decoding section 55 … reference deciding section 56 … attribute value calculation section 57 … decomposing section 58 … display section 90 … computer 91 … processor 93 … main storage 95 … storage device 97 … interface

Claims

1. A decoding method of decoding merged point cloud data that merges point clouds belonging to different frames, the decoding method having: a step of determining a reference value determined for each frame; a step of decoding attribute residuals of a plurality of points of the merged point cloud data from encoded residual data that encodes attribute residuals of the plurality of points, the attribute residuals being residuals of attribute values of the plurality of points with respect to the reference value; and a step of calculating, for each of the plurality of points, an attribute value of the point from the reference value corresponding to the frame to which the point belongs and the attribute residual of the point.

2. The decoding method according to claim 1, wherein having a step of decoding, for each of the plurality of points, a frame to which the point belongs from encoded attribute data that encodes a value showing the frame, in the step of determining the reference value, the reference value corresponding to the decoded frame is determined for each of the plurality of points.

3. A decoding method comprising: a step of decoding, for each of a plurality of points included in merged point cloud data that merges point clouds belonging to different frames, a frame index showing the frame to which the point belongs and an attribute residual of the point from encoded index data that encodes the frame index and encoded residual data that encodes the attribute residual, the attribute residual being a residual of an attribute value with respect to a reference value; a step of acquiring reference data showing a correspondence relationship of the frame index with the reference value; a step of determining, for each of the plurality of points, the reference value corresponding to the decoded frame index from the reference data; and a step of calculating, for each of the plurality of points, the attribute value from the decoded attribute residual and the determined reference value.

4. A decoding apparatus of decoding merged point cloud data that merges point clouds belonging to different frames, the decoding apparatus having: a reference determination section that determines a reference value determined for each frame; a residual decoding section that decodes attribute residuals of a plurality of points of the merged point cloud data from encoded residual data that encodes attribute residuals of the plurality of points, the attribute residuals being residuals of attribute values of the plurality of points with respect to the reference value; and an attribute value calculation section that calculates, for each of the plurality of points, an attribute value of the point from the reference value corresponding to the frame to which the point belongs and the attribute residual of the point.

5. A computer-readable recording medium that records a decoding program for causing a computer to decode merged point cloud data that merges point clouds belonging to different frames to execute the following steps: a step of determining a reference value determined for each frame from reference data showing a correspondence relationship of a plurality of points with a reference value; a step of decoding attribute residuals of a plurality of points of the merged point cloud data from encoded residual data that encodes attribute residuals of the plurality of points, the attribute residuals being residuals of attribute values of the plurality of points with respect to the reference value; and a step of calculating, for each of the plurality of points, an attribute value of the point from the reference value corresponding to the frame to which the point belongs and the attribute residual of the point.

6. A computer-readable recording medium that records encoded point cloud data including: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ encoded residual data that is data in which an attribute residual is encoded, the attribute residual being a residual of an attribute value of a plurality of points of merged point group data in which point groups belonging to different frames are merged, with respect to a reference value, the encoded residual data being used for a process of decoding, by a computer, the attribute residual; and and reference data that is data showing a correspondence relationship of the plurality of points with reference values, the reference data being used for a process of calculating, by the computer, the attribute value for each of the plurality of points from the reference value corresponding to the frame to which the point belongs and the attribute residual.

Images