Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

A three-dimensional data encoding method assigns a plurality of three-dimensional points included in point cloud data to a certain level among one or more levels based on a plurality of position information of the plurality of three-dimensional points (S11611), permits inter-level reference in a case where the number of the one or more levels is one, generates encoded attribute information by encoding a plurality of attribute information of the plurality of three-dimensional points, the inter-level reference generates a prediction value of attribute information of a three-dimensional point of an object of processing with reference to attribute information of another three-dimensional point included in the same level as the three-dimensional point of the object of processing (S11612), and generates a bitstream including the encoded attribute information (S11613).

Description

Technical Field

[0001] This disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Technology

[0002] In major fields such as computer vision, mapping, surveillance, infrastructure inspection, and image distribution for autonomous operation of automobiles or robots, devices and services that flexibly utilize 3D data will become increasingly common in the future. 3D data is acquired through various methods, including distance sensors such as rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0003] One method for representing 3D data is point cloud representation, which uses groups of points in 3D space to represent the shape of a 3D structure. The point cloud stores the position and color of these points. While point clouds are expected to become the mainstream method for representing 3D data, the data volume of point groups is enormous. Therefore, in the accumulation or transmission of 3D data, similar to 2D moving images (for example, MPEG-4 AVC or HEVC standardized using MPEG), data compression through encoding is necessary.

[0004] In addition, point cloud compression is partly supported by publicly available libraries (such as the PointCloud Library) that perform point cloud association processing.

[0005] Furthermore, there are known technologies that utilize three-dimensional map data to retrieve and display facilities located around a vehicle (for example, see Patent Document 1).

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent Document 1: International Publication No. 2014 / 020663 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] In the encoding and decoding of 3D data, it is desirable to improve encoding efficiency.

[0011] The purpose of this disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

[0012] Methods used to solve problems

[0013] The present disclosure discloses a three-dimensional data encoding method, which, based on multiple positional information of multiple three-dimensional points, assigns multiple three-dimensional points contained in point group data to a certain level in one or more levels. When the number of the one or more levels is 1, same-level reference is permitted. Encoded attribute information is generated by encoding multiple attribute information of the multiple three-dimensional points. The same-level reference refers to the attribute information of other three-dimensional points contained in the same level of the three-dimensional points of the processing object to generate predicted values ​​of the attribute information of the three-dimensional points of the processing object, and generates a bit stream containing the encoded attribute information.

[0014] The three-dimensional data decoding method disclosed herein obtains encoded attribute information from a bitstream. The encoded attribute information is information obtained by encoding multiple attribute information of multiple three-dimensional points contained in point group data. The multiple three-dimensional points are assigned to one or more levels based on multiple position information of the multiple three-dimensional points. When the number of the one or more levels is 1, same-level reference is allowed to decode the encoded attribute information. The same-level reference refers to the attribute information of other three-dimensional points contained in the same level of the three-dimensional points of the processing object to generate predicted values ​​of the attribute information of the three-dimensional points of the processing object.

[0015] Invention Effects

[0016] This disclosure provides a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. Attached Figure Description

[0017] Figure 1 This is a diagram showing the structure of the three-dimensional data encoding and decoding system of Implementation Method 1.

[0018] Figure 2 This is a diagram illustrating a structural example of point group data in Implementation Method 1.

[0019] Figure 3 This is a diagram illustrating an example of the structure of a data file describing point group data information in Implementation 1.

[0020] Figure 4 This is a diagram representing the types of point group data in Implementation Method 1.

[0021] Figure 5 This is a diagram showing the structure of the first coding section in Embodiment 1.

[0022] Figure 6 This is a block diagram of the first coding section of Embodiment 1.

[0023] Figure 7 This is a diagram showing the structure of the first decoding unit in Embodiment 1.

[0024] Figure 8 This is a block diagram of the first decoding unit in Embodiment 1.

[0025] Figure 9 This is a block diagram of the three-dimensional data encoding device according to Embodiment 1.

[0026] Figure 10 This is a diagram illustrating an example of location information in Implementation Method 1.

[0027] Figure 11 This is a diagram illustrating an example of an octree representation of the location information in Implementation 1.

[0028] Figure 12 This is a block diagram of the three-dimensional data decoding device according to Embodiment 1.

[0029] Figure 13 This is a block diagram of the attribute information encoding unit in Implementation Method 1.

[0030] Figure 14 This is a block diagram of the attribute information decoding unit in Implementation Method 1.

[0031] Figure 15 This is a block diagram showing the structure of the attribute information encoding unit in Implementation Method 1.

[0032] Figure 16 This is a block diagram of the attribute information encoding unit in Implementation Method 1.

[0033] Figure 17 This is a block diagram showing the structure of the attribute information decoding unit in Embodiment 1.

[0034] Figure 18 This is a block diagram of the attribute information decoding unit in Implementation Method 1.

[0035] Figure 19 This is a diagram showing the structure of the second coding section in Embodiment 1.

[0036] Figure 20 This is a block diagram of the second coding section of Embodiment 1.

[0037] Figure 21 This is a diagram showing the structure of the second decoding unit in Embodiment 1.

[0038] Figure 22 This is a block diagram of the second decoding unit in Embodiment 1.

[0039] Figure 23 This is a diagram showing the protocol stack for PCC encoded data in Implementation 1.

[0040] Figure 24 This is a diagram showing the structure of the encoding section and the multiplexing section in Embodiment 2.

[0041] Figure 25 This is a diagram illustrating an example of the structure of the encoded data in Implementation Method 2.

[0042] Figure 26 This is a diagram illustrating an example of the structure of the encoded data and NAL unit in Implementation Method 2.

[0043] Figure 27 This is a diagram representing a semantic example of pcc_nal_unit_type in implementation method 2.

[0044] Figure 28 This is a diagram illustrating an example of the output sequence of the NAL unit in Implementation Method 2.

[0045] Figure 29 This is a flowchart of the process performed by the three-dimensional data encoding device in Implementation Method 2.

[0046] Figure 30 This is a flowchart of the processing performed by the three-dimensional data decoding device in Implementation Method 2.

[0047] Figure 31 This is a flowchart of the reuse process in Implementation Method 2.

[0048] Figure 32 This is a flowchart of the inverse reuse process in Implementation Method 2.

[0049] Figure 33 This is a diagram showing an example of three-dimensional points in Implementation Method 3.

[0050] Figure 34 This is a diagram showing an example of the LoD configuration in implementation method 3.

[0051] Figure 35 This is a diagram illustrating an example of the threshold used in setting the LoD in Implementation 3.

[0052] Figure 36 This is a diagram illustrating an example of the attribute information used in the predicted values ​​of Implementation Method 3.

[0053] Figure 37 This is a diagram illustrating an example of the exponential Golomb code in implementation method 3.

[0054] Figure 38 This is a diagram illustrating the processing of exponential Golomb codes in Implementation Method 3.

[0055] Figure 39 This is a diagram showing the syntax example of the attribute header in implementation method 3.

[0056] Figure 40 This is a diagram representing the syntax example of the attribute data in Implementation Method 3.

[0057] Figure 41 This is a flowchart of the three-dimensional data encoding process in Implementation Method 3.

[0058] Figure 42 This is a flowchart of the attribute information encoding process in Implementation Method 3.

[0059] Figure 43 This is a diagram illustrating the processing of exponential Golomb codes in Implementation Method 3.

[0060] Figure 44 This is a diagram illustrating an example of a reverse-dated table representing the relationship between the remaining code and its value in Implementation Method 3.

[0061] Figure 45 This is a flowchart of the three-dimensional data decoding process in Implementation Method 3.

[0062] Figure 46 This is a flowchart of the attribute information decoding process in Implementation Method 3.

[0063] Figure 47 This is a block diagram of the three-dimensional data encoding device according to Embodiment 3.

[0064] Figure 48 This is a block diagram of the three-dimensional data decoding device according to Embodiment 3.

[0065] Figure 49 This is a diagram illustrating an example of the reference relationship in implementation method 4.

[0066] Figure 50 This is a diagram illustrating an example of the reference relationship in implementation method 4.

[0067] Figure 51 This is a diagram illustrating an example of setting the number of searches for each LoD in implementation method 4.

[0068] Figure 52 This is a diagram illustrating an example of the reference relationship in implementation method 4.

[0069] Figure 53 This is a diagram illustrating an example of the reference relationship in implementation method 4.

[0070] Figure 54 This is a diagram illustrating an example of the reference relationship in implementation method 4.

[0071] Figure 55 This is a diagram representing the syntax example of the attribute information header in implementation method 4.

[0072] Figure 56 This is a diagram representing the syntax example of the attribute information header in implementation method 4.

[0073] Figure 57 This is a flowchart of the three-dimensional data encoding process in Implementation Method 4.

[0074] Figure 58 This is a flowchart of the attribute information encoding process in Implementation Method 4.

[0075] Figure 59 This is a flowchart of the three-dimensional data decoding process in Implementation Method 4.

[0076] Figure 60 This is a flowchart of the attribute information decoding process in Implementation Method 4.

[0077] Figure 61 This is a flowchart of the surrounding point search process in Implementation Method 4.

[0078] Figure 62 This is a flowchart of the surrounding point search process in Implementation Method 4.

[0079] Figure 63 This is a flowchart of the surrounding point search process in Implementation Method 4.

[0080] Figure 64 This is a diagram showing the reference relationship of implementation method 5.

[0081] Figure 65 This is a diagram showing the reference relationship of implementation method 5.

[0082] Figure 66 This is a diagram showing the reference relationship of implementation method 5.

[0083] Figure 67 This is a diagram showing the syntax example of the attribute information header in implementation method 5.

[0084] Figure 68 This is a diagram showing the syntax example of the attribute information header in implementation method 5.

[0085] Figure 69 This is a flowchart of the three-dimensional data encoding process in Implementation Method 5.

[0086] Figure 70 This is a flowchart of the attribute information encoding process in Implementation Method 5.

[0087] Figure 71 This is a flowchart of the three-dimensional data decoding process in Implementation Method 5.

[0088] Figure 72 This is a flowchart of the attribute information decoding process in Implementation Method 5.

[0089] Figure 73 This is a flowchart of the surrounding point search process in Implementation Method 5.

[0090] Figure 74 This is a diagram illustrating an example of the reference relationship in implementation method 6.

[0091] Figure 75 This is a diagram illustrating an example of the reference relationship in implementation method 6.

[0092] Figure 76 This is a diagram showing the syntax example of the attribute information header in implementation method 6.

[0093] Figure 77 This is a diagram showing the syntax example of the attribute information header in implementation method 6.

[0094] Figure 78 This is a diagram illustrating an example of the reference relationship in implementation method 6.

[0095] Figure 79 This is a flowchart of the surrounding point search process in Implementation Method 6.

[0096] Figure 80 This is a flowchart of the three-dimensional data encoding process in Implementation Method 6.

[0097] Figure 81 This is a flowchart of the three-dimensional data decoding process in Implementation Method 6.

[0098] Figure 82 This is a block diagram of the three-dimensional data production apparatus according to Embodiment 7.

[0099] Figure 83 This is a flowchart of the three-dimensional data creation method of implementation method 7.

[0100] Figure 84 This is a diagram showing the structure of the system in Implementation Method 7.

[0101] Figure 85 This is a block diagram of the client device in Implementation Method 7.

[0102] Figure 86 This is a block diagram of the server in implementation method 7.

[0103] Figure 87 This is a flowchart of the three-dimensional data creation and processing of the client device in Implementation Method 7.

[0104] Figure 88 This is a flowchart of the sensor information transmission process of the client device in Implementation Method 7.

[0105] Figure 89 This is a flowchart of the three-dimensional data creation and processing of the server in Implementation Method 7.

[0106] Figure 90 This is a flowchart of the three-dimensional map sending process of the server in Implementation Method 7.

[0107] Figure 91 This is a diagram showing the structure of a modified example of the system according to embodiment 7.

[0108] Figure 92 This is a diagram showing the structure of the server and client device in Implementation Method 7.

[0109] Figure 93 This is a diagram showing the structure of the server and client device in Implementation Method 7.

[0110] Figure 94 This is a flowchart of the processing of the client device in Implementation 7.

[0111] Figure 95 This is a diagram showing the structure of the sensor information collection system according to embodiment 7.

[0112] Figure 96 This is a diagram illustrating an example of the system in Implementation Method 7.

[0113] Figure 97 This is a diagram showing a variation of the system according to embodiment 7.

[0114] Figure 98 This is a flowchart illustrating an example of the application processing in Implementation Method 7.

[0115] Figure 99 This is a diagram showing the sensor range of various sensors in Embodiment 7.

[0116] Figure 100 This is a diagram illustrating a structural example of the autonomous driving system according to embodiment 7.

[0117] Figure 101 This is a diagram illustrating a structural example of the bitstream in Implementation Method 7.

[0118] Figure 102 This is a flowchart of the point group selection process in Implementation Method 7.

[0119] Figure 103 This is a diagram illustrating a screen example of the dot group selection process in Implementation Method 7.

[0120] Figure 104 This is a diagram illustrating a screen example of the dot group selection process in Implementation Method 7.

[0121] Figure 105 This is a diagram illustrating a screen example of the dot group selection process in Implementation Method 7. Detailed Implementation

[0122] The present disclosure discloses a three-dimensional data encoding method, which, based on multiple positional information of multiple three-dimensional points, assigns multiple three-dimensional points contained in point group data to a certain level in one or more levels. When the number of the one or more levels is 1, same-level reference is permitted. Encoded attribute information is generated by encoding multiple attribute information of the multiple three-dimensional points. The same-level reference refers to the attribute information of other three-dimensional points contained in the same level of the three-dimensional points of the processing object to generate predicted values ​​of the attribute information of the three-dimensional points of the processing object, and generates a bit stream containing the encoded attribute information.

[0123] Therefore, this three-dimensional data encoding method can improve encoding efficiency by allowing same-layer references when the number of layers is 1.

[0124] For example, if the number of the one or more levels is two or more, the levels that allow the same-level reference and the levels that prohibit the same-level reference are determined among the two or more levels. Based on the determination result, the same-level reference is allowed or prohibited and the multiple attribute information is encoded to generate the encoded attribute information.

[0125] For example, it is also possible to allow the same-level references for the levels from the topmost level to the Nth level (N is a natural number) in the two or more levels, and to prohibit the same-level references for the levels below the Nth level, thereby encoding the multiple attribute information to generate the encoded attribute information.

[0126] For example, the bitstream may contain first information, which indicates a level that permits or prohibits the same-layer reference.

[0127] For example, the bitstream may also contain second information indicating the number of the one or more levels.

[0128] The three-dimensional data decoding method disclosed herein obtains encoded attribute information from a bitstream. The encoded attribute information is information obtained by encoding multiple attribute information of multiple three-dimensional points contained in point group data. The multiple three-dimensional points are assigned to one or more levels based on multiple position information of the multiple three-dimensional points. When the number of the one or more levels is 1, same-level reference is allowed to decode the encoded attribute information. The same-level reference refers to the attribute information of other three-dimensional points contained in the same level of the three-dimensional points of the processing object to generate predicted values ​​of the attribute information of the three-dimensional points of the processing object.

[0129] Therefore, this 3D data decoding method can improve encoding efficiency by allowing same-layer references when the number of layers is 1.

[0130] For example, if the number of the above-mentioned levels is two or more, the levels that allow the same-level reference and the levels that prohibit the same-level reference are determined among the two or more levels. Based on the determination result, the same-level reference is allowed or prohibited and the multiple attribute information is decoded.

[0131] For example, it is also possible to allow the same-level reference for the levels from the topmost level to the Nth level (N is a natural number) in the two or more levels, and to prohibit the same-level reference for the levels below the Nth level, thereby decoding the encoded attribute information.

[0132] For example, it is also possible to obtain first information from the bit stream, the first information indicating the level that allows or prohibits the same-layer reference.

[0133] For example, it is also possible to obtain second information representing the number of the more than one level from the bit stream.

[0134] For example, it is also possible to obtain (1) first information indicating whether the same-layer reference is permitted or prohibited, and (2) second information indicating the number of the more than one level from the bitstream. If the number of the more than one level indicated by the second information is 1, and if the first information does not indicate that the same-layer reference is permitted in one of the more than one levels, the bitstream is determined to be non-compliant with the standard.

[0135] In addition, a three-dimensional data encoding device of one technical solution disclosed herein includes a processor and a memory. The processor uses the memory to assign multiple three-dimensional points contained in the point group data to a certain level in one or more levels based on multiple position information of the multiple three-dimensional points. When the number of the one or more levels is 1, same-level reference is allowed. Encoded attribute information is generated by encoding multiple attribute information of the multiple three-dimensional points. The same-level reference refers to the attribute information of other three-dimensional points contained in the same level of the three-dimensional points of the processing object to generate predicted values ​​of the attribute information of the three-dimensional points of the processing object, and generates a bit stream containing the encoded attribute information.

[0136] Therefore, this three-dimensional data encoding device can improve encoding efficiency by allowing same-layer references when the number of layers is 1.

[0137] In addition, a three-dimensional data decoding device of one technical solution disclosed herein includes a processor and a memory. The processor uses the memory to obtain encoded attribute information from a bit stream. The encoded attribute information is information after encoding multiple attribute information of multiple three-dimensional points contained in point group data. The multiple three-dimensional points are assigned to one or more levels based on multiple position information of the multiple three-dimensional points. When the number of the one or more levels is 1, same-level reference is allowed to decode the encoded attribute information. The same-level reference refers to the attribute information of other three-dimensional points contained in the same level of the three-dimensional points of the processing object to generate predicted values ​​of the attribute information of the three-dimensional points of the processing object.

[0138] Therefore, this 3D data decoding device can improve encoding efficiency by allowing same-layer references when the number of layers is 1.

[0139] In addition, these inclusive or specific technical solutions can also be implemented by systems, methods, integrated circuits, computer programs or computer-readable CD-ROMs and other recording media, or by any combination of systems, methods, integrated circuits, computer programs and recording media.

[0140] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below represent specific examples of this disclosure. The numerical values, shapes, materials, constituent elements, arrangement and connection methods of constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, any constituent elements in the following embodiments that are not described in the independent claims representing the highest-level concept will be described as arbitrary constituent elements.

[0141] (Implementation Method 1)

[0142] When using encoded point cloud data in practical devices or services, it is desirable to transmit and receive the required information according to the intended purpose in order to conserve network bandwidth. However, current encoding structures for 3D data do not possess this functionality, and therefore, there is no corresponding encoding method.

[0143] This embodiment describes a three-dimensional data encoding method and a three-dimensional data encoding apparatus that provide the function of sending and receiving required information in the encoded data of a three-dimensional point cloud according to its purpose, a three-dimensional data decoding method and a three-dimensional data decoding apparatus for decoding the encoded data, a three-dimensional data multiplexing method for multiplexing the encoded data, and a three-dimensional data transmission method for transmitting the encoded data.

[0144] In particular, while the first and second encoding methods have been studied as encoding methods for point group data, no method has been defined for the composition of the encoded data or for saving the encoded data to a system format. In this case, there are issues such as the inability to perform MUX processing (multiplexing) in the encoding department, or for transmission or storage.

[0145] Furthermore, there is currently no method that supports formats that use a combination of two codecs, such as PCC (Point Cloud Compression).

[0146] In this embodiment, the structure of PCC encoded data in which two codecs, the first encoding method and the second encoding method, coexist, and the method for saving the encoded data to the system format will be described.

[0147] First, the structure of the three-dimensional data (point group data) encoding and decoding system of this embodiment will be explained. Figure 1 This is a diagram illustrating a structural example of the three-dimensional data encoding and decoding system of this embodiment. For example... Figure 1 As shown, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0148] The 3D data encoding system 4601 generates encoded data or reused data by encoding point group data, which is 3D data. Furthermore, the 3D data encoding system 4601 can be a 3D data encoding device implemented by a single device, or a system implemented by multiple devices. Additionally, the 3D data encoding device may be included as part of the multiple processing units included in the 3D data encoding system 4601.

[0149] The three-dimensional data encoding system 4601 includes a point group data generation system 4611, a prompting unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point group data generation system 4611 includes a sensor information acquisition unit 4617 and a point group data generation unit 4618.

[0150] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point group data generation unit 4618. The point group data generation unit 4618 generates point group data based on the sensor information and outputs the point group data to the encoding unit 4613.

[0151] The prompting unit 4612 prompts the user with sensor information or point group data. For example, the prompting unit 4612 displays information or images based on the sensor information or point group data.

[0152] The encoding unit 4613 encodes (compresses) the point group data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 4614. The additional information may include, for example, sensor information.

[0153] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data, control information, and additional information input from the encoding unit 4613. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0154] The input / output unit 4615 (e.g., a communication unit or interface) outputs multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as internal memory. The control unit 4616 (or application execution unit) controls each processing unit. That is, the control unit 4616 performs control such as encoding and multiplexing.

[0155] In addition, sensor information can also be input to the encoding unit 4613 or the multiplexing unit 4614. Furthermore, the input / output unit 4615 can also directly output point group data or encoded data to the outside.

[0156] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0157] The 3D data decoding system 4602 generates point group data as 3D data by decoding encoded data or multiplexed data. Furthermore, the 3D data decoding system 4602 can be a 3D data decoding device implemented by a single device, or a system implemented by multiple devices. Additionally, the 3D data decoding device may include a portion of the multiple processing units contained in the 3D data decoding system 4602.

[0158] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621, an input / output unit 4622, an inverse multiplexing unit 4623, a decoding unit 4624, a prompting unit 4625, a user interface 4626, and a control unit 4627.

[0159] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603.

[0160] The input / output unit 4622 acquires the transmission signal, decodes the multiplexed data (file format or packet) according to the transmission signal, and outputs the multiplexed data to the inverse multiplexing unit 4623.

[0161] The reverse multiplexing unit 4623 obtains encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 4624.

[0162] The decoding unit 4624 reconstructs the point group data by decoding the encoded data.

[0163] The prompting unit 4625 prompts the user with the dot group data. For example, the prompting unit 4625 displays information or images based on the dot group data. The user interface 4626 obtains instructions based on the user's operation. The control unit 4627 (or application execution unit) controls each processing unit. That is, the control unit 4627 performs control over inverse multiplexing, decoding, and prompting, etc.

[0164] Furthermore, the input / output unit 4622 can also directly acquire point group data or coded data from external sources. Additionally, the prompting unit 4625 can acquire additional information such as sensor information and provide prompts based on this additional information. Furthermore, the prompting unit 4625 can also provide prompts based on user instructions obtained from the user interface 4626.

[0165] Sensor terminal 4603 generates information obtained by the sensor, namely sensor information. Sensor terminal 4603 is a terminal equipped with a sensor or camera, such as a moving object like a car, a flying object like an airplane, a mobile terminal, or a camera.

[0166] Sensor information that can be obtained by sensor terminal 4603 includes, for example, (1) the distance between sensor terminal 4603 and the object, or the reflectivity of the object, obtained from LiDAR, millimeter-wave radar, or infrared sensors; and (2) the distance between the camera and the object, or the reflectivity of the object, obtained from multiple monocular camera images or stereo camera images. Additionally, sensor information may also include the sensor's posture, orientation, rotation (angular velocity), position (GPS information or altitude), velocity, or acceleration. Furthermore, sensor information may also include temperature, air pressure, humidity, or magnetism.

[0167] The external connection unit 4604 is implemented through an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, or broadcasting.

[0168] Next, the point group data will be explained. Figure 2 It is a graph representing the structure of point group data. Figure 3 This is a diagram illustrating the structure of a data file that records information about point group data.

[0169] Point cluster data contains data from multiple points. Each point's data includes location information (3D coordinates) and attribute information relative to that location. A group of such points is called a point cluster. For example, a point cluster represents the 3D shape of an object.

[0170] Sometimes, positional information such as three-dimensional coordinates is also referred to as geometry. Additionally, the data for each point can also contain attribute information across multiple attribute categories. Attribute categories could include, for example, color or reflectivity.

[0171] You can create a mapping between a single attribute and a single location, or you can create a mapping between multiple attribute categories and a single location. Alternatively, you can create a mapping between multiple attribute categories and a single location.

[0172] Figure 3 The data file structure shown is an example of a one-to-one correspondence between location information and attribute information, representing the location information and attribute information of N points that constitute the point group data.

[0173] Positional information includes, for example, information along the x, y, and z axes. Attribute information includes, for example, RGB color information. Representative data files include .ply files, etc.

[0174] Next, the types of point group data will be explained. Figure 4 This is a graph representing the types of point group data. For example... Figure 4 As shown, point cluster data includes static objects and dynamic objects.

[0175] A static object is a group of three-dimensional points at any given time (a single moment). A dynamic object is a group of three-dimensional points that changes over time. Hereinafter, the three-dimensional point group data at a given moment will be referred to as a PCC frame or a frame.

[0176] The objects can be a cluster of points in a region that is restricted to some extent, like typical image data, or a large cluster of points in a region that is not restricted, like map information.

[0177] In addition, point cluster data of various densities exist, including both sparse and dense point cluster data.

[0178] The following is a detailed description of each processing unit. Sensor information is acquired through various methods, such as LiDAR or rangefinder sensors, stereo cameras, or combinations of multiple monocular cameras. The point group data generation unit 4618 generates point group data based on the sensor information obtained by the sensor information acquisition unit 4617. The point group data generation unit 4618 generates location information as point group data and adds attribute information specific to that location information to the location information.

[0179] The point group data generation unit 4618 can also process the point group data when generating location information or additional attribute information. For example, the point group data generation unit 4618 can reduce the amount of data by deleting point groups with duplicate locations. In addition, the point group data generation unit 4618 can also transform the location information (position transformation, rotation, or normalization, etc.) and render the attribute information.

[0180] In addition, Figure 1 In this system, the point group data generation system 4611 is included in the three-dimensional data encoding system 4601, but it can also be set up independently outside the three-dimensional data encoding system 4601.

[0181] The encoding unit 4613 encodes the point group data based on a pre-defined encoding method, thereby generating encoded data. There are roughly two types of encoding methods. The first type uses location information and will be referred to hereafter as the first encoding method. The second type uses a video codec and will be referred to hereafter as the second encoding method.

[0182] The decoding unit 4624 decodes the encoded data based on a predefined encoding method, thereby decoding the point group data.

[0183] The multiplexing unit 4614 generates multiplexed data by multiplexing encoded data using existing multiplexing methods. The generated multiplexed data is transmitted or stored. In addition to PCC encoded data, the multiplexing unit 4614 also multiplexes other media such as images, sounds, subtitles, applications, and files, or reference time information. Furthermore, the multiplexing unit 4614 can also multiplex attribute information associated with sensor information or point group data.

[0184] As multiplexing methods or file formats, there are ISOBMFF, MPEG-DASH (based on ISOBMFF), MMT, MPEG-2TS Systems, RMP, etc.

[0185] The reverse multiplexing unit 4623 extracts PCC encoded data, other media, and time information from the multiplexed data.

[0186] The input / output unit 4615 transmits multiplexed data using a method consistent with transmission or storage media such as broadcasting or communication media. The input / output unit 4615 can communicate with other devices via the Internet, and can also communicate with storage units such as cloud servers.

[0187] As a communication protocol, it can use HTTP, FTP, TCP, or UDP, etc. It can use either PULL or PUSH communication methods.

[0188] Either wired or wireless transmission can be used. For wired transmission, Ethernet, USB, RS-232C, HDMI, or coaxial cable can be used. For wireless transmission, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave can be used.

[0189] In addition, broadcast methods can be used, such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3.

[0190] Figure 5 This is a diagram showing the structure of the first coding section 4630, which represents an example of the coding section 4613 performing the first coding method. Figure 6 This is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point group data using a first encoding method. The first encoding unit 4630 includes a location information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0191] The first encoding unit 4630 is characterized by encoding based on the three-dimensional structure. Furthermore, the first encoding unit 4630 is characterized by an attribute information encoding unit 4632 encoding information obtained from the position information encoding unit 4631. This first encoding method is also known as GPCC (Geometry-based PCC).

[0192] The point cluster data is PCC point cluster data like a PLY file, or PCC point cluster data generated based on sensor information, which includes position information, attribute information, and other supplementary information (MetaData). The position information is input to the position information encoding unit 4631, the attribute information is input to the attribute information encoding unit 4632, and the supplementary information is input to the supplementary information encoding unit 4633.

[0193] The location information encoding unit 4631 generates compressed geometry as encoded data by encoding the location information. For example, the location information encoding unit 4631 uses an N-ary tree structure such as an octree to encode the location information. Specifically, in the octree, the object space is divided into 8 nodes (subspaces), and 8 bits of information (occupancy code) indicating whether each node contains a point group are generated. In addition, nodes containing point groups are further divided into 8 nodes, and 8 bits of information indicating whether each of the 8 nodes contains a point group are generated. This process is repeated until a predetermined level or a threshold for the number of point groups contained in a node is reached.

[0194] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) as encoded data by encoding the composition information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines the reference point (reference node) referenced in the encoding of the object point (object node) of the processed object based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 refers to nodes in the octree of surrounding nodes or adjacent nodes whose parent node is the same as the parent node of the object node. However, the method for determining the reference relationship is not limited to this.

[0195] Furthermore, the encoding process for attribute information can include at least one of quantization, prediction, and arithmetic encoding. In this case, a reference refers to the use of a reference node in the calculation of the predicted value of the attribute information, or the use of the state of the reference node (e.g., indicating whether the reference node contains occupancy information of a point group). For example, the encoded parameter is the quantization parameter in quantization or the context in arithmetic encoding.

[0196] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) as encoded data by encoding the compressible data in the additional information.

[0197] The multiplexing unit 4634 generates a compressed stream as encoded data by multiplexing encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated compressed stream is output to the processing unit of the system layer (not shown).

[0198] Next, the first decoding unit 4640, which is an example of the decoding unit 4624 performing the first encoding method, will be described. Figure 7 This is a diagram showing the structure of the first decoding unit 4640. Figure 8 This is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point group data by decoding the encoded data (encoded stream) encoded using the first encoding method. The first decoding unit 4640 includes an inverse multiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0199] The processing unit of the system layer (not shown) inputs the compressed stream of encoded data to the first decoding unit 4640.

[0200] The reverse multiplexing unit 4641 separates the compressed geometry, compressed attribute, compressed metadata and other additional information from the encoded data.

[0201] The location information decoding unit 4642 generates location information by decoding the encoded location information. For example, the location information decoding unit 4642 reconstructs the location information of a group of points represented by three-dimensional coordinates based on the encoded location information represented by an N-ary tree structure such as an octree.

[0202] The attribute information decoding unit 4643 decodes the encoded attribute information based on the composition information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines the reference point (reference node) to be referenced in the decoding of the object point (object node) of the processed object based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 refers to a node in the octree of surrounding nodes or adjacent nodes whose parent node is the same as the parent node of the object node. However, the method for determining the reference relationship is not limited to this.

[0203] Alternatively, the decoding process for attribute information may include at least one of inverse quantization, prediction, and arithmetic decoding. In this case, a reference refers to the use of a reference node in the calculation of the predicted value of the attribute information, or the use of the state of the reference node in the determination of the decoding parameters (e.g., indicating whether the reference node contains occupancy information of a point group). For example, the decoding parameters may be the quantization parameters in inverse quantization or the context in arithmetic decoding.

[0204] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. Furthermore, the first decoding unit 4640 uses the location information and attribute information to decode the required additional information during decoding, and outputs the additional information required by the application to the outside.

[0205] Next, an example of the structure of the location information encoding unit will be described. Figure 9 This is a block diagram of the location information encoding unit 2700 of this embodiment. The location information encoding unit 2700 includes an octree generation unit 2701, a geometric information calculation unit 2702, an encoding table selection unit 2703, and an entropy encoding unit 2704.

[0206] The octree generation unit 2701 generates, for example, an octree based on the input position information, and generates occupancy codes for each node of the octree. The geometry information calculation unit 2702 obtains information indicating whether the neighboring nodes of the object node are occupied nodes. For example, the geometry information calculation unit 2702 calculates the occupancy information (information indicating whether the neighboring node is occupied) of the neighboring nodes based on the occupancy code of the parent node to which the object node belongs. Furthermore, the geometry information calculation unit 2702 can pre-store the encoded nodes in a list and search for neighboring nodes from that list. In addition, the geometry information calculation unit 2702 can switch neighboring nodes based on the position of the object node within its parent node.

[0207] The encoding table selection unit 2703 uses the occupancy information of neighboring nodes calculated by the geometric information calculation unit 2702 to select the encoding table used in the entropy encoding of the target node. For example, the encoding table selection unit 2703 can use the occupancy information of neighboring nodes to generate a bit string and select the encoding table of the index number generated from that bit string.

[0208] The entropy encoding unit 2704 performs entropy encoding on the occupancy encoding of the object node using an encoding table with the selected index number, generating encoded position information and metadata. The entropy encoding unit 2704 may also append information representing the selected encoding table to the encoded position information.

[0209] The following explains the octree representation and the scanning order of location information. Location information (location data) is transformed into an octree structure (octreeification) and then encoded. The octree structure consists of nodes and leaves. Each node has 8 nodes or leaves, and each leaf contains voxel (VXL) information. Figure 10 This is a diagram representing a structural example containing the positional information of multiple voxels. Figure 11 It means Figure 10 The diagram shows an example of how location information is transformed into an octree structure. Here, Figure 11 In the leaves shown, leaves 1, 2, and 3 represent respectively Figure 10 The voxels shown are VXL1, VXL2, and VXL3, and VXL (hereinafter, effective VXL) containing dot groups are also shown.

[0210] Specifically, node 1 corresponds to containing Figure 10 The entire space containing the location information is divided into 8 nodes. Each of these 8 nodes contains a valid VXL and is further divided into 8 nodes or leaves. This process is repeated corresponding to the hierarchy of the tree structure. Here, each node corresponds to a subspace, and the node information includes information indicating which position of the next node or leaf exists after the division (occupancy encoding). Furthermore, the lowest-level block is placed in a leaf, and the number of point groups contained within the leaf is retained as leaf information.

[0211] Next, an example of the structure of the location information decoding unit will be described. Figure 12 This is a block diagram of the location information decoding unit 2710 of this embodiment. The location information decoding unit 2710 includes an octree generation unit 2711, a geometric information calculation unit 2712, an encoding table selection unit 2713, and an entropy decoding unit 2714.

[0212] The octree generation unit 2711 uses header information or metadata of the bitstream to generate an octree for a certain space (node). For example, the octree generation unit 2711 uses the size of the space along the x-axis, y-axis, and z-axis directions appended to the header information to generate a large space (root node), and generates eight smaller spaces A (nodes A0 to A7) by dividing this space into two along the x-axis, y-axis, and z-axis directions respectively, thereby generating an octree. In addition, nodes A0 to A7 are sequentially set as object nodes.

[0213] The geometric information calculation unit 2712 obtains occupancy information indicating whether the neighboring nodes of an object node are occupant nodes. For example, the geometric information calculation unit 2712 calculates the occupancy information of neighboring nodes based on the occupancy rate encoding of the parent node to which the object node belongs. Furthermore, the geometric information calculation unit 2712 can pre-store decoded nodes in a list and search for neighboring nodes from that list. Additionally, the geometric information calculation unit 2712 can switch neighboring nodes based on the position of the object node within its parent node.

[0214] The encoding table selection unit 2713 uses the occupancy information of neighboring nodes calculated by the geometric information calculation unit 2712 to select the encoding table (decoding table) used in the entropy decoding of the object node. For example, the encoding table selection unit 2713 can generate a bit string using the occupancy information of neighboring nodes and select the encoding table with the index number generated from that bit string.

[0215] The entropy decoding unit 2714 performs entropy decoding on the occupancy encoding of the object node using the selected encoding table to generate position information. Alternatively, the entropy decoding unit 2714 can also decode and obtain information about the selected encoding table from the bitstream, and use the encoding table represented by this information to perform entropy decoding on the occupancy encoding of the object node.

[0216] The structure of the attribute information encoding unit and the attribute information decoding unit will be explained below. Figure 13 This is a block diagram illustrating a structural example of the attribute information encoding unit A100. The attribute information encoding unit may include multiple encoding units that perform different encoding methods. For example, the attribute information encoding unit may switch between using the following two methods depending on the application.

[0217] The attribute information encoding unit A100 includes a LoD attribute information encoding unit A101 and a transform attribute information encoding unit A102. The LoD attribute information encoding unit A101 uses the position information of the 3D points to classify each 3D point into multiple levels, predicts the attribute information of the 3D points belonging to each level, and encodes the prediction residual. Here, each classified level is referred to as LoD (Level of Detail).

[0218] The transformation attribute information encoding unit A102 uses RAHT (Region Adaptive Hierarchical Transform) to encode attribute information. Specifically, the transformation attribute information encoding unit A102 applies RAHT or Haar transform to each attribute information based on the position information of the three-dimensional points, generates high-frequency and low-frequency components at each level, and encodes these values ​​using quantization and entropy coding.

[0219] Figure 14 This is a block diagram illustrating a structural example of the attribute information decoding unit A110. The attribute information decoding unit may include multiple decoding units that perform different decoding methods. For example, the attribute information decoding unit may also switch between the following two decoding methods based on information contained in the header or metadata.

[0220] The attribute information decoding unit A110 includes a LoD attribute information decoding unit A111 and a transform attribute information decoding unit A112. The LoD attribute information decoding unit A111 uses the position information of the 3D points to classify each 3D point into multiple levels, and decodes the attribute values ​​while predicting the attribute information of the 3D points belonging to each level.

[0221] The attribute information decoding unit A112 uses RAHT (Region Adaptive Hierarchical Transform) to decode the attribute information. Specifically, the attribute information decoding unit A112 applies inverse RAHT or inverse Haar transform to the high-frequency and low-frequency components of each attribute value based on the position information of the three-dimensional points to decode the attribute values.

[0222] Figure 15 This is a block diagram showing the structure of the attribute information encoding unit 3140, which is an example of the LoD attribute information encoding unit A101.

[0223] The attribute information encoding unit 3140 includes a LoD generation unit 3141, a surrounding search unit 3142, a prediction unit 3143, a prediction residual calculation unit 3144, a quantization unit 3145, an arithmetic encoding unit 3146, an inverse quantization unit 3147, a decoded value generation unit 3148, and a memory 3149.

[0224] The LoD generation unit 3141 generates LoDs using the position information of three-dimensional points.

[0225] The surrounding search unit 3142 uses the LoD generation result performed by the LoD generation unit 3141 and the distance information representing the distance between each 3D point to search for neighboring 3D points adjacent to each 3D point.

[0226] The prediction unit 3143 generates predicted values ​​of the attribute information of the three-dimensional points of the encoded object.

[0227] The prediction residual calculation unit 3144 calculates (generates) the prediction residual of the predicted values ​​of the attribute information generated by the prediction unit 3143.

[0228] The quantization unit 3145 quantizes the prediction residual of the attribute information calculated by the prediction residual calculation unit 3144.

[0229] The arithmetic encoding unit 3146 performs arithmetic encoding on the prediction residual quantized by the quantization unit 3145. The arithmetic encoding unit 3146 outputs a bit stream containing the arithmetic-encoded prediction residual to, for example, a three-dimensional data decoding device.

[0230] Furthermore, the prediction residual can also be binarized, for example, by the quantization unit 3145, before arithmetic encoding by the arithmetic encoding unit 3146.

[0231] Furthermore, for example, the arithmetic coding unit 3146 may initialize the coding table used for arithmetic coding before arithmetic coding. The arithmetic coding unit 3146 may initialize the coding table used for arithmetic coding on a layer-by-layer basis. In addition, the arithmetic coding unit 3146 may also include information indicating the position of the layer whose coding table has been initialized in the bitstream for output.

[0232] The inverse quantization unit 3147 performs inverse quantization on the prediction residuals quantized by the quantization unit 3145.

[0233] The decoding value generation unit 3148 generates a decoding value by adding the predicted value of the attribute information generated by the prediction unit 3143 to the prediction residual after inverse quantization by the inverse quantization unit 3147.

[0234] The memory 3149 is a memory that stores the decoded values ​​of the attribute information of each three-dimensional point decoded by the decoded value generation unit 3148. For example, when generating a prediction value for a three-dimensional point that has not yet been encoded, the prediction unit 3143 uses the decoded values ​​of the attribute information of each three-dimensional point stored in the memory 3149 to generate the prediction value.

[0235] Figure 16This is a block diagram of an attribute information encoding unit 6600, which is an example of an attribute information encoding unit A102. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transformation unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transformation unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0236] The sorting unit 6601 generates Morton codes using the position information of the three-dimensional points and sorts the multiple three-dimensional points according to the Morton code order. The Haar transform unit 6602 generates coding coefficients by applying the Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0237] The inverse quantization unit 6604 performs inverse quantization on the quantized coding coefficients. The inverse Haar transform unit 6605 applies the inverse Haar transform to the coding coefficients. The memory 6606 stores the values ​​of the attribute information of the decoded multiple 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 can also be used for the prediction of uncoded 3D points.

[0238] The arithmetic coding unit 6607 calculates ZeroCnt based on the quantized coding coefficients and performs arithmetic coding on ZeroCnt. Additionally, the arithmetic coding unit 6607 performs arithmetic coding on the non-zero quantized coding coefficients. The arithmetic coding unit 6607 can binarize the coding coefficients before arithmetic coding. Furthermore, the arithmetic coding unit 6607 can also generate and encode various header information.

[0239] Figure 17 This is a block diagram showing the structure of the attribute information decoding unit 3150, which is an example of the LoD attribute information decoding unit A111.

[0240] The attribute information decoding unit 3150 includes a LoD generation unit 3151, a surrounding search unit 3152, a prediction unit 3153, an arithmetic decoding unit 3154, an inverse quantization unit 3155, a decoded value generation unit 3156, and a memory 3157.

[0241] LoD generation unit 3151, using the location information decoding unit (in Figure 17 (Not shown in the figure) The location information of the decoded 3D points is used to generate the LoD.

[0242] The surrounding search unit 3152 uses the LoD generation result of the LoD generation unit 3151 and the distance information representing the distance between each three-dimensional point to search for neighboring three-dimensional points adjacent to each three-dimensional point.

[0243] The prediction unit 3153 generates predicted values ​​of the attribute information of the three-dimensional points of the decoded object.

[0244] Arithmetic decoding unit 3154 pairs from Figure 15 The prediction residuals within the bitstream obtained by the attribute information encoding unit 3140 are subjected to arithmetic decoding. Furthermore, the arithmetic decoding unit 3154 can initialize the decoding table used for arithmetic decoding. Figure 15 The arithmetic encoding unit 3146, as shown, performs encoding processing on the layers, and the arithmetic decoding unit 3154 initializes the decoding table used for arithmetic decoding. The arithmetic decoding unit 3154 can initialize the decoding table for arithmetic decoding on a layer-by-layer basis. Furthermore, the arithmetic decoding unit 3154 can initialize the decoding table based on information contained in the bitstream indicating the position of the layer after the encoding table initialization.

[0245] The inverse quantization unit 3155 performs inverse quantization on the prediction residual after arithmetic decoding by the arithmetic decoding unit 3154.

[0246] The decoding value generation unit 3156 adds the predicted value generated by the prediction unit 3153 and the predicted residual after inverse quantization by the inverse quantization unit 3155 to generate a decoded value. The decoding value generation unit 3156 outputs the decoded attribute information data to other devices.

[0247] The memory 3157 is a memory that stores the decoded values ​​of the attribute information of each three-dimensional point decoded by the decoded value generation unit 3156. For example, when generating a prediction value for a three-dimensional point that has not yet been decoded, the prediction unit 3153 uses the decoded values ​​of the attribute information of each three-dimensional point stored in the memory 3157 to generate the prediction value.

[0248] Figure 18 This is a block diagram of an attribute information decoding unit 6610, which is an example of a transformation attribute information decoding unit A112. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0249] The arithmetic decoding unit 6611 performs arithmetic decoding on the ZeroCnt and coding coefficients contained in the bitstream. Additionally, the arithmetic decoding unit 6611 can also decode various header information.

[0250] The inverse quantization unit 6612 performs inverse quantization on the coded coefficients decoded arithmetically. The inverse Haar transform unit 6613 applies the inverse Haar transform to the inverse quantized coded coefficients. The memory 6614 stores the values ​​of the attribute information of the decoded multiple 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 can also be used for the prediction of undecoded 3D points.

[0251] Next, the second encoding section 4650, which is an example of the encoding section 4613 performing the second encoding method, will be described. Figure 19 This is a diagram showing the structure of the second encoding section 4650. Figure 20 This is a block diagram of the second encoding section 4650.

[0252] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point group data using the second encoding method. The second encoding unit 4650 includes an additional information generation unit 4651, a location image generation unit 4652, an attribute image generation unit 4653, an image encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0253] The second encoding unit 4650 has the following characteristics: it generates a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encodes the generated position image and attribute image using an existing image encoding method. This second encoding method is also known as VPCC (video-based PCC).

[0254] Point cluster data is PCC point cluster data like a PLY file, or PCC point cluster data generated based on sensor information, which includes position information, attribute information, and other additional information (Metadata).

[0255] The additional information generation unit 4651 generates mapping information of multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0256] The position image generation unit 4652 generates a position image (Geometry Image) based on position information and mapping information generated by the additional information generation unit 4651. This position image may be, for example, a distance image representing distance as pixel values. Furthermore, this distance image may be an image of multiple point groups viewed from a single viewpoint (an image of multiple point groups projected onto a two-dimensional plane), multiple images of multiple point groups viewed from multiple viewpoints, or a single image formed by merging these multiple images.

[0257] The attribute image generation unit 4653 generates an attribute image based on attribute information and mapping information generated by the additional information generation unit 4651. This attribute image may be, for example, an image representing attribute information (e.g., color (RGB)) as pixel values. Alternatively, the image may be an image of multiple point groups viewed from a single viewpoint (an image of multiple point groups projected onto a two-dimensional plane), multiple images of multiple point groups viewed from multiple viewpoints, or a single image formed by merging these multiple images.

[0258] The image encoding unit 4654 encodes the location image and the attribute image using an image encoding method, thereby generating an encoded location image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image) as encoded data. Furthermore, any known encoding method can be used as the image encoding method. For example, the image encoding method could be AVC or HEVC.

[0259] The additional information encoding unit 4655 generates compressed metadata by encoding additional information and mapping information contained in the point group data.

[0260] The multiplexing unit 4656 generates a compressed stream as encoded data by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to the processing unit of the system layer (not shown).

[0261] Next, the second decoding unit 4660, which is an example of the decoding unit 4624 performing the second encoding method, will be described. Figure 21 This is a diagram showing the structure of the second decoding unit 4660. Figure 22 This is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point group data by decoding the encoded data (encoded stream) encoded using the second encoding method. The second decoding unit 4660 includes an inverse multiplexing unit 4661, an image decoding unit 4662, an additional information decoding unit 4663, a position information generation unit 4664, and an attribute information generation unit 4665.

[0262] The processing unit of the system layer (not shown) inputs the compressed stream of encoded data to the second decoding unit 4660.

[0263] The inverse multiplexing unit 4661 separates the compressed geometry image, compressed attribute image, compressed metadata, and other additional information from the encoded data.

[0264] The image decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using an image encoding method. Furthermore, any known encoding method can be used as the image encoding method. For example, the image encoding method could be AVC or HEVC.

[0265] The additional information decoding unit 4663 generates additional information, including mapping information, by decoding the encoded additional information.

[0266] The location information generation unit 4664 generates location information using a location image and mapping information. The attribute information generation unit 4665 generates attribute information using an attribute image and mapping information.

[0267] The second decoding unit 4660 uses the additional information required for decoding during decoding and outputs the additional information required for the application to the outside.

[0268] The following explains the issues related to the PCC encoding method. Figure 23 This is a diagram representing the protocol stack related to PCC encoded data. Figure 23 This refers to examples of data from other media, such as video (e.g., HEVC) or sound, that are multiplexed, transmitted, or stored in PCC encoded data.

[0269] Multiplexing methods and file formats have the function of multiplexing, transmitting, or storing various types of encoded data. In order to transmit or store encoded data, the encoded data must be transformed into the format of the multiplexing method. For example, HEVC specifies the technique of storing encoded data in a data structure called a NAL unit and storing the NAL unit in an ISOBMFF.

[0270] On the other hand, currently, as a method of encoding point group data, the first encoding method (Codec1) and the second encoding method (Codec2) are being studied, but the structure of the encoded data and the method of saving the encoded data to the system format have not been defined. There are problems such as the inability to directly perform MUX processing (multiplexing), transmission and accumulation in the encoding department.

[0271] In addition, unless a specific encoding method is described below, it refers to either the first encoding method or the second encoding method.

[0272] (Implementation Method 2)

[0273] In this embodiment, the types of encoded data (geometry, attribute, and metadata) generated by the first encoding unit 4630 or the second encoding unit 4650, the method for generating the metadata, and the multiplexing processing in the multiplexing unit will be described. Additionally, the metadata may also be expressed as a parameter set or control information.

[0274] In this embodiment, to be in Figure 4The example described above uses a dynamic object (a group of three-dimensional points that changes over time), but the same method can also be used for a static object (a group of three-dimensional points at any given time).

[0275] Figure 24 This diagram illustrates the structure of the encoding unit 4801 and the multiplexing unit 4802 included in the three-dimensional data encoding apparatus of this embodiment. The encoding unit 4801 corresponds, for example, to the first encoding unit 4630 or the second encoding unit 4650 described above. The multiplexing unit 4802 corresponds to the multiplexing unit 4634 or 4656 described above.

[0276] The encoding unit 4801 encodes the point group data of multiple PCC (Point Cloud Compression) frames to generate encoded data (Multiple Compressed Data) containing multiple location information, attribute information, and additional information.

[0277] The multiplexing unit 4802 transforms data into a data structure that takes into account data access in the decoding device by performing NAL unitization on data of multiple data types (location information, attribute information, and additional information).

[0278] Figure 25 This diagram illustrates a structural example of the encoded data generated by the encoding unit 4801. The arrows in the diagram represent dependencies related to the decoding of the encoded data; the root of the arrow depends on the data at the tip of the arrow. That is, the decoding device decodes the data at the tip of the arrow, and uses this decoded data to decode the data at the root of the arrow. In other words, dependency refers to the reference (use) of the data of the dependent target in the processing (encoding or decoding, etc.) of the data of the dependent source.

[0279] First, the generation and processing of the encoded location information data will be explained. The encoding unit 4801 generates compressed geometry data for each frame by encoding the location information of each frame. Furthermore, the compressed geometry data is represented by G(i), where i represents the frame number or the time of the frame, etc.

[0280] Furthermore, the encoding unit 4801 generates a position parameter set (GPS(i)) corresponding to each frame. The position parameter set contains parameters that can be used in the decoding of the encoded position data. In addition, the encoded position data for each frame depends on the corresponding position parameter set.

[0281] Furthermore, the encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit 4801 generates a position sequence parameter set (Geometry Sequence PS: also denoted as position SPS) that stores parameters commonly used in the decoding processing of multiple frames within the position sequence. The position sequence depends on the position SPS.

[0282] Next, the generation process of the encoded attribute information data will be explained. The encoding unit 4801 generates compressed attribute data for each frame by encoding the attribute information of each frame. Furthermore, the compressed attribute data is represented by A(i). Additionally, in Figure 25 The example shown has attributes X and Y. The encoded attribute data of attribute X is represented by AX(i), and the encoded attribute data of attribute Y is represented by AY(i).

[0283] Furthermore, the encoding unit 4801 generates an attribute parameter set (APS(i)) corresponding to each frame. The attribute parameter set for attribute X is represented by AXPS(i), and the attribute parameter set for attribute Y is represented by AYPS(i). The attribute parameter set contains parameters that can be used in the decoding of the encoded attribute information. The encoded attribute data depends on the corresponding attribute parameter set.

[0284] Furthermore, the encoded attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit 4801 generates an attribute sequence parameter set (also denoted as attribute SPS) that stores parameters that are commonly used in the decoding processing of multiple frames within the attribute sequence. The attribute sequence depends on the attribute SPS.

[0285] Furthermore, in the first encoding method, the encoded attribute data depends on the encoded position data.

[0286] In addition, Figure 25 The example shown illustrates a scenario where two types of attribute information exist (attribute X and attribute Y). In such cases, for example, two encoding units generate their respective data and metadata. Furthermore, for example, attribute sequences are defined for each type of attribute information, and attribute SPSs are generated for each type of attribute information.

[0287] In addition, Figure 25The example shown illustrates one type of location information and two types of attribute information, but it is not limited to this; there can be one type of attribute information or more than three types. In this case, the encoded data can still be generated using the same method. Furthermore, in the case of point group data without attribute information, there may be no attribute information. In this case, the encoding unit 4801 may not generate a parameter set associated with the attribute information.

[0288] Next, the generation process of additional information (metadata) will be explained. The encoding unit 4801 generates a parameter set for the entire PCC stream, namely the PCC stream PS (also denoted as stream PS). The encoding unit 4801 stores parameters in the stream PS that can be used in the decoding processing of one or more position sequences and one or more attribute sequences. For example, the stream PS contains identification information of the encoder / decoder representing the point group data and information representing the algorithm used in the encoding. The position sequences and attribute sequences depend on the stream PS.

[0289] Next, the access unit and GOF will be explained. In this embodiment, the considerations for introducing a new access unit (AU) and GOF (Group of Frame) will be discussed.

[0290] An access unit is the basic unit used to access data during decoding, consisting of one or more data elements and one or more metadata elements. For example, an access unit may consist of location information at the same time and one or more attribute information. GOF (God of Randomness) is a random access unit, consisting of one or more access units.

[0291] The encoding unit 4801 generates an Access Unit Header (AU Header) as identification information indicating the beginning of an access unit. The encoding unit 4801 stores parameters related to the access unit in the AU Header. For example, the AU Header contains the structure or information of the encoded data contained in the access unit. Furthermore, the AU Header contains parameters commonly used for the data contained in the access unit, such as parameters related to the decoding of the encoded data.

[0292] Alternatively, the encoding unit 4801 may generate an access unit delimiter that does not contain parameters related to the access unit, instead of the access unit header. This access unit delimiter is used as identification information indicating the beginning of the access unit. The decoding device identifies the beginning of the access unit by detecting the access unit header or the access unit delimiter.

[0293] Next, the generation of the identification information beginning with GOF will be explained. The encoding unit 4801 generates a GOF header (GOFHeader) as identification information representing the beginning of GOF. The encoding unit 4801 stores parameters related to GOF in the GOF header. For example, the GOF header contains the structure or information of the encoded data contained in the GOF. Furthermore, the GOF header contains parameters commonly used for the data contained in the GOF, such as parameters related to the decoding of the encoded data.

[0294] Alternatively, the encoding unit 4801 can also generate a GOF delimiter that does not contain parameters related to GOF, instead of the GOF header. This GOF delimiter is used as identification information indicating the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting either the GOF header or the GOF delimiter.

[0295] In PCC encoded data, an access unit is defined as a PCC frame unit. The decoding device accesses the PCC frame based on the identification information at the beginning of the access unit.

[0296] Furthermore, for example, a GOF (Go of Frames) is defined as a random access unit. The decoding device accesses the random access unit based on the identification information at the beginning of the GOF. For example, if PCC (Personalized Controlled Frames) frames are independent of each other and can be decoded independently, then PCC frames can also be defined as random access units.

[0297] Alternatively, more than two PCC frames can be allocated to one access unit, or multiple random access units can be allocated to one GOF.

[0298] In addition, the encoding unit 4801 can also define and generate parameter sets or metadata other than those mentioned above. For example, the encoding unit 4801 can also generate SEI (Supplemental Enhancement Information) that stores parameters that may not be used during decoding (optional parameters).

[0299] Next, the structure of the encoded data and the method for storing the encoded data in the NAL unit will be explained.

[0300] For example, the data format is specified according to each type of encoded data. Figure 26 This is a diagram showing examples of encoded data and NAL units.

[0301] For example, such as Figure 26 As shown, the encoded data includes a header and a payload. Alternatively, the encoded data may also contain encoded data, a header, or length information indicating the length (data volume) of the payload. Furthermore, the encoded data may not include a header.

[0302] The header may contain identification information used to determine the data. This identification information may indicate, for example, the type of data or the frame number.

[0303] The header, for example, contains identification information indicating a reference relationship. This identification information is stored in the header, for example, when there is a dependency between data, and is used to refer to a reference target from a reference source. For example, the header of the reference target contains identification information used to identify that data. The header of the reference source contains identification information indicating the reference target.

[0304] In addition, if the reference target or reference source can be identified or derived from other information, the identification information used to determine the data or the identification information indicating the reference relationship can be omitted.

[0305] The multiplexing unit 4802 stores the encoded data in the payload of the NAL unit. The NAL unit header contains pcc_nal_unit_type, which serves as identification information for the encoded data. Figure 27 This is a diagram representing a semantic instance of pcc_nal_unit_type.

[0306] like Figure 27 As shown, when pcc_codec_type is codec1 (Codec1: the first encoding method), the values ​​0 to 10 of pcc_nal_unit_type are assigned to the following data in codec1: Geometry, AttributeX, AttributeY, Geometry PS, AttrX PS, AttrX PS, AttrX PS, Geometry Sequence PS, AttributeX Sequence PS, AttributeY Sequence PS, AU Header, and GOF Header. Furthermore, values ​​from 11 onwards are reserved for codec1.

[0307] When pcc_codec_type is codec2 (Codec2: second encoding method), the values ​​0 to 2 of pcc_nal_unit_type are assigned to the codec's data A (DataA), metadata A (MetaDataA), and metadata B (MetaDataB). Furthermore, values ​​3 and later are assigned as reserves for codec2.

[0308] Next, the data output order will be explained. The constraints on the output order of the NAL unit will be explained below.

[0309] The multiplexing unit 4802 sends out the NAL unit together as a GOF or AU unit. The multiplexing unit 4802 configures the GOF header at the beginning of the GOF and the AU header at the beginning of the AU.

[0310] The multiplexing unit 4802 can also be configured with a sequence parameter set (SPS) for each AU, so that the decoding device can start decoding from the next AU even if data is lost due to packet loss or other reasons.

[0311] In cases where there are dependencies related to decoding in the encoded data, the decoding device decodes the reference source data after decoding the reference target data. In the decoding device, in order to enable decoding in the received order without rearranging the data, the multiplexing unit 4802 first sends out the reference target data.

[0312] Figure 28 This is a diagram illustrating an example of the output sequence of NAL units. Figure 28 These represent three examples: location information priority, parameter priority, and data merging.

[0313] The location-priority delivery order is an example where location-related information and attribute-related information are delivered together. In this delivery order, location-related information is delivered before attribute-related information.

[0314] For example, by using this transmission order, a decoding device that does not decode attribute information may be able to set a time during which no processing is performed by ignoring the decoding of attribute information. Furthermore, for example, in the case of a decoding device that wants to decode location information earlier, it may be possible to decode the location information earlier by obtaining the encoded data of the location information earlier.

[0315] In addition, Figure 28 In this context, the attributes XSPS and YSPS are combined and denoted as attribute SPS, but attributes XSPS and YSPS can also be configured separately.

[0316] In the parameter set priority sending order, the parameter set is sent first, followed by the data.

[0317] As described above, the multiplexing unit 4802 can send the NAL units in any order, as long as the constraint of the NAL unit sending order is followed. For example, order identification information can be defined, and the multiplexing unit 4802 has the function of sending NAL units in multiple order patterns. For example, the order identification information of the NAL units can be stored in the stream PS.

[0318] The 3D data decoding device can also perform decoding based on sequence recognition information. Alternatively, the 3D data decoding device can instruct the 3D data encoding device of the desired transmission sequence, and the 3D data encoding device (multiplexing unit 4802) can control the transmission sequence according to the instructed transmission sequence.

[0319] Furthermore, as long as the data follows the same sending order constraints as data merging, the multiplexing unit 4802 can also generate coded data that merges multiple functions. For example, such as Figure 28 As shown, the GOF header can also be merged with the AU header, and the AXPS header can also be merged with the AYPS header. In this case, the identifier representing data with multiple functions is defined in pcc_nal_unit_type.

[0320] Hereinafter, a variation of this embodiment will be described. As with frame-level PS, sequence-level PS, and PCC sequence-level PS, PS has levels. If the PCC sequence level is set as the higher level and the frame level is set as the lower level, the parameter storage method can also be used as described below.

[0321] The default PS value is represented by the higher-order PS. Furthermore, if the lower-order PS value differs from the higher-order PS value, the lower-order PS value is used to represent the PS value. Alternatively, the PS value may not be recorded in the higher-order PS, but may be recorded in the lower-order PS. Alternatively, one or both of the lower-order and higher-order PS values ​​may be used to indicate whether the PS value is represented by the lower-order PS, the higher-order PS, or both. Alternatively, the lower-order PS may be merged into the higher-order PS. Alternatively, if the lower-order PS and the higher-order PS are duplicated, the multiplexing unit 4802 may omit the transmission of one of them.

[0322] Alternatively, the encoding unit 4801 or the multiplexing unit 4802 can also segment the data into slices or tiles, and send out the segmented data. The segmented data contains information used to identify the segmented data, and the parameter set includes parameters used in decoding the segmented data. In this case, an identifier representing data related to tiles or slices, or data storing parameters, is defined in pcc_nal_unit_type.

[0323] The following explains the processing of sequence identification information. Figure 29 This is a flowchart of the processing performed by the three-dimensional data encoding device (encoding unit 4801 and multiplexing unit 4802) regarding the output sequence of NAL units.

[0324] First, the 3D data encoding device determines the output order of the NAL units (position information priority or parameter set priority) (S4801). For example, the 3D data encoding device determines the output order based on a specification from the user or an external device (e.g., a 3D data decoding device).

[0325] If the determined output order prioritizes position information (position information priority in S4802), the 3D data encoding device sets the sequence identification information contained in the stream PS to prioritize position information (S4803). That is, in this case, the sequence identification information indicates that the NAL units are output in a position-priority order. Furthermore, the 3D data encoding device outputs the NAL units in a position-priority order (S4804).

[0326] On the other hand, if the determined output order is parameter set priority (parameter set priority in S4802), the 3D data encoding device sets the order identification information contained in the stream PS to parameter set priority (S4805). That is, in this case, the order identification information indicates that the NAL units are output in parameter set priority order. Furthermore, the 3D data encoding device outputs the NAL units in parameter set priority order (S4806).

[0327] Figure 30 This is a flowchart of the processing performed by the three-dimensional data decoding device regarding the output order of the NAL units. First, the three-dimensional data decoding device parses the order identification information contained in the stream PS (S4811).

[0328] If the transmission order represented by the sequence identification information is position information priority (position information priority in S4812), the three-dimensional data decoding device assumes that the transmission order of the NAL unit is position information priority and decodes the NAL unit (S4813).

[0329] On the other hand, if the transmission order represented by the sequence identification information is parameter set priority (parameter set priority in S4812), the three-dimensional data decoding device assumes that the transmission order of the NAL unit is parameter set priority and decodes the NAL unit (S4814).

[0330] For example, the three-dimensional data decoding device may not obtain all NAL units in step S4813, but obtain NAL units related to position information, and decode the position information based on the obtained NAL units, without decoding the attribute information.

[0331] Next, the process of generating AU and GOF will be explained. Figure 31 This is a flowchart of the processing performed by the three-dimensional data encoding device (multiplexing unit 4802) on the AU and GOF generation in the multiplexing of NAL units.

[0332] First, the three-dimensional data encoding device determines the type of encoded data (S4821). Specifically, the three-dimensional data encoding device determines whether the encoded data of the object being processed is data starting with AU, data starting with GOF, or data other than those specified.

[0333] In the case where the encoded data begins with GOF (in S4822, it begins with GOF), the three-dimensional data encoding device configures the GOF header and AU header at the beginning of the encoded data belonging to GOF to generate NAL units (S4823).

[0334] When the encoded data begins with AU (in S4822), the three-dimensional data encoding device assigns the AU header to the beginning of the encoded data belonging to AU to generate a NAL unit (S4824).

[0335] If the encoded data does not start with either GOF or AU (other than GOF or AU in S4822), the 3D data encoding device will assign the encoded data after the AU header of the AU to which the encoded data belongs to generate a NAL unit (S4825).

[0336] Next, the handling of access to AU and GOF will be explained. Figure 32 This is a flowchart of the processing of a three-dimensional data decoding device for accessing AU and GOF in the inverse multiplexing of the NAL unit.

[0337] First, the 3D data decoding device determines the type of encoded data contained in the NAL unit by parsing the nal_unit_type contained in the NAL unit (S4831). Specifically, the 3D data decoding device determines whether the encoded data contained in the NAL unit is data starting with AU, data starting with GOF, or data other than those specified.

[0338] If the encoded data contained in the NAL unit starts with GOF (GOF in S4832), the three-dimensional data decoding device determines that the NAL unit is the starting position of random access, accesses the NAL unit, and starts the decoding process (S4833).

[0339] On the other hand, if the encoded data contained in the NAL unit is data starting with AU (in S4832, it starts with AU), the three-dimensional data decoding device determines that the NAL unit starts with AU, accesses the data contained in the NAL unit, and decodes the AU (S4834).

[0340] On the other hand, if the encoded data contained in the NAL unit does not start with either GOF or AU (other than GOF or AU in S4832), the three-dimensional data decoding device does not process the NAL unit.

[0341] (Implementation Method 3)

[0342] The information of a 3D point group includes location information (geometry) and attribute information (attributes). Location information includes the coordinates (x-coordinate, y-coordinate, z-coordinate) relative to a given point. Instead of directly encoding the coordinates of each 3D point, a method is used to reduce the amount of encoding by representing the position of each 3D point using an octree and encoding the octree information.

[0343] On the other hand, attribute information includes information representing the color (RGB, YUV, etc.), reflectivity, and normal vectors of each 3D point. For example, a 3D data encoding device can encode attribute information using an encoding method different from that used for position information.

[0344] In this embodiment, the encoding method for attribute information is described. Furthermore, in this embodiment, integer values ​​are used as the values ​​of the attribute information. For example, when each color component of the RGB or YUV color information has 8-bit precision, each color component takes an integer value from 0 to 255. When the reflectance value has 10-bit precision, the reflectance value takes an integer value from 0 to 1023. Additionally, when the bit precision of the attribute information is decimal precision, the three-dimensional data encoding device may multiply the value by a scaling value and round it to an integer value to make the attribute information value an integer. Furthermore, the three-dimensional data encoding device may also append this scaling value to the header of the bitstream.

[0345] As a method for encoding attribute information of 3D points, consider calculating the predicted value of the attribute information of the 3D point and encoding the difference (prediction residual) between the original attribute information value and the predicted value. For example, if the attribute information value of 3D point p is Ap and the predicted value is Pp, the 3D data encoding device encodes its absolute difference value Diffp = |Ap-Pp|. In this case, if the predicted value Pp can be generated with high accuracy, the value of the absolute difference value Diffp becomes smaller. Therefore, for example, by using an encoding table that generates fewer bits as the value is smaller, the amount of encoding can be reduced by entropy encoding the absolute difference value Diffp.

[0346] As a method for generating predicted values ​​of attribute information, the attribute information of other three-dimensional points, i.e., reference three-dimensional points, located around the object's three-dimensional point are considered. Here, a reference three-dimensional point refers to a three-dimensional point within a predefined distance range from the object's three-dimensional point. For example, given the existence of object three-dimensional points p = (x1, y1, z1) and q = (x2, y2, z2), the three-dimensional data encoding device calculates the Euclidean distance d(p, q) between three-dimensional points p and q as shown in Equation A1.

[0347] [Formula 1]

[0348]

[0349] If the Euclidean distance d(p, q) is less than a predetermined threshold THd, the 3D data encoding device determines that the position of 3D point q is close to the position of object 3D point p, and determines that the value of the attribute information of 3D point q will be used in the generation of the predicted value of the attribute information of object 3D point p. Alternatively, other methods can be used for distance calculation, such as Mahalanobis distance. Furthermore, the 3D data encoding device can also determine not to use 3D points outside the predetermined distance range from the object 3D point for prediction processing. For example, if there exists a 3D point r, and the distance d(p, r) between the object 3D point p and the 3D point r is greater than or equal to the threshold THd, the 3D data encoding device can also determine not to use the 3D point r for prediction. In addition, the 3D data encoding device can append information representing the threshold THd to the header of the bitstream.

[0350] Figure 33 This is a diagram illustrating an example of a three-dimensional point. In this example, the distance d(p, q) between object three-dimensional point p and three-dimensional point q is less than the threshold THd. Therefore, the three-dimensional data encoding device determines three-dimensional point q as a reference three-dimensional point for object three-dimensional point p, and determines it as the value of attribute information Aq of three-dimensional point q used in the generation of the predicted value Pp of attribute information Ap of object three-dimensional point p.

[0351] On the other hand, the distance d(p, r) between the object's 3D point p and 3D point r is above the threshold THd. Therefore, the 3D data encoding device determines that 3D point r is not a reference 3D point for the object's 3D point p, and determines that the value of the attribute information Ar of 3D point r is not used in the generation of the predicted value Pp of the attribute information Ap of the object's 3D point p.

[0352] Furthermore, when encoding the attribute information of 3D points of an object using predicted values, the 3D data encoding device uses 3D points whose attribute information has already been encoded and decoded as reference 3D points. Similarly, when decoding the attribute information of 3D points of a decoded object using predicted values, the 3D data decoding device uses 3D points whose attribute information has already been decoded as reference 3D points. Thus, the same predicted values ​​can be generated during encoding and decoding, enabling accurate decoding of the bitstream of 3D points generated through encoding on the decoding side.

[0353] Furthermore, when encoding the attribute information of 3D points, it is advisable to classify each 3D point into multiple levels using its positional information before encoding. Here, each classified level is referred to as LoD (Level of Detail). Figure 34 The method for generating LoD is explained.

[0354] First, the 3D data encoding device selects an initial point a0 and assigns it to LoD0. Next, the 3D data encoding device extracts point a1 whose distance from point a0 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. Then, the 3D data encoding device extracts point a2 whose distance from point a1 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. In this way, the 3D data encoding device constructs LoD0 such that the distance between all points within LoD0 is greater than the threshold Thres_LoD[0].

[0355] Next, the 3D data encoding device selects point b0, which has not yet been assigned a LoD, and assigns it to LoD1. Then, the 3D data encoding device extracts point b1, which is not assigned a LoD and whose distance from point b0 is greater than the LoD1 threshold Thres_LoD[1], and assigns it to LoD1. Next, the 3D data encoding device extracts point b2, which is not assigned a LoD and whose distance from point b1 is greater than the LoD1 threshold Thres_LoD[1], and assigns it to LoD1. Thus, the 3D data encoding device constructs LoD1 such that the distance between points within LoD1 is greater than the threshold Thres_LoD[1].

[0356] Next, the 3D data encoding device selects point c0, which has not yet been assigned a LoD, and assigns it to LoD2. Then, the 3D data encoding device extracts point c1, which is not assigned a LoD and whose distance from point c0 is greater than the LoD2 threshold Thres_LoD[2], and assigns it to LoD2. Next, the 3D data encoding device extracts point c2, which is not assigned a LoD and whose distance from point c1 is greater than the LoD2 threshold Thres_LoD[2], and assigns it to LoD2. Thus, the 3D data encoding device constructs LoD2 such that the distance between points within LoD2 is greater than the threshold Thres_LoD[2]. For example, as... Figure 35 As shown, the thresholds Thres_LoD[0], Thres_LoD[1] and Thres_LoD[2] for each LoD are set.

[0357] Additionally, the 3D data encoding device can also append information representing the threshold of each LoD to the header of the bitstream. For example, in Figure 35 In the example shown, the three-dimensional data encoding device can also append thresholds Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] to the head.

[0358] Alternatively, the 3D data encoding device can assign all 3D points without assigned LoDs to the lowest level of the LoD. In this case, the 3D data encoding device can reduce the encoding amount of the head by not appending a threshold of the lowest level of the LoD to the head. For example, in Figure 35 In the example shown, the 3D data encoding device appends thresholds Thres_LoD[0] and Thres_LoD[1] to the header, but does not append Thres_LoD[2]. In this case, the 3D data decoding device can also estimate the value of Thres_LoD[2] as 0. Alternatively, the 3D data encoding device can also append the number of LoD levels to the header. Thus, the 3D data decoding device can use the number of LoD levels to determine the lowest-level LoD.

[0359] In addition, such as Figure 35 As shown, the threshold values ​​for each layer of LoD are set to be larger as the layer increases. Therefore, higher layers (those closer to LoD0) tend to become sparse clusters of points with greater distances between them, while lower layers tend to become dense clusters of points with closer distances between them. Furthermore, in... Figure 35 In the example shown, LoD0 is the highest level.

[0360] Furthermore, the method for selecting the initial 3D point when setting each LoD can also depend on the encoding order during position information encoding. For example, the 3D data encoding device selects the 3D point that was first encoded during position information encoding as the initial point a0 of LoD0, and uses the initial point a0 as the base point to select points a1 and a2 to construct LoD0. Moreover, the 3D data encoding device can also select the 3D point whose position information was encoded earliest among the 3D points that do not belong to LoD0 as the initial point b0 of LoD1. That is, the 3D data encoding device can also select the 3D point whose position information was encoded earliest among the 3D points in the upper layer (LoD0 to LoDn-1) that does not belong to LoDn as the initial point n0 of LoDn. Thus, by using the same initial point selection method during decoding, the 3D data decoding device can construct the same LoD as during encoding, and therefore can appropriately decode the bitstream. Specifically, the 3D data decoding device selects the 3D point whose position information was decoded earliest among the 3D points in the upper layer that does not belong to LoDn as the initial point n0 of LoDn.

[0361] The following explains the method for generating predicted values ​​of attribute information for 3D points using LoD information. For example, when encoding sequentially starting from the 3D points contained in LoD0, the 3D data encoding device uses the encoded and decoded (hereinafter referred to as "encoded") attribute information contained in LoD0 and LoD1 to generate the object 3D points contained in LoD1. Thus, the 3D data encoding device uses the encoded attribute information contained in LoDn' (n' <= n) to generate predicted values ​​of the attribute information for the 3D points contained in LoDn. That is, the 3D data encoding device does not use the attribute information of the 3D points contained in the lower layers of LoDn in calculating the predicted values ​​of the attribute information for the 3D points contained in LoDn.

[0362] For example, a 3D data encoding device generates predicted values ​​for the attribute information of a 3D point by calculating the average of the attribute values ​​of up to N encoded 3D points surrounding the 3D point of the object being encoded. Furthermore, the 3D data encoding device can append the value of N to the beginning of the bitstream. Alternatively, the 3D data encoding device can change the value of N for each 3D point, appending the value of N to each 3D point. This allows for the selection of an appropriate N for each 3D point, thereby improving the accuracy of the predicted values ​​and reducing prediction residuals. Alternatively, the 3D data encoding device can append the value of N to the beginning of the bitstream, fixing the value of N within the bitstream. This eliminates the need to encode or decode the value of N for each 3D point, thus reducing processing load. Alternatively, the 3D data encoding device can encode the value of N separately for each LoD. This improves encoding efficiency by selecting an appropriate N for each LoD.

[0363] Alternatively, the 3D data encoding device can calculate the predicted value of the attribute information of a 3D point by using a weighted average of the attribute information of N surrounding encoded 3D points. For example, the 3D data encoding device uses the distance information between the object's 3D point and each of the N surrounding 3D points to calculate the weights.

[0364] When a 3D data encoding device encodes the value of N for each LoD (Location of Detail), for example, the higher the LoD level, the larger the value of N is set, and the lower the LoD level, the smaller the value of N is set. In the higher LoD levels, the distances between 3D points belonging to that level are large, so it is possible to improve prediction accuracy by setting a large value of N and selecting multiple surrounding 3D points for averaging. On the other hand, since the distances between 3D points belonging to the lower LoD levels are small, it is possible to achieve efficient prediction while suppressing the processing load of averaging by setting a small value of N.

[0365] Figure 36 This is a diagram illustrating an example of the attribute information used in the predicted values. As described above, predicted values ​​for points P contained in LoDn are generated using the encoded surrounding points P' contained in LoDN' (N' <= N). Here, the surrounding points P' are selected based on their distance from point P. For example, attribute information from points a0, a1, a2, b0, and b1 is used to generate... Figure 36 The predicted value of the attribute information of point b2 shown.

[0366] The selected surrounding points change depending on the value of N. For example, when N=5, a0, a1, a2, b0, and b1 are selected as surrounding points of point b2. When N=4, points a0, a1, a2, and b1 are selected based on distance information.

[0367] The predicted value is calculated by using a distance-weighted average. For example, in Figure 36 In the example shown, the predicted value a2p of point a2, as shown in equations A2 and A3, is calculated by a weighted average of the attribute information of points a0 and a1. Additionally, A i It is the value of the attribute information of point ai.

[0368] [Formula 2]

[0369]

[0370]

[0371] Furthermore, the predicted value b2p of point b2, as shown in equations A4 to A6, is calculated by a weighted average of the attribute information of points a0, a1, a2, b0, and b1. Additionally, B i It is the value of the attribute information of point bi.

[0372] [Formula 3]

[0373]

[0374]

[0375]

[0376] In addition, the 3D data encoding device can also calculate the difference (prediction residual) between the attribute information values ​​of 3D points and the predicted values ​​generated from surrounding points, and quantize the calculated prediction residual. For example, the 3D data encoding device quantizes by dividing the prediction residual by the quantization scale (also known as the quantization step size). In this case, the smaller the quantization scale, the smaller the error (quantization error) caused by quantization. Conversely, the larger the quantization scale, the larger the quantization error.

[0377] Furthermore, the 3D data encoding device can also change the quantization scale used for each LoD. For example, the higher the 3D data encoding layer, the smaller the quantization scale; the lower the quantization layer, the larger the quantization scale. The attribute information of 3D points belonging to the higher layer may be used as the predicted value of the attribute information of 3D points belonging to the lower layer. Therefore, reducing the quantization scale at the higher layer can suppress quantization errors generated at the higher layer, and by improving the accuracy of the predicted values, encoding efficiency can be improved. In addition, the 3D data encoding device can also append the quantization scale used for each LoD to the header. Thus, the 3D data decoding device can correctly decode the quantization scale, and therefore can appropriately decode the bitstream.

[0378] Furthermore, the 3D data encoding device can also transform the signed integer values ​​(signed quantized values) of the quantized prediction residuals into unsigned integer values ​​(unsigned quantized values). Therefore, when entropy encoding the prediction residuals, the generation of negative integers does not need to be considered. Moreover, the 3D data encoding device does not necessarily need to transform signed integer values ​​into unsigned integer values; for example, entropy encoding can be performed separately on the sign bits.

[0379] The prediction residual is calculated by subtracting the predicted value from the original value. For example, as shown in Equation A7, the prediction residual a2r of point a2 is calculated by subtracting the predicted value a2p of point a2 from the attribute information value A2 of point a2. As shown in Equation A8, the prediction residual b2r of point b2 is calculated by subtracting the predicted value b2p of point b2 from the attribute information value B2 of point b2.

[0380] a2r=A2-a2p…(Equation A7)

[0381] b2r=B2-b2p…(Equation A8)

[0382] Furthermore, the prediction residual is quantized by dividing by QS (Quantization Step). For example, the quantized value a2q for point a2 is calculated using (Equation A9). The quantized value b2q for point b2 is calculated using (Equation A10). Here, QS_LoD0 is the QS used for LoD0, and QS_LoD1 is the QS used for LoD1. That is, QS can be changed according to LoD.

[0383] a2q=a2r / QS_LoD0…(Equation A9)

[0384] b2q=b2r / QS_LoD1…(Equation A10)

[0385] Furthermore, as described below, the three-dimensional data encoding device transforms the signed integer value, which is the quantization value mentioned above, into an unsigned integer value. When the signed integer value a2q is less than 0, the three-dimensional data encoding device sets the unsigned integer value a2u to -1 - (2 × a2q). When the signed integer value a2q is 0 or greater, the three-dimensional data encoding device sets the unsigned integer value a2u to 2 × a2q.

[0386] Similarly, when the signed integer value b2q is less than 0, the 3D data encoding device sets the unsigned integer value b2u to -1-(2×b2q). When the signed integer value b2q is 0 or greater, the 3D data encoding device sets the unsigned integer value b2u to 2×b2q.

[0387] Furthermore, the 3D data encoding device can also encode the quantized prediction residuals (unsigned integer values) using entropy encoding. For example, binary arithmetic encoding can be applied after binarizing the unsigned integer values.

[0388] Furthermore, in this case, the 3D data encoding device can also switch the binarization method based on the value of the prediction residual. For example, when the prediction residual pu is less than the threshold R_TH, the 3D data encoding device binarizes the prediction residual pu with the required fixed number of bits to represent the threshold R_TH. Conversely, when the prediction residual pu is greater than or equal to the threshold R_TH, the 3D data encoding device uses Exponential-Golomb or similar methods to binarize the binarized data of the threshold R_TH and the value of (pu-R_TH).

[0389] For example, when the threshold R_TH is 63 and the prediction residual pu is less than 63, the 3D data encoding device binarizes the prediction residual pu with 6 bits. Alternatively, when the prediction residual pu is greater than or equal to 63, the 3D data encoding device uses exponential Golomb to binarize the binary data (111111) and (pu-63) of the threshold R_TH, thereby performing arithmetic encoding.

[0390] In a more specific example, when the prediction residual pu is 32, the 3D data encoding device generates 6 bits of binary data (100000) and performs arithmetic encoding on this bit string. Alternatively, when the prediction residual pu is 66, the 3D data encoding device generates binary data (111111) representing the threshold R_TH in exponential Golbus representation and a bit string (00100) containing the value 3 (66-63), and performs arithmetic encoding on this bit string (111111+00100).

[0391] In this way, the 3D data encoding device switches the binarization method according to the magnitude of the prediction residual, thereby enabling encoding while suppressing the sharp increase in the number of binarized bits as the prediction residual increases. Furthermore, the 3D data encoding device can also append a threshold R_TH to the header of the bitstream.

[0392] For example, when encoding at a high bit rate, i.e., with a small quantization scale, the quantization error is smaller, the prediction accuracy is higher, and the prediction residual may not increase. Therefore, in this case, the 3D data encoding device sets a large threshold R_TH. This reduces the likelihood of encoding binary data at the threshold R_TH, thus improving encoding efficiency. Conversely, when encoding at a low bit rate, i.e., with a large quantization scale, the quantization error increases, the prediction accuracy decreases, and the prediction residual may increase. Therefore, in this case, the 3D data encoding device sets a small threshold R_TH. This prevents a sharp increase in the bit length of the binary data.

[0393] Furthermore, the 3D data encoding device can also switch the threshold R_TH for each LoD and append the threshold R_TH for each LoD to the header. That is, the 3D data encoding device can also switch the binarization method for each LoD. For example, in the upper layer, due to the large distance between 3D points, the prediction accuracy deteriorates, and the resulting prediction residual may become larger. Therefore, the 3D data encoding device sets a small threshold R_TH for the upper layer to prevent a sharp increase in the bit length of the binarized data. Conversely, in the lower layer, due to the small distance between 3D points, the prediction accuracy is higher, and the resulting prediction residual may become smaller. Therefore, the 3D data encoding device sets a large threshold R_TH for each layer to improve encoding efficiency.

[0394] Figure 37 This is a diagram illustrating an example of exponential Golomb codes, showing the relationship between the values ​​before binarization (multi-valued) and the bits after binarization (encoding). Additionally, it can also be... Figure 37 The 0s and 1s shown are reversed.

[0395] Furthermore, the 3D data encoding device applies arithmetic coding to the binarized data of the prediction residuals. This improves encoding efficiency. Moreover, when applying arithmetic coding, the probability of 0 and 1 occurrences of each bit may differ between the portion binarized with n bits (n-bit code) and the portion binarized using exponential Golombian (remaining code). Therefore, the 3D data encoding device can also switch between the arithmetic coding application method using n-bit code and remaining code.

[0396] For example, for an n-bit code, the three-dimensional data encoding device uses a different encoding table (probability table) for arithmetic encoding of each bit. In this case, the three-dimensional data encoding device can also change the number of encoding tables used for each bit. For example, the three-dimensional data encoding device uses one encoding table to arithmetically encode the first bit b0 of the n-bit code. Additionally, the three-dimensional data encoding device uses two encoding tables for the next bit b1. Furthermore, the three-dimensional data encoding device switches the encoding table used in the arithmetic encoding of bit b1 according to the value of b0 (0 or 1). Similarly, the three-dimensional data encoding device also uses four encoding tables for the next bit b2. Additionally, the three-dimensional data encoding device switches the encoding table used in the arithmetic encoding of bit b2 according to the values ​​of b0 and b1 (0-3).

[0397] Thus, when the three-dimensional data encoding device performs arithmetic encoding on each bit bn-1 of the n-bit code, it uses 2 n-1 The three-dimensional data encoding device uses an encoding table. Furthermore, it switches the encoding table based on the values ​​(occurrence patterns) of the bits preceding bn-1. This allows the three-dimensional data encoding device to use the appropriate encoding table for each bit, thus improving encoding efficiency.

[0398] Furthermore, the three-dimensional data encoding device can also reduce the number of encoding tables used in each bit. For example, when performing arithmetic encoding on each bit bn-1, the three-dimensional data encoding device can also switch between two encoding tables based on the values ​​(occurrence patterns) of the m bits preceding bn-1 (m < n-1). mThe three-dimensional data encoding device uses a number of encoding tables. This reduces the number of encoding tables used per bit and improves encoding efficiency. Furthermore, the device can update the occurrence probabilities of 0 and 1 in each encoding table based on the actual value of the generated binarized data. Additionally, the three-dimensional data encoding device can fix the occurrence probabilities of 0 and 1 in the encoding tables of a subset of bits. This reduces the number of probability updates and thus decreases processing complexity.

[0399] For example, in the case of n-bit code b0b1b2…bn-1, b0 uses a single encoding table (CTb0). b1 uses two encoding tables (CTb10, CTb11). Furthermore, the encoding table used is switched based on the value of b0 (0-1). b2 uses four encoding tables (CTb20, CTb21, CTb22, CTb23). Additionally, the encoding table used is switched based on the values ​​of b0 and b1 (0-3). bn-1 uses a 2-bit encoding table. n-1 (CTbn0, CTbn1, ..., CTbn(2)) n-1 -1)). Additionally, based on the values ​​of b0b1…bn-2 (0~2) n -1 -1) to switch the encoding table used.

[0400] Furthermore, the three-dimensional data encoding device can also use a setting of 0 to 2 without binarizing the n-bit code. n The arithmetic code of the m-ary value of -1 (m=2) n Furthermore, when a 3D data encoding device uses m-ary to arithmetically encode an n-bit code, a 3D data decoding device can also recover the n-bit code through arithmetic decoding of m-ary.

[0401] Figure 38 This is a diagram used to illustrate the processing, for example, when the residual code is an exponential Golomb code. (e.g.) Figure 38 As shown, the portion binarized using the exponential Golbus method, i.e., the residual code, includes a prefix part and a suffix part. For example, the three-dimensional data encoding device switches the encoding table between the prefix part and the suffix part. That is, the three-dimensional data encoding device uses the encoding table for the prefix to perform arithmetic encoding on each bit contained in the prefix part, and uses the encoding table for the suffix to perform arithmetic encoding on each bit contained in the suffix part.

[0402] Furthermore, the 3D data encoding device can update the occurrence probabilities of 0 and 1 in each encoding table based on the actual values ​​of the generated binarized data. Alternatively, the 3D data encoding device can fix the occurrence probabilities of 0 and 1 in a certain encoding table. This reduces the number of probability updates and thus decreases the processing load. For example, the 3D data encoding device can update the occurrence probabilities for the prefix section and fix the occurrence probabilities for the suffix section.

[0403] Furthermore, the 3D data encoding device decodes the quantized prediction residual through inverse quantization and reconstruction, and uses the decoded prediction residual, i.e., the decoded value, for subsequent predictions of the 3D points of the encoded object. Specifically, the 3D data encoding device calculates the inverse quantization value by multiplying the quantized prediction residual (quantized value) by the quantization scale, and adds the inverse quantization value to the predicted value to obtain the decoded value (reconstructed value).

[0404] For example, the inverse quantization value a2iq of point a2 is calculated using the quantization value a2q of point a2 via (Equation A11). The inverse quantization value b2iq of point b2 is calculated using the quantization value b2q of point b2 via (Equation A12). Here, QS_LoD0 is the QS used for LoD0, and QS_LoD1 is the QS used for LoD1. That is, the QS can be changed according to the LoD.

[0405] a2iq=a2q×QS_LoD0…(Equation A11)

[0406] b2iq=b2q×QS_LoD1…(Equation A12)

[0407] For example, as shown in Equation A13, the decoded value a2rec of point a2 is calculated by adding the predicted value a2p of point a2 to the inverse quantized value a2iq of point a2. As shown in Equation A14, the decoded value b2rec of point b2 is calculated by adding the predicted value b2p of point b2 to the inverse quantized value b2iq of point b2.

[0408] a2rec=a2iq+a2p…(Equation A13)

[0409] b2rec=b2iq+b2p…(Equation A14)

[0410] The syntax of the bitstream in this embodiment will be explained below. Figure 39 This is a diagram illustrating the syntax of the attribute header in this implementation. The attribute header is the header information for attribute information. For example... Figure 39As shown, the attribute header includes level number information (NumLoD), 3D point number information (NumOfPoint[i]), level threshold (Thres_Lod[i]), surrounding point number information (NumNeighborPoint[i]), prediction threshold (THd[i]), quantization scale (QS[i]), and binarization threshold (R_TH[i]).

[0411] The level number information (NumLoD) indicates the number of LoD levels used.

[0412] The 3D point number information (NumOfPoint[i]) indicates the number of 3D points belonging to level i. In addition, the 3D data encoding apparatus may also attach 3D point total number information (AllNumOfPoint) indicating the total number of 3D points to another header. In this case, the 3D data encoding apparatus may not attach NumOfPoint[NumLoD-1], which indicates the number of 3D points belonging to the lowest level, to the header. In this case, the 3D data decoding apparatus can calculate NumOfPoint[NumLoD-1] through (Equation A15). As a result, the coding amount of the header can be reduced.

[0413] [Equation 4]

[0414]

[0415] The level threshold (Thres_Lod[i]) is a threshold for setting level i. The 3D data encoding apparatus and the 3D data decoding apparatus construct LoDi such that the distance between each point in LoDi is greater than the threshold Thres_LoD[i]. In addition, the 3D data encoding apparatus may not attach the value of Thres_Lod[NumLoD-1] (the lowest level) to the header. In this case, the 3D data decoding apparatus estimates the value of Thres_Lod[NumLoD-1] as 0. As a result, the coding amount of the header can be reduced.

[0416] The surrounding point number information (NumNeighborPoint[i]) indicates the upper limit value of the number of surrounding points used in generating the prediction value of the 3D points belonging to level i. When the number of surrounding points M is less than NumNeighborPoint[i] (M<NumNeighborPoint[i]), the 3D data encoding apparatus may also use the M surrounding points to calculate the prediction value. In addition, when it is not necessary to separate the value of NumNeighborPoint[i] for each LoD, the 3D data encoding apparatus may attach one piece of surrounding point number information (NumNeighborPoint) used in all LoDs to the header.

[0417] The prediction threshold (THd[i]) represents the upper limit of the distance between the surrounding 3D points and the object's 3D point used in the prediction of the object's 3D point in level i. The 3D data encoding and decoding devices do not use 3D points farther from the object's 3D point than THd[i] for prediction. Alternatively, when it is not necessary to separate the value of THd[i] in each LoD, the 3D data encoding device may append a single prediction threshold (THd) used in all LoDs to the head.

[0418] The quantization scale (QS[i]) represents the quantization scale used in quantization and dequantization at level i.

[0419] The binarization threshold (R_TH[i]) is a threshold used to switch the binarization method for the prediction residuals of 3D points belonging to level i. For example, when the prediction residual is less than the threshold R_TH, the 3D data encoding device binarizes the prediction residual pu with a fixed number of bits; when the prediction residual is greater than the threshold R_TH, it uses exponential Golomb binarization to binarize the binarized data of the threshold R_TH and the value of (pu-R_TH). Alternatively, if it is not necessary to switch the value of R_TH[i] in each LoD, the 3D data encoding device may also append a single binarization threshold (R_TH) used in all LoDs to the head.

[0420] Furthermore, R_TH[i] can also be the maximum value represented in n bits. For example, in 6 bits, R_TH is 63, and in 8 bits, R_TH is 255. Alternatively, the three-dimensional data encoding device can encode the number of bits instead of using the maximum value represented in n bits as the binarization threshold. For example, the three-dimensional data encoding device can append the value 6 to the header when R_TH[i] = 63, and append the value 8 to the header when R_TH[i] = 255. Additionally, the three-dimensional data encoding device can define a minimum number of bits representing R_TH[i] (minimum number of bits) and append the value to the header based on the relative number of bits of the minimum value. For example, the three-dimensional data encoding device can append the value 0 to the header when R_TH[i] = 63 and the minimum number of bits is 6, and append the value 2 to the header when R_TH[i] = 255 and the minimum number of bits is 6.

[0421] Alternatively, the 3D data encoding device can also entropy encode at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] and append it to the header. For example, the 3D data encoding device can also binarize each value for arithmetic encoding. Furthermore, to reduce processing overhead, the 3D data encoding device can also encode each value with a fixed length.

[0422] Furthermore, the 3D data encoding device may not append at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] to the header. For example, the value of at least one of them may also be specified by a profile or level, such as a standard. This allows for a reduction in the number of bits in the header.

[0423] Figure 40 This is a diagram illustrating the syntax of the attribute data in this embodiment. The attribute data includes encoded data of attribute information for multiple three-dimensional points. For example... Figure 40 As shown, the attribute data includes n-bit code and remaining code.

[0424] An n-bit code is the encoded data of the prediction residual of attribute information values, or a portion thereof. The bit length of an n-bit code depends on the value of R_TH[i]. For example, when R_TH[i] is 63, the n-bit code is 6 bits; when R_TH[i] is 255, the n-bit code is 8 bits.

[0425] The remaining code is the encoded data of the predicted residual of the attribute information value, encoded using exponential Golomb coding. If the n-bit code is the same as R_TH[i], the remaining code is encoded or decoded. Alternatively, the 3D data decoding device adds the value of the n-bit code and the value of the remaining code to decode the predicted residual. Furthermore, if the n-bit code is not the same as R_TH[i], the remaining code may not be encoded or decoded.

[0426] The following describes the processing flow in the three-dimensional data encoding device. Figure 41 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device.

[0427] First, the 3D data encoding device encodes the location information (geometry) (S3001). For example, the 3D data encoding device uses an octree representation for encoding.

[0428] After encoding the position information, the 3D data encoding device redistributes the original attribute information of the 3D points when the position of a 3D point changes due to quantization or other reasons (S3002). For example, the 3D data encoding device redistributes the attribute information values ​​by interpolating based on the amount of position change. For example, the 3D data encoding device detects N 3D points before the change that are close to the changed 3D position and performs a weighted average of the attribute information values ​​of the N 3D points. For example, in the weighted average, the 3D data encoding device determines the weights based on the distance from the changed 3D position to each of the N 3D points. Then, the 3D data encoding device determines the attribute information value of the changed 3D point based on the value obtained by the weighted average. In addition, when two or more 3D points change to the same 3D position due to quantization or other reasons, the 3D data encoding device may also assign the average of the attribute information of the two or more 3D points before the change as the attribute information value of the changed 3D point.

[0429] Next, the three-dimensional data encoding device encodes the redistributed attribute information (S3003). For example, when encoding multiple attribute information, the three-dimensional data encoding device can encode the multiple attribute information sequentially. For example, when color and reflectivity are encoded as attribute information, the three-dimensional data encoding device can generate a bitstream with the encoded result of reflectivity appended after the encoded result of color. Furthermore, the order of the multiple encoded results of attribute information appended to the bitstream is not limited to this order and can be any order.

[0430] Furthermore, the 3D data encoding device can also append information representing the beginning of the encoded data for each attribute within the bitstream to the header. This allows the 3D data decoding device to selectively decode only the attribute information that needs decoding, thus omitting the decoding of unnecessary attribute information. Therefore, the processing load of the 3D data decoding device can be reduced. Moreover, the 3D data encoding device can encode multiple attribute information in parallel and merge the encoding results into a single bitstream. This enables the 3D data encoding device to encode multiple attribute information at high speed.

[0431] Figure 42 This is a flowchart of the attribute information encoding process (S3003). First, the 3D data encoding device sets the LoD (S3011). That is, the 3D data encoding device assigns each 3D point to any one of the multiple LoDs.

[0432] Next, the 3D data encoding device begins the loop of LoD units (S3012). That is, the 3D data encoding device repeatedly performs the processing steps S3013 to S3021 for each LoD.

[0433] Next, the three-dimensional data encoding device begins the cycle of three-dimensional point units (S3013). That is, the three-dimensional data encoding device repeatedly performs the processing steps S3014 to S3020 for each three-dimensional point.

[0434] First, the 3D data encoding device searches for multiple surrounding 3D points, which exist around the object's 3D points and are used in calculating the predicted value of the object's 3D points (S3014). Next, the 3D data encoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the resulting value as the predicted value P (S3015). Next, the 3D data encoding device calculates the difference between the object's 3D point's attribute information and the predicted value, i.e., the prediction residual (S3016). Next, the 3D data encoding device calculates a quantized value by quantizing the prediction residual (S3017). Next, the 3D data encoding device performs arithmetic encoding on the quantized value (S3018).

[0435] Furthermore, the 3D data encoding device calculates the inverse quantization value by performing inverse quantization on the quantized value (S3019). Next, the 3D data encoding device generates the decoded value by adding the predicted value to the inverse quantization value (S3020). Then, the 3D data encoding device ends the loop of 3D point units (S3021). Additionally, the 3D data encoding device ends the loop of LoD units (S3022).

[0436] The following describes the three-dimensional data decoding process in the three-dimensional data decoding apparatus that decodes the bitstream generated by the aforementioned three-dimensional data encoding apparatus.

[0437] The 3D data decoding device generates decoded binary data by performing arithmetic decoding on the binarized data of attribute information within the bitstream generated by the 3D data encoding device, using the same method as the 3D data encoding device. Furthermore, in the 3D data encoding device, when the application method of arithmetic encoding is switched between the portion binarized with n bits (n-bit code) and the portion binarized with exponential Golomb (residual code), the 3D data decoding device performs decoding accordingly when applying arithmetic decoding.

[0438] For example, in the arithmetic decoding method of an n-bit code, the three-dimensional data decoding device uses a different encoding table (decoding table) for each bit during arithmetic decoding. At this time, the three-dimensional data decoding device can also change the number of encoding tables used for each bit. For example, it might use one encoding table for the first bit b0 of the n-bit code for arithmetic decoding. Furthermore, the three-dimensional data decoding device uses two encoding tables for the next bit b1. Moreover, the three-dimensional data decoding device switches the encoding table used in the arithmetic decoding of bit b1 based on the value of b0 (0 or 1). Similarly, the three-dimensional data decoding device further uses four encoding tables for the next bit b2. Furthermore, the three-dimensional data decoding device switches the encoding table used in the arithmetic decoding of bit b2 based on the values ​​of b0 and b1 (0 to 3).

[0439] Thus, when the three-dimensional data decoding device performs arithmetic decoding on each bit bn-1 of the n-bit code, it uses 2 n-1 The three-dimensional data decoding device uses an encoding table. Furthermore, it switches the encoding table based on the values ​​(occurrence patterns) of the bits preceding bit bn-1. Thus, the three-dimensional data decoding device can appropriately decode the bitstream, improving encoding efficiency, by using the appropriate encoding table for each bit.

[0440] Furthermore, the 3D data decoding device can also reduce the number of encoding tables used in each bit. For example, the 3D data decoding device can also switch between two encoding tables based on the values ​​(occurrence patterns) of the preceding m bits (m < n - 1) when performing arithmetic decoding on each bit bn-1. m The three-dimensional data decoding apparatus uses a number of encoding tables. Therefore, it can appropriately decode the bitstream with improved coding efficiency while suppressing the number of encoding tables used in each bit. Furthermore, the three-dimensional data decoding apparatus can update the occurrence probabilities of 0 and 1 in each encoding table based on the actual values ​​of the generated binarized data. Additionally, the three-dimensional data decoding apparatus can also fix the occurrence probabilities of 0 and 1 in the encoding tables of a subset of bits. This reduces the number of probability updates and thus reduces the processing load.

[0441] For example, in the case of n-bit code b0b1b2…bn-1, b0 uses a single encoding table (CTb0). b1 uses two encoding tables (CTb10, CTb11). Furthermore, the encoding table is switched based on the value of b0 (0-1). b2 uses four encoding tables (CTb20, CTb21, CTb22, CTb23). Furthermore, the encoding table is switched based on the values ​​of b0 and b1 (0-3). bn-1 uses a 2-bit encoding table. n-1 (CTbn0, CTbn1, ..., CTbn(2)) n-1 -1)). Additionally, based on the values ​​of b0b1…bn-2 (0~2) n-1-1) to switch the encoding table.

[0442] For example, Figure 43 This diagram illustrates the handling of cases where the residual code is an exponential Golomb code. For example... Figure 43 As shown, the three-dimensional data encoding device uses exponential Golbus binarization to encode the portion (residual code) which includes a prefix part and a suffix part. For example, the three-dimensional data decoding device switches the encoding table between the prefix part and the suffix part. That is, the three-dimensional data decoding device uses the encoding table for the prefix to perform arithmetic decoding on each bit contained in the prefix part, and uses the encoding table for the suffix to perform arithmetic decoding on each bit contained in the suffix part.

[0443] Furthermore, the 3D data decoding device can update the occurrence probabilities of 0 and 1 in each encoding table based on the values ​​of the binarized data generated during decoding. Alternatively, the 3D data decoding device can fix the occurrence probabilities of 0 and 1 in a certain encoding table. This reduces the number of probability updates and thus decreases the processing load. For example, the 3D data decoding device can update the occurrence probability for the prefix section and fix the occurrence probability for the suffix section.

[0444] Furthermore, the 3D data decoding device multivalues ​​the binarized data of the arithmetically decoded prediction residuals by matching it with the encoding method used in the 3D data encoding device, thereby decoding the quantized prediction residuals (unsigned integer values). The 3D data decoding device first calculates the value of the n-bit code decoded by arithmetic decoding of the binarized n-bit code. Then, the 3D data decoding device compares the value of the n-bit code with the value of R_TH.

[0445] If the value of the n-bit code matches the value of R_TH, the 3D data decoding device determines that there are bits encoded using Exponential Golombs, and performs arithmetic decoding on the binary data encoded using Exponential Golombs, i.e., the residual code. Then, based on the decoded residual code, the 3D data decoding device uses a back-calculation table representing the relationship between the residual code and the value to calculate the value of the residual code. Figure 44 This is a diagram illustrating an example of a reverse-calculation table representing the relationship between the residual code and its value. Next, the three-dimensional data decoding device obtains the multi-valued, quantized prediction residual by adding the obtained residual code value to R_TH.

[0446] On the other hand, when the value of the n-bit code is inconsistent with the value of R_TH (the value is less than R_TH), the 3D data decoding device directly determines the value of the n-bit code as the prediction residual after multi-valued quantization. Therefore, the 3D data decoding device can appropriately decode the bitstream generated in the 3D data encoding device by switching the binarization method according to the value of the prediction residual.

[0447] Furthermore, when a threshold R_TH is appended to the beginning of the bitstream, the 3D data decoding device can also decode the value of the threshold R_TH from the beginning and use the decoded threshold R_TH value to switch the decoding method. Additionally, when the 3D data decoding device appends the threshold R_TH to the beginning for each LoD, it uses the decoded threshold R_TH to switch the decoding method for each LoD.

[0448] For example, when the threshold R_TH is 63 and the decoded n-bit code value is 63, the three-dimensional data decoding device obtains the value of the remaining code by decoding the remaining code using exponential Golombian methods. For example, in Figure 44 In the example shown, the residual code is 00100, which is used as the value of the residual code to obtain 3. Next, the 3D data decoding device adds the value of the threshold R_TH (63) to the value of the residual code (3) to obtain the value of the prediction residual (66).

[0449] Furthermore, when the value of the decoded n-bit code is 32, the three-dimensional data decoding device sets the value of the n-bit code, 32, as the value of the prediction residual.

[0450] Furthermore, the 3D data decoding device transforms the decoded quantized prediction residual from unsigned integer values ​​to signed integer values ​​through a process, for example, the reverse of the process in the 3D data encoding device. Thus, when entropy encoding the prediction residual, the 3D data decoding device can appropriately decode the bitstream generated without considering the generation of negative integers. Alternatively, the 3D data decoding device does not necessarily need to transform unsigned integer values ​​to signed integer values; for example, it can decode the signed bits even when decoding a bitstream generated by separately entropy encoding the signed bits.

[0451] The 3D data decoding device decodes the quantized prediction residual, which has been transformed into signed integer values, through inverse quantization and reconstruction, thereby generating a decoded value. Furthermore, the 3D data decoding device uses the generated decoded value for subsequent predictions of 3D points of the decoded object. Specifically, the 3D data decoding device calculates the inverse quantization value by multiplying the quantized prediction residual by the decoded quantization scale, and adds the inverse quantization value to the prediction value to obtain the decoded value.

[0452] The decoded unsigned integer value (unsigned quantized value) is transformed into a signed integer value through the following processing. If the LSB (least significant bit) of the decoded unsigned integer value a2u is 1, the 3D data decoding device sets the signed integer value a2q to -((a2u+1)>>1). If the LSB of the unsigned integer value a2u is not 1, the 3D data decoding device sets the signed integer value a2q to (a2u>>1).

[0453] Similarly, when the LSB of the decoded unsigned integer value b2u is 1, the 3D data decoding device sets the signed integer value b2q to -((b2u+1)>>1). When the LSB of the unsigned integer value n2u is not 1, the 3D data decoding device sets the signed integer value b2q to (b2u>>1).

[0454] Furthermore, the details of the inverse quantization and reconstruction processing performed by the 3D data decoding device are the same as those of the inverse quantization and reconstruction processing in the 3D data encoding device.

[0455] The following describes the processing flow in the 3D data decoding device. Figure 45 This is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding device. First, the three-dimensional data decoding device decodes the position information (geometry) from the bitstream (S3031). For example, the three-dimensional data decoding device uses an octree representation for decoding.

[0456] Next, the 3D data decoding device decodes attribute information from the bitstream (S3032). For example, when decoding multiple attribute information, the 3D data decoding device can decode the multiple attribute information sequentially. For example, when decoding color and reflectance as attribute information, the 3D data decoding device decodes the color encoding result and the reflectance encoding result in the order appended to the bitstream. For example, if the reflectance encoding result is appended after the color encoding result in the bitstream, the 3D data decoding device decodes the color encoding result and then decodes the reflectance encoding result. Furthermore, the 3D data decoding device can decode the attribute information encoding results appended to the bitstream in any order.

[0457] Furthermore, the 3D data decoding device can also obtain the start information of the attribute information representation encoded data within the bitstream by decoding the header, etc. Therefore, the 3D data decoding device can selectively decode the attribute information that needs to be decoded, thus omitting the decoding processing of attribute information that does not need to be decoded. This reduces the processing load of the 3D data decoding device. Additionally, the 3D data decoding device can decode multiple attribute information in parallel and merge the decoding results into a single 3D point group. Therefore, the 3D data decoding device can decode multiple attribute information at high speed.

[0458] Figure 46 This is a flowchart of the attribute information decoding process (S3032). First, the 3D data decoding device sets the Location of Dimensions (LoD) (S3041). That is, the 3D data decoding device assigns multiple 3D points with decoded location information to any one of the multiple LoDs. For example, this assignment method is the same as the assignment method used in the 3D data encoding device.

[0459] Next, the 3D data decoding device begins the loop of LoD units (S3042). That is, the 3D data decoding device repeatedly performs the processing steps S3043 to S3049 for each LoD.

[0460] Next, the 3D data decoding device begins the loop of 3D point units (S3043). That is, the 3D data decoding device repeatedly performs the processing steps S3044 to S3048 for each 3D point.

[0461] First, the 3D data decoding device searches for multiple surrounding points, which exist around the object's 3D points and are used in calculating the predicted value of the object's 3D points (S3044). Next, the 3D data decoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the resulting value as the predicted value P (S3045). Furthermore, these processes are the same as those in the 3D data encoding device.

[0462] Next, the 3D data decoding device performs arithmetic decoding on the quantized values ​​from the bitstream (S3046). Furthermore, the 3D data decoding device calculates the inverse quantized value by performing inverse quantization on the decoded quantized value (S3047). Next, the 3D data decoding device generates the decoded value by adding the predicted value to the inverse quantized value (S3048). Next, the 3D data decoding device ends the loop for 3D point units (S3049). Furthermore, the 3D data decoding device ends the loop for LoD units (S3050).

[0463] Next, the structure of the three-dimensional data encoding device and the three-dimensional data decoding device of this embodiment will be described. Figure 47This is a block diagram showing the structure of the three-dimensional data encoding device 3000 according to this embodiment. The three-dimensional data encoding device 3000 includes a position information encoding unit 3001, an attribute information redistribution unit 3002, and an attribute information encoding unit 3003.

[0464] The attribute information encoding unit 3003 encodes the geometry of the multiple three-dimensional points contained in the input point group. The attribute information redistribution unit 3002 uses the encoding and decoding results of the geometry to redistribute the attribute information values ​​of the multiple three-dimensional points contained in the input point group. The attribute information encoding unit 3003 encodes the redistributed attribute information. Furthermore, the three-dimensional data encoding apparatus 3000 generates a bitstream containing the encoded geometry and the encoded attribute information.

[0465] Figure 48 This is a block diagram showing the structure of the three-dimensional data decoding device 3010 according to this embodiment. The three-dimensional data decoding device 3010 includes a position information decoding unit 3011 and an attribute information decoding unit 3012.

[0466] The position information decoding unit 3011 decodes the position information (geometry) of multiple three-dimensional points from the bitstream. The attribute information decoding unit 3012 decodes the attribute information (attribute) of multiple three-dimensional points from the bitstream. In addition, the three-dimensional data decoding device 3010 generates an output point group by combining the decoded position information and the decoded attribute information.

[0467] (Implementation Method 4)

[0468] As another example of using LoD information to encode the attribute information of 3D points, a method for sequentially encoding multiple 3D points starting from the 3D points contained in the upper layer of the LoD is explained. For example, when the 3D data encoding device calculates the predicted values ​​of the attribute values ​​(attribute information) of the 3D points contained in LoDn, it can also use flags to switch which LoD's attribute values ​​of 3D points can be referenced. For example, the 3D data encoding device generates information indicating whether referencing other 3D points within the same LoD as the object's 3D points is permitted, namely EnableReferringSameLoD (same-layer reference permission flag). For example, when EnableReferringSameLoD is 1, referencing within the same LoD is permitted; when EnableReferringSameLoD is 0, referencing within the same LoD is prohibited.

[0469] For example, a 3D data encoding device selects surrounding 3D points based on EnableReferringSameLoD, calculates the average of attribute values ​​of up to N predefined surrounding 3D points, and thereby generates a predicted value for the attribute information of the object's 3D point. Furthermore, the 3D data encoding device appends the value of N to the header of the bitstream. Alternatively, the 3D data encoding device can append the value of N to each 3D point that generates the predicted value. Thus, an appropriate N can be selected for each 3D point that generates the predicted value, thereby improving the accuracy of the predicted value and reducing the prediction residual.

[0470] Alternatively, the 3D data encoding device can append the value of N to the beginning of the bitstream, fixing the value of N within the bitstream. This eliminates the need to encode or decode the value of N for each 3D point, reducing processing complexity.

[0471] Alternatively, the 3D data encoding device can encode the information representing the value of N separately for each LoD. Thus, by selecting an appropriate value of N for each LoD, encoding efficiency can be improved. Furthermore, the 3D data encoding device can also calculate the predicted value of the attribute information of a 3D point based on the weighted average of the attribute information of N surrounding 3D points. For example, the 3D data encoding device uses the distance information between the object's 3D point and each of the N surrounding 3D points to calculate the weights.

[0472] Thus, EnableReferringSameLoD indicates whether referencing of 3D points within the same LoD is permitted. For example, a value of 1 indicates referencing is allowed, and a value of 0 indicates no referencing is allowed. Alternatively, with a value of 1, already encoded or decoded 3D points within the same LoD can be referenced.

[0473] Figure 49 This is a diagram illustrating an example of referencing relationships when EnableReferringSameLoD = 0. The predicted value of a point P contained in the LoDN is generated using the reconstructed value P' contained in the LoDN' (N' < N) above the LoDN. Here, the reconstructed value P' is the encoded and decoded attribute value (attribute information). For example, the reconstructed value P' based on distance-based neighboring points can be used.

[0474] In addition, Figure 49 In the example shown, the predicted value of b2 is generated using any one of the attribute values ​​a0, a1, and a2. Even after b0 and b1 have been encoded and decoded, referencing b0 and b1 is prohibited.

[0475] Therefore, the 3D point data encoding device and the 3D data decoding device can generate the predicted value of b2 without waiting for the encoding or decoding processing of b0 and b1 to be completed. That is, the 3D point data encoding device and the 3D data decoding device can calculate multiple predicted values ​​of attribute values ​​for multiple 3D points within the same LoD in parallel, thus reducing processing time.

[0476] Figure 50 This is a diagram illustrating an example of referencing relationships when EnableReferringSameLoD = 1. The predicted value of a point P contained in the LoDN is generated using the reconstructed value P' contained in the LoDN' (N'≤N) at the same or higher level as the LoDN. Here, the reconstructed value P' is the encoded and decoded attribute value (attribute information). For example, the reconstructed value P' of neighboring points based on distance can be used.

[0477] In addition, Figure 50 In the example shown, the predicted value of b2 is generated using any one of the attribute values ​​a0, a1, a2, b0, and b1. That is, it can be referenced after b0 and b1 have been encoded and decoded.

[0478] Therefore, the 3D data encoding device can use a large amount of attribute information from neighboring 3D points to generate the predicted value of b2. As a result, the prediction accuracy is improved and the encoding efficiency is enhanced.

[0479] The following describes a method for limiting the number of searches when selecting N 3D points for generating predicted values ​​of attribute information for 3D points. This reduces the processing load.

[0480] For example, define SearchNumPoint (search point count information). SearchNumPoint represents the number of searches required to select N 3D points for prediction from a group of 3D points within the LoD. For instance, a 3D data encoding device can also select the same number of 3D points as represented by SearchNumPoint from a total of T 3D points contained in the LoD, and then select N 3D points for prediction from the selected 3D points. Thus, the 3D data encoding device does not need to search all T 3D points contained in the LoD, thereby reducing processing load.

[0481] Additionally, the 3D data encoding device can switch the selection method of the SearchNumPoint value based on the position of the referenced LoD. An example is shown below.

[0482] For example, when the reference LoD used as a reference is at a higher level than the LoD to which the object 3D point belongs, the 3D data encoding device searches for a 3D point A among the 3D points contained in the reference LoD that is closest to the object 3D point. Then, the 3D data encoding device selects 3D points that are adjacent to 3D point A by a number represented by SearchNumPoint. Thus, the 3D data encoding device can efficiently search for higher-level 3D points that are close to the object 3D point, thereby improving prediction efficiency.

[0483] For example, when the reference LoD and the object 3D point belong to the same LoD, the 3D data encoding device selects a number of 3D points, represented by SearchNumPoint, that were encoded and decoded before the object 3D point. Alternatively, the 3D data encoding device selects a number of 3D points, represented by SearchNumPoint, that were encoded and decoded immediately before the object 3D point.

[0484] Therefore, the 3D data encoding device can select a number of 3D points represented by SearchNumPoint with low processing power. Furthermore, the 3D data encoding device can also select a 3D point B that is closer to the object 3D point from among the 3D points that were encoded and decoded before the object 3D point, and then select the number of 3D points represented by SearchNumPoint that are adjacent to 3D point B. Thus, the 3D data encoding device can efficiently search for 3D points in the same layer that are close to the object 3D point, thereby improving prediction efficiency.

[0485] Furthermore, when selecting N 3D points for prediction from a set of 3D points represented by SearchNumPoint, the 3D data encoding device can, for example, select the N nearest 3D points to the target 3D point. This improves prediction accuracy and thus enhances encoding efficiency.

[0486] Alternatively, a SearchNumPoint can be prepared for each LoD, and the number of searches can be changed for each LoD. Figure 51 This is a graph illustrating an example of setting the number of searches for each LoD. For example, as shown... Figure 51 As shown, SearchNumPoint[LoD0] = 3 is defined for LoD0 and SearchNumPoint[LoD1] = 2 for LoD1. In this way, by changing the number of searches for each LoD, a balance between processing volume and encoding efficiency can be achieved.

[0487] exist Figure 51In the example shown, a0, a1, and a2 are selected from LoD0 as the 3D points used for predicting b2, and b0 and b1 are selected from LoD1. N 3D points are selected from the selected a0, a1, a2, b0, and b1, and the predicted value is generated using the selected N 3D points.

[0488] Additionally, the predicted value of a point P contained in a LoDN is generated using the reconstructed value P' contained in a LoDN' (N'≤N) at the same or higher level as the LoDN. Here, the reconstructed value P' is the encoded and decoded attribute value (attribute information). For example, the reconstructed value P' of neighboring points based on distance can be used.

[0489] Additionally, SearchNumPoint can also represent the total number of searches across all LoDs. For example, with SearchNumPoint = 5, if 3 searches are performed in LoD0, the remaining 2 searches can be performed in LoD1. This ensures the worst-case scenario for the number of searches, thus stabilizing the processing time.

[0490] The 3D data encoding device can also append the SearchNumPoint to the header. Therefore, the 3D data decoding device, by decoding the SearchNumPoint from the header, can generate the same predicted value as the 3D data encoding device, and can appropriately decode the bitstream. Furthermore, the SearchNumPoint does not necessarily need to be appended to the header; for example, the value of the SearchNumPoint can be specified by a profile or level of a standard. This allows for a reduction in the number of bits in the header.

[0491] When calculating the predicted values ​​of the attribute values ​​of the 3D points contained in LoDn, the following EnableReferenceLoD can also be defined. Thus, the 3D data encoding device and the 3D data decoding device can refer to EnableReferenceLoD (referencing permission layer information) to determine which LoD contains the attribute values ​​of the 3D points that can be referenced.

[0492] EnableReferenceLoD indicates whether, when an object's 3D point belongs to LoDn, references to 3D points at a level higher than LoD(n-EnableReferenceLoD) are permitted. For example, if EnableReferenceLoD = 0, references to 3D points in LoDn and higher levels are permitted. If EnableReferenceLoD = 1, references to 3D points in higher levels (LoDn-1 and above) are permitted. Furthermore, if EnableReferenceLoD = 2, references to 3D points in higher levels (LoDn-2 and above) are permitted. Thus, by setting the value of EnableReferenceLoD, the levels of LoDs that can be referenced can be determined, thereby achieving a balance between encoding efficiency and processing time by controlling the levels that can be processed in parallel. Alternatively, references can be made to 3D points that have already been encoded or decoded within each level.

[0493] Figure 52 This is a diagram showing the reference relationships when EnableReferenceLoD = 0. For example... Figure 52 As shown, for example, any one of the attribute values ​​a0, a1, a2, b0, b1, b2, c0, and c1 can be used to generate the predicted value of c2. Alternatively, each 3D point can be referenced after it has been encoded or decoded. Therefore, by using the attribute information of a larger number of adjacent 3D points to generate the predicted value of c2, the prediction accuracy can be improved, and the encoding efficiency can be increased.

[0494] Figure 53 This is a diagram showing the reference relationships when EnableReferenceLoD = 1. For example... Figure 53 As shown, for example, any one of the attribute values ​​a0, a1, a2, b0, b1, and b2 can be used to generate the predicted value of c2. c0 and c1 are prohibited from reference even after encoding or decoding. Therefore, the 3D data encoding device and the 3D data decoding device can generate the predicted value of c2 without waiting for the encoding or decoding of c0 and c1 to complete. That is, the 3D data encoding device and the 3D data decoding device can calculate the predicted values ​​of attribute values ​​for 3D points within the same LoD in parallel, thus reducing processing time.

[0495] Figure 54 This is a diagram showing the reference relationships when EnableReferenceLoD=2. For example... Figure 54As shown, for example, any one of the attribute values ​​a0, a1, and a2 can be used to generate the predicted value of c2. c0, c1, b0, b1, and b2 are prohibited from reference even after encoding or decoding. Therefore, the 3D data encoding device and the 3D data decoding device can generate the predicted value of c2 without waiting for the encoding or decoding processing of c0, c1, b0, b1, and b2 to complete. That is, the 3D data encoding device and the 3D data decoding device can calculate the predicted values ​​of the attribute values ​​for the 3D points of LoD1 and LoD2 in parallel, thus reducing processing time.

[0496] Figure 55 This is a diagram illustrating the syntax of the attribute header in this implementation. The attribute header is the header information of the attribute information. For example... Figure 55 As shown, the attribute header contains the same-level referencing permission flag (EnableReferringSameLoD), level number information (NumLoD), search point number information (SearchNumPoint[i]), and surrounding point number information (NumNeighborPoint[i]).

[0497] EnableReferringSameLoD indicates whether referencing information about 3D points within the same LoD as the object's 3D point is permitted. For example, a value of 1 indicates referencing is allowed, and a value of 0 indicates no referencing (referencing is prohibited). Alternatively, with a value of 1, referencing already encoded or decoded 3D points within the same LoD is permitted. NumLoD indicates the number of LoD levels used.

[0498] SearchNumPoint[i] represents the number of searches required to select N 3D points for prediction from the group of 3D points within the i-th LoD. For example, the 3D data encoding device can also select the same number of 3D points as represented by SearchNumPoint from the total T 3D points contained in the LoD, and then select N 3D points for prediction from the selected 3D points. Therefore, the 3D data encoding device does not need to search all T 3D points contained in the LoD, thus reducing the processing load.

[0499] NumNeighborPoint[i] represents the upper limit N of the number of surrounding points used in generating the predicted value of a 3D point belonging to level i. The 3D data encoding device can also use M surrounding 3D points to calculate the predicted value if the number of surrounding 3D points M is less than NumNeighborPoint[i] (M < NumNeighborPoint[i]). Furthermore, the 3D data encoding device can append one NumNeighborPoint used across all LoDs to the head if it is not necessary to divide the value of NumNeighborPoint[i] in each LoD.

[0500] The 3D data encoding device can also entropy encode EnableReferringSameLoD and SearchNumPoint and append them to the header. For example, the 3D data encoding device performs arithmetic encoding after binarizing each value. Alternatively, the 3D data encoding device can also encode with a fixed length to reduce processing load.

[0501] Furthermore, EnableReferringSameLoD and SearchNumPoint do not necessarily need to be appended to the header; their values ​​can be specified through standards or levels. This allows for a reduction in the number of bits in the header.

[0502] Furthermore, the 3D data encoding device can also append headers to WLD, SPC, or volume units to switch between EnableReferringSameLoD and SearchNumPoint in WLD, SPC, or volume units. Additionally, the 3D data encoding device can append information indicating whether references at the same level are permitted to the bitstream based on the hierarchy, and switch whether references at the same level are permitted based on the hierarchy.

[0503] The value of EnableReferringSameLoD can also be constrained based on the encoding method of the attribute values. For example, when using encoding method L, the encoding or decoding processing of 3D points within the same layer is not yet complete, so they cannot be referenced in the generation of prediction values. Encoding method L temporarily calculates all the prediction residuals of the attribute values ​​of the 3D points within the lower layer containing the LoD and feeds these prediction residuals back to the upper layer for encoding. When using such encoding method L, the value of EnableReferringSameLoD can also be constrained to 0. Furthermore, when using such encoding method L, if the value of EnableReferringSameLoD is 1, the 3D data decoding device can also determine that it is a standard similarity error (conformance error).

[0504] Furthermore, the 3D data encoding device can also append information indicating whether encoding mode L has been used to the bitstream. If encoding mode L has not been used, the value of EnableReferringSameLoD is appended to the bitstream; if encoding mode L has been used, EnableReferringSameLoD is not appended. Thus, the 3D data decoding device, by decoding the information indicating whether encoding mode L has been used from the beginning, can determine whether EnableReferringSameLoD has been appended to the bitstream and can correctly decode the bitstream. Additionally, if encoding mode L has not been used, the 3D data decoding device can also decode the EnableReferringSameLoD in the header; if encoding mode L has been used, the value of EnableReferringSameLoD is estimated to be 0.

[0505] Alternatively, the 3D data encoding device may not append EnableReferringSameLoD to the header. In this case, the 3D data decoding device may estimate the value of EnableReferringSameLoD as 0 if the bitstream of the object being processed is a bitstream encoded in encoding mode L, and otherwise estimate the value of EnableReferringSameLoD as 1 for decoding.

[0506] Figure 56 This is a diagram illustrating another syntax example of the attribute header in this implementation. Figure 56 The attribute information header shown is relative to Figure 55 The attribute header shown replaces the same-level reference license flag (EnableReferringSameLoD) with reference license level information (EnableReferenceLoD). Additionally, the meanings of other information are the same as... Figure 55 same.

[0507] EnableReferenceLoD indicates whether references to 3D points at levels above LoD(n-EnableReferenceLoD) are permitted when an object's 3D point belongs to LoDn. For example, if EnableReferenceLoD=0, references to 3D points contained in LoDn and higher levels are permitted. If EnableReferenceLoD=1, references to 3D points contained in higher levels above LoDn-1 are permitted. Furthermore, if EnableReferenceLoD=2, references to 3D points contained in higher levels above LoDn-2 are permitted. Thus, the levels of LoDs that can be referenced can be set according to the value of EnableReferenceLoD, thereby achieving a balance between encoding efficiency and processing time by controlling the levels that can be processed in parallel. Alternatively, references can be made to 3D points that have already been encoded or decoded within each layer.

[0508] The 3D data encoding device can also entropy encode EnableReferenceLoD and SearchNumPoint and append them to the header. For example, the 3D data encoding device performs arithmetic encoding after binarizing each value. Alternatively, the 3D data encoding device can encode with a fixed length to reduce processing load.

[0509] Furthermore, EnableReferenceLoD and SearchNumPoint do not necessarily need to be appended to the header; their values ​​can be specified through standards or levels. This reduces the number of bits in the header.

[0510] In addition, in order to switch between WLD, SPC, or volume units, the 3D data encoding device can also attach EnableReferenceLoD and SearchNumPoint to the WLD, SPC, or volume header.

[0511] The value of EnableReferenceLoD can also be constrained based on the encoding method of the attribute value. For example, when using the encoding method L mentioned above, the value of EnableReferenceLoD can be constrained to be 1 or higher. Furthermore, when using encoding method L, if the value of EnableReferenceLoD is 0, the 3D data decoding device can also determine that it is a standard similarity error.

[0512] Figure 57This is a flowchart of the three-dimensional data encoding process in this embodiment. First, the three-dimensional data encoding device encodes the location information (geometry) (S6901). For example, the three-dimensional data encoding device uses an octree representation for encoding.

[0513] Next, the 3D data encoding device transforms the attribute information (S6902). For example, after encoding the position information, if the position of a 3D point changes due to quantization or other reasons, the 3D data encoding device redistributes the original attribute information of the 3D point after the change. Alternatively, the 3D data encoding device can interpolate the attribute information values ​​based on the amount of position change to redistribute the information. For example, the 3D data encoding device detects N 3D points before the change that are close to the changed 3D position, and calculates a weighted average of the attribute information values ​​of these N 3D points based on the distance from the changed 3D position to each of the N 3D points, setting the resulting value as the attribute information value of the changed 3D point. Furthermore, if two or more 3D points change to the same 3D position due to quantization or other reasons, the 3D data encoding device can also assign the average of the attribute information of the two or more 3D points before the change as the attribute information value of the changed 3D point.

[0514] Next, the three-dimensional data encoding device encodes the attribute information (S6903). For example, when encoding multiple attribute information, the three-dimensional data encoding device can encode the multiple attribute information sequentially. For example, when color and reflectivity are encoded as attribute information, the three-dimensional data encoding device can generate a bitstream with the encoded result of reflectivity appended after the encoded result of color. Furthermore, the order of the multiple encoded results of attribute information appended to the bitstream is not limited to this order and can be any order.

[0515] Furthermore, the 3D data encoding device can also append information representing the beginning of the encoded data for each attribute within the bitstream to the header. This allows the 3D data decoding device to selectively decode only the attribute information that needs decoding, thus omitting the decoding of unnecessary attribute information. Therefore, the processing load of the 3D data decoding device can be reduced. Moreover, the 3D data encoding device can encode multiple attribute information in parallel and merge the encoding results into a single bitstream. This enables the 3D data encoding device to encode multiple attribute information at high speed.

[0516] Figure 58 This is a flowchart of the attribute information encoding process (S6903). First, the 3D data encoding device sets the LoD (S6911). That is, the 3D data encoding device assigns each 3D point to one of the multiple LoDs.

[0517] Next, the three-dimensional data encoding device begins the loop of LoD units (S6912). That is, the three-dimensional data encoding device repeatedly performs the processing steps S6913 to S6921 for each LoD.

[0518] Next, the three-dimensional data encoding device begins the cycle of three-dimensional point units (S6913). That is, the three-dimensional data encoding device repeatedly performs the processing steps S6914 to S6920 for each three-dimensional point.

[0519] First, the 3D data encoding device searches for multiple surrounding points, which exist around the object's 3D points and are used in calculating the predicted value of the object's 3D points (S6914). Next, the 3D data encoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the resulting value as the predicted value P (S6915). Next, the 3D data encoding device calculates the difference between the object's 3D point's attribute information and the predicted value, i.e., the prediction residual (S6916). Next, the 3D data encoding device calculates a quantized value by quantizing the prediction residual (S6917). Next, the 3D data encoding device performs arithmetic encoding on the quantized value (S6918).

[0520] Furthermore, the 3D data encoding device calculates the inverse quantization value by performing inverse quantization on the quantized value (S6919). Next, the 3D data encoding device generates the decoded value by adding the predicted value to the inverse quantization value (S6920). Then, the 3D data encoding device ends the loop for 3D point units (S6921). Additionally, the 3D data encoding device ends the loop for LoD units (S6922).

[0521] Figure 59 This is a flowchart of the three-dimensional data decoding process in this embodiment. First, the three-dimensional data decoding device decodes the position information (geometry) from the bitstream (S6541). For example, the three-dimensional data decoding device uses an octree representation for decoding.

[0522] Next, the 3D data decoding device decodes the attribute information from the bitstream (S6932). For example, when decoding multiple attribute information, the 3D data decoding device can decode the multiple attribute information sequentially. For example, when decoding color and reflectance as attribute information, the 3D data decoding device decodes the color encoding result and the reflectance encoding result in the order appended to the bitstream. For example, if the reflectance encoding result is appended after the color encoding result in the bitstream, the 3D data decoding device decodes the color encoding result and then decodes the reflectance encoding result. Furthermore, the 3D data decoding device can decode the attribute information encoding results appended to the bitstream in any order.

[0523] Furthermore, the 3D data decoding device can also obtain information about the start of the encoded data representing each attribute information within the bitstream by means of a header. Therefore, the 3D data decoding device can selectively decode only the attribute information that needs to be decoded, thus omitting the decoding processing of attributes that do not need to be decoded. This reduces the processing load of the 3D data decoding device. Additionally, the 3D data decoding device can decode multiple attribute information in parallel and merge the decoding results into a single 3D point group. Therefore, the 3D data decoding device can decode multiple attribute information at high speed.

[0524] Figure 60 This is a flowchart of the attribute information decoding process (S6932). First, the 3D data decoding device sets the LoD (S6941). That is, the 3D data decoding device assigns multiple 3D points with decoded position information to one of the multiple LoDs. For example, this assignment method is the same as the assignment method used in the 3D data encoding device.

[0525] Next, the 3D data decoding device begins the loop of LoD units (S6942). That is, the 3D data decoding device repeatedly performs the processing steps S6943 to S6949 for each LoD.

[0526] Next, the 3D data decoding device begins the loop of 3D point units (S6943). That is, the 3D data decoding device repeatedly performs the processing steps S6944 to S6948 for each 3D point.

[0527] First, the 3D data decoding device searches for multiple surrounding points, which exist around the object's 3D points and are used in calculating the predicted value of the object's 3D points (S6944). Next, the 3D data decoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the resulting value as the predicted value P (S6945). Furthermore, these processes are the same as those in the 3D data encoding device.

[0528] Next, the 3D data decoding device performs arithmetic decoding on the quantized values ​​from the bitstream (S6946). Furthermore, the 3D data decoding device calculates the inverse quantized value by performing inverse quantization on the decoded quantized value (S6947). Next, the 3D data decoding device generates the decoded value by adding the predicted value to the inverse quantized value (S6948). Next, the 3D data decoding device ends the loop for 3D point units (S6949). Furthermore, the 3D data decoding device ends the loop for LoD units (S6950).

[0529] Figure 61This is a flowchart of the surrounding point search process (S6914). First, the 3D data encoding device selects SearchNumPoint[upper LoD] 3D points contained in the LoD layer above the layer to which the object 3D point belongs, and calculates N 3D points for generating the prediction value based on the selected 3D points (S6961). Next, the 3D data encoding device determines whether EnableReferringSameLoD is 1 (S6962). If EnableReferringSameLoD is 1 ("Yes" in S6962), the 3D data encoding device selects SearchNumPoint[same LoD] 3D points contained in the LoD layer of the same layer as the layer to which the object 3D point belongs, and updates the N 3D points for generating the prediction value (S6963). For example, the 3D data encoding device calculates N 3D points based on the SearchNumPoint[upper LoD] 3D points selected in step S6961 and the SearchNumPoint[same LoD] 3D points selected in step S6963.

[0530] On the other hand, if EnableReferringSameLoD is 0 (No in S6962), the N three-dimensional points selected in step S6961 are used directly.

[0531] Figure 62 This is a flowchart illustrating another example of the surrounding point search process (S6914). First, the 3D data encoding device determines whether EnableReferringSameLoD is 1 (S6971). If EnableReferringSameLoD is 1 ("Yes" in S6971), the 3D data encoding device selects SearchNumPoints [same LoD] 3D points contained in the same LoD layer as the layer to which the object 3D point belongs, and calculates N 3D points for generating prediction values ​​based on the selected 3D points (S6972). Next, the 3D data encoding device selects SearchNumPoints [higher LoD] 3D points contained in the LoD layer above the layer to which the object 3D point belongs, and updates the N 3D points for generating prediction values ​​(S6973). For example, the 3D data encoding device calculates N 3D points based on the SearchNumPoints [same LoD] 3D points selected in step S6972 and the SearchNumPoints [higher LoD] 3D points selected in step S6973.

[0532] On the other hand, when EnableReferringSameLoD is 0 (No in S6971), the three-dimensional data encoding device selects SearchNumPoint[higher LoD] three-dimensional points contained in the LoD layer above the layer to which the object's three-dimensional point belongs, and calculates N three-dimensional points for generating the prediction value based on the selected three-dimensional points (S6973).

[0533] Figure 63 This is a flowchart illustrating another example of the surrounding point search process (S6914). First, the 3D data encoding device selects SearchNumPoint 3D points contained in the LoD (n-EnableReferenceLoD) and above, and calculates N 3D points for generating prediction values ​​based on the selected 3D points (S6981).

[0534] Furthermore, the surrounding point search processing (S6944) in the three-dimensional data decoding device is the same as the surrounding point search processing (S6914) in the three-dimensional data encoding device.

[0535] (Implementation Method 5)

[0536] In this embodiment, another example of encoding attribute information of 3D points using LoD (Level of Detail) is described. This embodiment describes a method using EnableReferringSameLoD (same-level reference permission flag) as information indicating whether a 3D point belonging to the same LoD level as the object's 3D point is referenced, and ReferringSameLoDminLayer (same-level reference permission lower limit layer information) as information indicating that EnableReferringSameLoD is a valid LoD level. Furthermore, hereinafter, other 3D points referencing the same LoD level as the object's 3D point will also be referred to as same-level references.

[0537] EnableReferringSameLoD indicates whether referencing of 3D points within the same LoD is permitted. A value of 1 indicates referencing is allowed, and a value of 0 indicates no referencing is allowed. Alternatively, in the case of a value of 1, 3D points within the same LoD that have already been encoded or decoded can be referenced.

[0538] ReferringSameLoDminLayer represents the threshold (lower limit) of the LoD level that enables ReferringSameLoD information (i.e., allows same-level referencing). For example, if the lowest LoD level is LoD0, and the levels above it are LoD1 and LoD2, if ReferringSameLoDminLayer = 0, then the ReferringSameLoD information is valid at levels above LoD0; if ReferringSameLoDminLayer = 1, then the ReferringSameLoD information is valid at levels above LoD1; and if ReferringSameLoDminLayer = 2, then the ReferringSameLoD information is valid at levels above LoD2.

[0539] That is, if ReferringSameLoDminLayer = N, then the information from EnableReferringSameLoD is valid at LoDN levels and above. Therefore, when EnableReferringSameLoD = 1, the information from EnableReferringSameLoD is invalid up to LoD level N-1 (i.e., same-layer referencing is prohibited), allowing parallel computation of predicted attribute information (also called attribute values) for multiple 3D points belonging to LoD levels N-1 and below. Furthermore, at LoD levels N and above, the information from EnableReferringSameLoD is valid, allowing the computation of predicted attribute information for 3D points belonging to LoD levels N and above using attribute information from a large number of neighboring 3D points at LoD level N. As a result, a balance can be achieved between reduced processing time and improved coding efficiency.

[0540] Furthermore, when EnableReferringSameLoD=0, the 3D data encoding device can also omit the ReferringSameLoDminLayer from the header. This reduces the amount of encoding required in the header.

[0541] Furthermore, the above example illustrates that if ReferringSameLoDminLayer = N, then the information that enables ReferringSameLoD at LoDN levels and above is valid, but this is not necessarily the case. For example, it could also be that if ReferringSameLoDminLayer = N, then the information that enables ReferringSameLoD at LoDN levels and below is valid, while the information that enables ReferringSameLoD at LoDN+1 levels and above is invalid. Therefore, it is possible to compute the predicted values ​​of attribute information for 3D points belonging to LoD levels N+1 and above in parallel. Additionally, by using the attribute information of a large number of neighboring 3D points at LoD level N to compute the predicted values ​​of attribute information for 3D points belonging to LoD levels below LoD level N, coding efficiency can be improved.

[0542] Additionally, the value of ReferringSameLoDminLayer can be above 0 and below maxLoD-1, provided the maximum number of LoD levels is maxLoD. Alternatively, if ReferringSameLoDminLayer = 0, the EnableReferringSameLoD information is valid across all LoD levels. Also, if ReferringSameLoDminLayer = maxLoD-1, the EnableReferringSameLoD information is valid only at the topmost level of the generated LoD.

[0543] Figure 64 This diagram represents the referencing relationships when EnableReferringSameLoD = 1 and ReferringSameLoDminLayer = 0. In this case, EnableReferringSameLoD = 1 is effective above LoD0 level, allowing same-level referencing for attribute information of 3D points contained in all levels. For example, the predicted value of c2 can be generated using any one of the attribute information of b0, b1, and b2 belonging to the higher layer and c0 and c1 belonging to the same layer. Alternatively, c0 and c1 can be referenced if they have already been encoded or decoded. Thus, a larger amount of attribute information of adjacent 3D points is used to generate the predicted value of c2, thereby improving prediction accuracy and encoding efficiency.

[0544] Figure 65This diagram represents the referencing relationships when EnableReferringSameLoD = 1 and ReferringSameLoDminLayer = 1. In this case, EnableReferringSameLoD = 1 is effective at LoD1 level and above, and same-layer referencing is permitted for both LoD1 and LoD2. For example, the predicted value of b2 can be generated using any one of the attribute information from the higher-level layers a0, a1, and a2, and the same-layer layers b0 and b1. Alternatively, b0 and b1 can be referenced after they have been encoded or decoded. Therefore, by using a larger amount of attribute information from adjacent 3D points to generate the predicted value of b2, the prediction accuracy is improved, thus enhancing coding efficiency.

[0545] On the other hand, same-layer referencing is prohibited for LoD0. For example, the predicted value of c2 can be generated using any one of the attribute information from the higher-level layers b0, b1, and b2. Furthermore, even if c0 and c1 are encoded or decoded, referencing is still prohibited. Therefore, the 3D data encoding or decoding device can generate the predicted value of c2 without waiting for the encoding or decoding of c0 and c1 to complete. That is, the 3D data encoding or decoding device can compute the predicted values ​​of attribute information for multiple 3D points within the same LoD in parallel, thus reducing processing time.

[0546] Furthermore, in the top layer of LoD, the 3D data encoding or decoding device can make the information of EnableReferringSameLoD valid regardless of the value of ReferringSameLoDminLayer. That is, in the top layer of LoD, when EnableReferringSameLoD = 1, the 3D data encoding or decoding device can also permit same-layer referencing without being limited to the value of ReferringSameLoDminLayer. For example, the 3D data encoding or decoding device can also determine whether same-layer referencing is permissible in the calculation of the predicted values ​​of attribute information of 3D points belonging to the LoDN level using the following conditions.

[0547] (1) Same-layer referencing is permitted when EnableReferringSameLoD == 1 and LoDN is the topmost layer, or (2) when EnableReferringSameLoD == 1 and LoDN >= ReferringSameLoDminLayer. In all other cases, same-layer referencing is prohibited.

[0548] Figure 66This is a diagram showing the reference relationships when EnableReferringSameLoD = 1 and ReferringSameLoDminLayer = LoDmax - 1.

[0549] At the top layer of LoD, the information for EnableReferringSameLoD is effective regardless of the value of ReferringSameLoDminLayer. Figure 66 In the example shown, the maximum number of LoD levels is LoDmax, and the generated LoD results do not reach LoDmax-1 at the top; the top level is LoD2. Here, the maximum number of LoD levels is the maximum configurable LoD value, and the top level is the top level of the actually generated LoD levels.

[0550] In this case, the EnableReferringSameLoD information is valid in the top-level LoD2, allowing same-level referencing. Therefore, when EnableReferringSameLoD = 1, the EnableReferringSameLoD information becomes valid at least in level 1, thus improving coding efficiency.

[0551] exist Figure 66 In the example, either the attribute information of a0 or a1 is used to generate the predicted value of a2. Alternatively, a0 and a1 can be referenced after they have been encoded or decoded. This allows the generation of the predicted value of a2 using the attribute information of a larger number of adjacent 3D points, thus improving prediction accuracy and coding efficiency. Especially at the top layer, where there are no 3D points from the upper layer, it is impossible to reference them. Therefore, allowing same-layer referencing has a high potential to improve coding efficiency.

[0552] Additionally, constraints can be imposed on the values ​​of EnableReferringSameLoD and ReferringSameLoDminLayer based on the encoding method of the attribute information. For example, when a 3D data encoding device calculates the prediction residuals of the attribute information of 3D points contained in the lower layer of the LoD, and an encoding method that feeds back these prediction residuals to the upper layer for encoding (e.g., Lifting) is used, the 3D points contained in the same layer cannot be referenced in the generation of the prediction value because the encoding or decoding process has not been completed. In the case of using such an encoding method, the value of EnableReferringSameLoD can also be constrained to 0.

[0553] Additionally, while an example of enablingReferringSameLoD = 0 is shown when using the Lifting mode, it is not limited to this. For example, in the Lifting mode, the top layer of LoD can sometimes reference 3D points contained in the same layer, so the EnableReferringSameLoD information can also be effectively set for the top layer. For example, when using the Lifting mode and EnableReferringSameLoD = 1, if LoDN is the top layer, same-layer referencing can also be permitted for LoDN.

[0554] Additionally, when using the Lifting method, the value of ReferringSameLoDminLayer can also be constrained to maxLoD-1. Therefore, when using the Lifting method, the EnableReferringSameLoD information becomes effective at the top layer of the generated LoD. Furthermore, by constraining the ReferringSameLoDminLayer value to maxLoD-1 when using the Lifting method, the EnableReferringSameLoD information is invalid outside the top layer. That is, when using the Lifting method and EnableReferringSameLoD=1, same-layer references are prohibited outside the top layer, while same-layer references are permitted at the top layer.

[0555] That is, the following conditions can also be applied to the Lifting method and other methods. (1) When EnableReferringSameLoD == 1 and LoDN is the topmost layer, or (2) when EnableReferringSameLoD == 1 and LoDN >= ReferringSameLoDminLayer, same-layer referencing is permitted. In other cases, same-layer referencing is prohibited. Thus, by using the common conditions described above through the Lifting method and other methods, the processing of conditional branches, etc., in multiple methods can be commonalized, thereby reducing the processing load.

[0556] Figure 67 This is a diagram illustrating the syntax of an attribute header. Here, the attribute header is, for example, the header of a frame, slice, or tile unit; it is the header of the attribute information. Figure 67 The attribute header shown contains EnableReferringSameLoD, maxLoD (maximum number of levels), and ReferringSameLoDminLayer.

[0557] EnableReferringSameLoD indicates whether referencing of 3D points within the same LoD is permitted. A value of 1 indicates referencing is allowed, and a value of 0 indicates no referencing is allowed. Alternatively, with a value of 1, referencing can be made of 3D points within the same LoD that have already been encoded or decoded.

[0558] `maxLoD` represents the maximum number of LoD levels. During encoding or decoding, LoDs are generated such that the number of LoD levels is less than or equal to the maximum number of LoD levels indicated by `maxLoD`. For example, if the resulting LoD has fewer LoD levels than `maxLoD`, the topmost LoDN (N < `maxLoD-1`) obtained as a result can be set as the topmost LoD level during encoding or decoding.

[0559] ReferringSameLoDminLayer represents the threshold of the LoD level at which EnableReferringSameLoD information is valid. If EnableReferringSameLoD = N, then EnableReferringSameLoD information is valid at LoD levels above LoD level N. Therefore, when EnableReferringSameLoD = 1, EnableReferringSameLoD information is invalid up to LoD level N-1, allowing for parallel computation of predicted attribute values ​​for 3D points below LoD level N-1. Furthermore, at LoD levels N and above, EnableReferringSameLoD information is valid, enabling the computation of predicted attribute values ​​for 3D points above LoD level N using attribute information from a large number of neighboring 3D points at LoD level N.

[0560] Figure 68 This is a diagram representing another syntax example of an attribute header. Figure 68 The attribute information header shown, in addition to Figure 67 In addition to the attribute header shown, it also includes CodingType (encoding method information).

[0561] The CodingType indicates the encoding method (encoding style) applied to the encoding of attribute information. For example, the encoding method may include a style that calculates the prediction residuals of attribute information of 3D points contained in the lower layer of LoD, feeds the prediction residuals back to the upper layer, and encodes them (e.g., Lifting). Alternatively, the encoding method may also include a style that does not feed the prediction residuals back to the upper layer (e.g., Predicting).

[0562] Furthermore, when CodingType indicates Lifting mode, the value of ReferringSameLoDminLayer can also be constrained to maxLoD-1. Additionally, when CodingType indicates Lifting mode, the 3D data encoding device may not append ReferringSameLoDminLayer to the header. Furthermore, when CodingType indicates Lifting mode, the 3D data encoding device may not append ReferringSameLoDminLayer to the header, but instead encode the value of ReferringSameLoDminLayer as maxLoD-1. Additionally, the 3D data decoding device may decode ReferringSameLoDminLayer as maxLoD-1 even when it is not appended to the header.

[0563] The 3D data encoding device can also perform entropy encoding on the aforementioned EnableReferringSameLoD, maxLoD, and ReferringSameLoDminLayer and append it to the header. For example, the 3D data encoding device binarizes each value and performs arithmetic encoding. Additionally, to reduce processing overhead, the 3D data encoding device can also encode with a fixed length.

[0564] Furthermore, the 3D data encoding device may not need to append EnableReferringSameLoD, maxLoD, and ReferringSameLoDminLayer to the header. For example, these values ​​can be specified through a profile or level, such as a standard. This reduces the number of bits in the header.

[0565] In addition, the 3D data encoding device can also switch between EnableReferringSameLoD, maxLoD, and ReferringSameLoDminLayer in WLD, SPC, or volume units and attach them to the header of WLD, SPC, or volume units.

[0566] Furthermore, to switch between co-referencing at multiple LoD levels, the 3D data encoding device can generate EnableReferringSameLoD for each LoD level. For example, if the maximum number of levels is maxLoD, the 3D data encoding device can append maxLoD EnableReferringSameLoDs to the beginning. This allows switching between co-referencing at each level, thus achieving a balance between encoding efficiency and low quantization. Additionally, if CodingType indicates Lifting mode, the 3D data encoding device can also append the EnableReferringSameLoD used for the topmost level to the beginning.

[0567] Figure 69 This is a flowchart of the three-dimensional data encoding process in this embodiment. First, the three-dimensional data encoding device encodes the location information (geometry) (S8201). For example, the three-dimensional data encoding device uses an octree representation for encoding.

[0568] Next, the 3D data encoding device transforms the attribute information (S8202). For example, after encoding the position information, if the position of a 3D point changes due to quantization or other reasons, the 3D data encoding device redistributes the original attribute information of the 3D point to the changed 3D point. Alternatively, the 3D data encoding device can interpolate the attribute information values ​​based on the amount of position change for redistribution. For example, the 3D data encoding device detects N 3D points before the change that are close to the changed 3D position, and calculates a weighted average of the attribute information values ​​of the N 3D points based on the distance from the changed 3D position to each of the N 3D points, determining the resulting value as the attribute information value of the changed 3D point. Furthermore, if two or more 3D points change to the same 3D position due to quantization or other reasons, the 3D data encoding device can also assign the average of the attribute information from the two or more 3D points before the change as the value of the changed attribute information.

[0569] Next, the three-dimensional data encoding device encodes the attribute information (S8203). For example, when encoding multiple attribute information, the three-dimensional data encoding device may encode the multiple attribute information sequentially. For example, when color and reflectivity are encoded as attribute information, the three-dimensional data encoding device generates a bitstream with the encoded result of reflectivity appended after the encoded result of color. Furthermore, the multiple encoded results of attribute information appended to the bitstream can be in any order.

[0570] Furthermore, the 3D data encoding device can also append information from the beginning of the encoded data representing each attribute information within the bitstream to the header. This allows the 3D data decoding device to selectively decode only the attribute information that needs decoding, thus omitting the decoding of unnecessary attribute information. Therefore, the processing load of the 3D data decoding device can be reduced. Additionally, the 3D data encoding device can encode multiple attribute information in parallel and integrate the encoding results into a single bitstream. This allows the 3D data encoding device to encode multiple attribute information at high speed.

[0571] Figure 70 This is a flowchart of the attribute information encoding process (S8203). First, the 3D data encoding device sets the LoD (S8211). That is, the 3D data encoding device assigns each 3D point to any one of the multiple LoDs.

[0572] Next, the 3D data encoding device begins the loop of LoD units (S8212). That is, the 3D data encoding device repeatedly performs the processing steps S8213 to S8221 for each LoD.

[0573] Next, the three-dimensional data encoding device begins the cycle of three-dimensional point units (S8213). That is, the three-dimensional data encoding device repeatedly performs the processing steps S8214 to S8220 for each three-dimensional point.

[0574] First, the 3D data encoding device searches for multiple surrounding 3D points, which exist around the object's 3D point and are used in calculating the predicted value of the object's 3D point (S8214). Next, the 3D data encoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the resulting value as the predicted value P (S8215). Next, the 3D data encoding device calculates the difference between the object's 3D point's attribute information and the predicted value, i.e., the prediction residual (S8216). Next, the 3D data encoding device calculates a quantized value by quantizing the prediction residual (S8217). Next, the 3D data encoding device performs arithmetic encoding on the quantized value (S8218).

[0575] Furthermore, the 3D data encoding device calculates the inverse quantization value by performing inverse quantization on the quantized value (S8219). Next, the 3D data encoding device generates the decoded value by adding the predicted value to the inverse quantization value (S8220). Then, the 3D data encoding device ends the loop for 3D point units (S8221). Additionally, the 3D data encoding device ends the loop for LoD units (S8222).

[0576] Figure 71This is a flowchart of the three-dimensional data decoding process in this embodiment. First, the three-dimensional data decoding device decodes the position information (geometry) from the bitstream (S8231). For example, the three-dimensional data decoding device uses an octree representation for decoding.

[0577] Next, the 3D data decoding device decodes the attribute information from the bitstream (S8232). For example, when decoding multiple attribute information, the 3D data decoding device can decode the multiple attribute information sequentially. For example, when decoding color and reflectance as attribute information, the 3D data decoding device decodes the color encoding result and the reflectance encoding result in the order appended to the bitstream. For example, if the reflectance encoding result is appended after the color encoding result in the bitstream, the 3D data decoding device decodes the color encoding result and then decodes the reflectance encoding result. Furthermore, the 3D data decoding device can decode the attribute information encoding results appended to the bitstream in any order.

[0578] Furthermore, the 3D data decoding device can also obtain the start information of the attribute information representation encoded data within the bitstream by decoding the header, etc. Therefore, the 3D data decoding device can selectively decode the attribute information that needs to be decoded, thus omitting the decoding processing of attribute information that does not need to be decoded. This reduces the processing load of the 3D data decoding device. Additionally, the 3D data decoding device can decode multiple attribute information in parallel and merge the decoding results into a single 3D point group. Therefore, the 3D data decoding device can decode multiple attribute information at high speed.

[0579] Figure 72 This is a flowchart of the attribute information decoding process (S8232). First, the 3D data decoding device sets the Location of Dimensions (LoD) (S8241). That is, the 3D data decoding device assigns multiple 3D points with decoded location information to any one of the multiple LoDs. For example, this assignment method is similar to... Figure 70 The same allocation method is used in the three-dimensional data encoding device shown.

[0580] Next, the 3D data decoding device begins the loop of LoD units (S8242). That is, the 3D data decoding device repeatedly performs the processing steps S8243 to S8249 for each LoD.

[0581] Next, the 3D data decoding device begins the loop of 3D point units (S8243). That is, the 3D data decoding device repeatedly performs the processing steps S8244 to S8248 for each 3D point.

[0582] First, the 3D data decoding device searches for multiple surrounding points, which exist around the object's 3D points and are used in calculating the predicted value of the object's 3D points (S8244). Next, the 3D data decoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the resulting value as the predicted value P (S8245). Furthermore, these processes are the same as those in the 3D data encoding device.

[0583] Next, the 3D data decoding device performs arithmetic decoding on the quantized values ​​from the bitstream (S8246). Furthermore, the 3D data decoding device calculates the inverse quantized value by performing inverse quantization on the decoded quantized value (S8247). Next, the 3D data decoding device generates the decoded value by adding the predicted value to the inverse quantized value (S8248). Next, the 3D data decoding device ends the loop for 3D point units (S8249). Additionally, the 3D data decoding device ends the loop for LoD units (S8250).

[0584] Figure 73 This is a flowchart of the process for searching surrounding points (S8214). Furthermore, the process in S8244 is the same as that in S8214.

[0585] First, the 3D data encoding device selects SearchNumPoint[upper LoD] 3D points from the higher LoD layer of the layer to which the object's 3D point belongs, and uses the selected 3D points to calculate N 3D points for generating the prediction value (S8261). Here, SearchNumPoint represents the number of searches when selecting N 3D points for prediction from the group of 3D points within the LoD. Additionally, SearchNumPoint[upper LoD] represents the number of searches for layers higher than the object's 3D point.

[0586] Next, the three-dimensional data encoding device determines whether it satisfies one of the following conditions: (1) EnableReferringSameLoD==1 and LoDN is the top layer, and (2) EnableReferringSameLoD==1 and LoDN>=ReferringSameLoDminLayer (S8262).

[0587] (1) If either EnableReferringSameLoD == 1 and LoDN is the topmost layer, or (2) EnableReferringSameLoD == 1 and LoDN >= ReferringSameLoDminLayer ("Yes" in S8262), the 3D data encoding device selects SearchNumPoint[SameLoD] 3D points from the same LoD layer and uses the selected 3D points to update the N 3D points used for generating the prediction value (S8263). Here, SearchNumPoint[SameLoD] represents the number of searches for the layer to which the object 3D point belongs.

[0588] On the other hand, if (1) EnableReferringSameLoD==1 and LoDN is the topmost layer, and (2) EnableReferringSameLoD==1 and LoDN>=ReferringSameLoDminLayer are not satisfied ("No" in S8262), no update is performed, and the N three-dimensional points calculated in S8261 are determined as the three-dimensional points used for generating the prediction value.

[0589] In addition, Figure 73 In the example shown, after the 3D data encoding device calculates N 3D points for generating prediction values ​​from the upper LoD layer, it selects and updates the 3D points for generating prediction values ​​from the same LoD. However, it can also select and update the 3D points for generating prediction values ​​from the upper LoD after calculating N 3D points for generating prediction values ​​from the same LoD layer.

[0590] (Implementation Method 6)

[0591] This illustrates another example of using LoD (Level of Detail) information to encode the attribute information (attribute information) of 3D points. Specifically, it illustrates the use of the ReferringSameLoDNumLayer method, which represents the range of permitted same-level references. Same-level references refer to 3D points that belong to the same LoD level as the 3D points of the object being encoded.

[0592] ReferringSameLoDNumLayer indicates the range of LoD levels where referencing 3D points within the same LoD is permitted. Specifically, ReferringSameLoDNumLayer represents the range of LoD levels where same-level referencing is allowed. For example, within the range from the topmost LoD level to the number of values ​​in ReferringSameLoDNumLayer, 3D points within the same LoD can be referenced (same-level referencing is allowed); at other levels, 3D points within the same LoD cannot be referenced (same-level referencing is not allowed). Furthermore, the ability to perform same-level referencing means that same-level referencing is permitted. The inability to perform same-level referencing means that same-level referencing is prohibited (not permitted).

[0593] For example, in the case where the maximum LoD layer number is 3, the lowest layer is LoD0, the layer above LoD0 is LoD1, and the highest layer above LoD1 is LoD2. If ReferringSameLoDNumLayer = 1, then LoD2 (the highest layer) can be referenced at the same level. Furthermore, if ReferringSameLoDNumLayer = 2, then references at the same level can be made between LoD2 and LoD1. Additionally, if ReferringSameLoDNumLayer = 3 (the maximum LoD layer number), then references at the same level can be made across all layers (LoD0, LoD1, LoD2).

[0594] That is, if ReferringSameLoDNumLayer = N, then same-layer referencing is possible in the N layers starting from the top layer, but not in layers lower than them. Thus, by prohibiting same-layer referencing in the N layers lower than the top layer, the predicted values ​​of the attribute information of 3D points belonging to these layers can be calculated in parallel. Furthermore, since same-layer referencing is possible in the N layers starting from the top layer, the predicted values ​​of the attribute information of 3D points belonging to the N layers starting from the top layer can be calculated using the attribute information of a large number of adjacent 3D points belonging to the N layers starting from the top layer. As a result, a balance can be achieved between reducing processing time and improving coding efficiency.

[0595] Furthermore, the above example illustrates that if ReferringSameLoDNumLayer = N, then same-level referencing is possible across the N levels starting from the top layer, but this is not necessarily the case. For instance, if ReferringSameLoDNumLayer = N, then 3D points within the same LoD can be referenced (same-level referencing is possible) across the N levels starting from the bottom layer, but not across levels above that. This allows for improved coding efficiency by enabling same-level referencing across the N levels starting from the bottom layer, while allowing for parallel computation of predicted attribute information for 3D points belonging to that level in levels above that.

[0596] Additionally, ReferringSameLoDNumLayer can represent either the range of layers that can be referenced at the same level or the range of layers that cannot be referenced at the same level. For example, if ReferringSameLoDNumLayer = N, then references at the same level cannot be made in the N layers starting from the lowest level, but are possible in layers above them. Alternatively, if ReferringSameLoDNumLayer = N, then references at the same level cannot be made in the N layers starting from the highest level, but are possible in layers below them.

[0597] Furthermore, the range of the value of ReferringSameLoDNumLayer can be restricted to 1 or higher but less than maxLoD, provided the maximum number of LoD levels is maxLoD. That is, by limiting the minimum value of ReferringSameLoDNumLayer to 1, same-layer referencing is guaranteed in the topmost layer. Therefore, for 3D points belonging to the topmost layer that do not refer to other layers, references can be made to 3D points within the same LoD, thus suppressing a decrease in coding efficiency. Additionally, when ReferringSameLoDNumLayer = maxLoD, same-layer referencing is possible in all LoD levels. This improves the coding efficiency for 3D points belonging to all LoD levels.

[0598] Figure 74This diagram illustrates an example of the referencing relationship when ReferringSameLoDNumLayer = 1 (minimum value). In this case, same-layer referencing is possible within the LoD2 layer. That is, in the generation of predicted values ​​for the attribute information of 3D points belonging to LoD2, 3D points within the same layer are referential. For example, the predicted value of a2 is generated using the attribute information of either a0 or a1. Since a0 and a1 have already been encoded or decoded, a0 and a1 can be referential. Therefore, because the predicted value of a2 is generated using the attribute information of more adjacent 3D points within the same LoD, the prediction accuracy is improved, thus enhancing coding efficiency.

[0599] On the other hand, in generating the predicted values ​​of attribute information for 3D points belonging to LoD0, referencing 3D points at the same level is prohibited. For example, the predicted value of c2 is generated using the attribute information of b0, b1, and b2. Even if c0 and c1 have already been encoded or decoded, referencing c0 and c1 is prohibited. Thus, the predicted value of c2 can be generated without waiting for the encoding or decoding of c0 and c1 to be completed. That is, the 3D data encoding device or the 3D data decoding device can calculate the predicted values ​​of attribute information for 3D points within the same LoD in parallel, thereby reducing processing time.

[0600] Figure 75 This diagram illustrates an example of referencing relationships when ReferringSameLoDNumLayer = maxLoD (maximum value). In this case, same-layer referencing is possible across all layers (LoD0, LoD1, and LoD2). That is, in generating predicted values ​​for the attribute information of all 3D points, reference can be made to 3D points contained within the same layer. For example, the predicted value of c2 is generated using the attribute information of b0, b1, b2, c0, and c1. c0 and c1 can be referenced if they have already been encoded or decoded. Therefore, since the predicted value of c2 is generated using more attribute information from adjacent 3D points, the prediction accuracy is improved, thus enhancing encoding efficiency.

[0601] Figure 76 This is a diagram illustrating the syntax of an attribute header. Here, the attribute header is, for example, the header of a frame, slice, or tile unit, and is the header of attribute information. Figure 76 The attribute header shown contains maxLoD (maximum number of layers) and ReferringSameLoDNumLayer_minus1.

[0602] `maxLoD` represents the maximum number of LoD levels. During encoding or decoding, LoD levels are generated such that the number of LoD levels is below the maximum number represented by `maxLoD`. For example, if the generated LoD level does not reach `maxLoD`, the highest LoDN (N < `maxLoD-1`) obtained as a result can be set as the highest LoD level during encoding or decoding. Furthermore, `maxLoD_minus1` (= `maxLoD-1`) can be appended to the bitstream instead of `maxLoD`. This reduces the header size.

[0603] ReferringSameLoDNumLayer_minus1 represents the range of LoD levels that can be referenced to 3D points within the same LoD. Specifically, ReferringSameLoDNumLayer_minus1 indicates the range of LoD levels that allow (or disallow) referencing within the same layer. Additionally, ReferringSameLoDNumLayer_minus1 represents the value obtained by subtracting 1 from ReferringSameLoDNumLayer. In the 3D data decoding device, ReferringSameLoDNumLayer is calculated based on ReferringSameLoDNumLayer = ReferringSameLoDNumLayer_minus1 + 1.

[0604] The value of ReferringSameLoDNumLayer indicates that same-level referencing is permitted within the first one levels of the LoD represented by maxLoD (the topmost level), and prohibited within other levels. For example, when ReferringSameLoDNumLayer = 1 (minimum value), same-level referencing is permitted within the topmost level of the LoD represented by maxLoD, and prohibited within other levels.

[0605] Additionally, for example, when ReferringSameLoDNumLayer=2, same-level referencing is possible in the two levels from the topmost level of the LoD shown by maxLoD, namely the topmost level and the levels below it. In other levels, namely the levels other than the topmost level and the levels below it, same-level referencing is not possible.

[0606] Additionally, for example, when ReferringSameLoDNumLayer=N, same-level referencing is possible in the N levels starting from the topmost level of the LoD shown by maxLoD, i.e., the N levels including the topmost level. In other levels, i.e., levels other than the N levels including the topmost level, same-level referencing is not possible.

[0607] Additionally, for example, when ReferringSameLoDNumLayer = maxLoD (maximum value), same-level referencing is possible in all levels, including maxLoD layers starting from the topmost layer of the LoD shown by maxLoD.

[0608] Furthermore, the minimum value of ReferringSameLoDNumLayer_minus1 is 0, and the maximum value is maxLoD-1. That is, the minimum value of ReferringSameLoDNumLayer is 1, and the maximum value is maxLoD. Therefore, even in the topmost layer where other LoD levels cannot be referenced, same-layer references are always possible. This helps to suppress the reduction in coding efficiency.

[0609] Alternatively, in the same layer reference, it is possible to refer to 3D points that have been encoded or decoded within the same LoD, but not to 3D points that have not been encoded or decoded.

[0610] Additionally, when maxLoD = 1 (i.e., the number of LoD levels = 1), the 3D data encoding device can also be configured to referringSameLoDNumLayer_minus1 = 0 (i.e., ReferringSameLoDNumLayer = 1) to enable encoding via same-layer referencing. Furthermore, as... Figure 76 As shown, the 3D data encoding device can also omit the ReferringSameLoDNumLayer_minus1 appended to the bitstream when maxLoD=1. Therefore, with only one layer, same-layer references can be used to improve encoding efficiency and reduce header size.

[0611] Furthermore, when maxLoD = 1, the 3D data decoding device can also estimate the value of ReferringSameLoDNumLayer_minus1 as 0, that is, estimate the value of ReferringSameLoDNumLayer as 1. Therefore, when maxLoD = 1, the 3D data decoding device determines that ReferringSameLoDNumLayer_minus1 = 0, i.e., ReferringSameLoDNumLayer = 1, and can appropriately decode the encoded bitstream using the same-layer reference.

[0612] Figure 77 This is a diagram representing another syntax example of the attribute information header. For example... Figure 77 As shown, even when maxLoD is 1, ReferringSameLoDNumLayer_minus1 can still be appended to the bitstream. In this case, with maxLoD = 1, ReferringSameLoDNumLayer_minus1 can be specified as 0 by the profile or level of the standard, etc.

[0613] Furthermore, if the 3D data decoding device has maxLoD=1 and does not have ReferringSameLoDNumLayer_minus1=0, it can determine that the bitstream does not conform to the standard (does not conform to the predetermined standard, or violates the standard).

[0614] Thus, with a level of 1, by setting this single level to be able to reference other levels within the same level, coding efficiency can be improved and header size can be reduced.

[0615] Alternatively, the 3D data encoding device can append maxLoD or ReferringSameLoDNumLayer_minus1 to the header after entropy encoding. For example, the 3D data encoding device binarizes each value and performs arithmetic encoding. Furthermore, to reduce processing overhead, the 3D data encoding device can also use a fixed-length encoding.

[0616] Furthermore, 3D data encoding devices do not necessarily need to append maxLoD or ReferringSameLoDNumLayer_minus1 to the header. For example, these values ​​can be specified using standard profiles or levels. This allows for a reduction in the number of bits in the header.

[0617] Additionally, the 3D data encoding device can also switch maxLoD or ReferringSameLoDNumLayer_minus1 to WLD, SPC, or volume units and append it to the header of WLD, SPC, or volume units.

[0618] Furthermore, the value of ReferringSameLoDNumLayer_minus1 can also be constrained by the encoding method of attribute information. For example, in the case of an encoding method that uses the prediction residual of the attribute information of the 3D points contained in the lower layer of the LoD and feeds this prediction residual back to the upper layer for encoding (Lifting method), sometimes the uppermost LoD is set to be able to be referenced at the same layer. Therefore, the value of ReferringSameLoDNumLayer_minus1 can be constrained to 0 (always set to 0). This can improve the encoding efficiency of the Lifting method.

[0619] Furthermore, within the highest-level layer resulting from the generation of LoD, same-level referencing is possible regardless of the value of ReferringSameLoDNumLayer. For example, in calculating the predicted attribute information of 3D points belonging to the LoDN level, the following conditional expression can be used to determine whether 3D points within the same LoD can be referenced.

[0620] "If (the layer that becomes the highest level as a result of generating LoDN == LoD or (N >= maxLoD - ReferringSameLoDNumLayer)) then it can reference 3D points of the same LoD."

[0621] Otherwise, referencing 3D points within the same LoD is prohibited.

[0622] That is, same-level referencing is permitted when the LoDN (N is an integer from 0 to maxLoD-1) of the object being processed (the object to be determined) is the highest level. Additionally, same-level referencing is permitted when the LoDN is a level where same-level referencing is allowed as shown by ReferringSameLoDNumLayer. Same-level referencing is prohibited when the LoDN is neither the highest level nor a level where same-level referencing is allowed as shown by ReferringSameLoDNumLayer.

[0623] Figure 78 This is a diagram illustrating an example of the referencing relationship when ReferringSameLoDNumLayer = 1, as described above.

[0624] For example, in Figure 78In the example shown, the maximum number of LoD levels is maxLoD, and the result of generating LoDs is that the highest level does not reach maxLoD-1; the highest level is LoD2. Here, since ReferringSameLoDNumLayer = 1, the level that can be referenced at the same level as the LoD level, as defined by ReferringSameLoDNumLayer, is maxLoD-1. In this case, although LoD2 is not included in the range of levels that can be referenced at the same level as defined by ReferringSameLoDNumLayer, it is permitted to be referenced at the same level as the highest level. That is, the level that becomes the highest level as a result of generating LoDs is permitted to be referenced at the same level, regardless of the value of ReferringSameLoDNumLayer. Therefore, since it is possible to be referenced at the same level in at least one level, coding efficiency can be improved.

[0625] In the above scenario, during the generation of predicted values ​​for the attribute information of 3D points belonging to the highest-level LoD2, reference to 3D points within the same level is permitted. For example, the predicted value of a2 can be generated using the attribute information of either a0 or a1. Furthermore, a0 and a1 can also be referenced if they have already been encoded or decoded. Therefore, since the predicted value of a2 is generated using attribute information from more adjacent 3D points, prediction accuracy is improved, thus enhancing coding efficiency.

[0626] Furthermore, whether a LoDN is the highest-level layer can be determined, for example, based on whether 3D points assigned to a higher-level layer than the LoDN remain in buffers or similar containers. That is, the LoDN can be determined as the highest-level layer if no 3D points assigned to a higher-level layer remain in buffers or similar containers, and otherwise, it is determined not to be the highest-level layer. Thus, the 3D data encoding device can determine whether a LoDN is the highest-level layer and, based on the result, switch between allowing same-level referencing.

[0627] Furthermore, the value of ReferringSameLoDNumLayer can also be constrained by the encoding method of attribute information. For example, in the case of a limping encoding method that uses the prediction residuals of attribute information of 3D points contained in the lower layer of the LoD to feed back these prediction residuals to the upper layer for encoding, 3D points in the same layer cannot be referenced in the generation of prediction values ​​because they have not been encoded or decoded. In this encoding method, regardless of the value of ReferringSameLoDNumLayer, it can be set to prevent same-layer reference. Additionally, in the case of 3D data encoding in limping mode, the ReferringSameLoDNumLayer can be omitted from the header. This reduces the header size.

[0628] Additionally, by importing the following conditional expression, when the encoding method is Lifting, it can also be set to prevent same-layer references.

[0629] If (the encoding method is not Lifting and (as a result of generating LoDN == LoD, it becomes the highest level layer or (N >= maxLoD - ReferringSameLoDNumLayer))) then it can reference three-dimensional points of the same LoD.

[0630] Otherwise, referencing 3D points within the same LoD is prohibited.

[0631] Additionally, this example illustrates how to disable intra-level referencing when using the Lifting method, but it is not necessarily limited to this. For instance, when using the Lifting method, there may be cases where the topmost LoD can refer to 3D points within the same level; therefore, it is also possible to configure the topmost LoD to allow intra-level referencing. For example, when using the Lifting method, if the LoDN is the topmost layer, the LoDN can be configured to allow intra-level referencing.

[0632] Furthermore, when using the Lifting method, the value of ReferringSameLoDNumLayer can also be constrained to 1. Therefore, in the Lifting method, same-level referencing is allowed at the topmost layer of the generated LoD, while same-level referencing is prohibited outside the topmost layer. That is, the following condition can also be applied to the Lifting method and other methods.

[0633] "If (the result of generating LoDN == LoD, which becomes the highest level or (N >= maxLoD - ReferringSameLoDNumLayer)) then it can refer to three-dimensional points of the same LoD."

[0634] Otherwise, referencing 3D points within the same LoD is prohibited.

[0635] Therefore, since a common formula as described above can be used in both the Lifting method and other methods, it is possible to reduce processing such as conditional branching based on multiple methods.

[0636] Figure 79 This is a flowchart of the surrounding point search process, which searches for multiple surrounding points that exist around the object's 3D points in the calculation of the predicted value of the object's 3D points.

[0637] First, the 3D data encoding device selects SearchNumPoint[upper LoD] 3D points from the upper LoD layer of the level to which the object's 3D points belong, and uses the selected 3D points to calculate N 3D points for generating the prediction value (S11601). Here, SearchNumPoint represents the number of searches when selecting N 3D points for prediction from the group of 3D points within the LoD. SearchNumPoint[upper LoD] represents the number of searches in the upper layer of the object's 3D points.

[0638] Next, the 3D data encoding device selects SearchNumPoint[SameLoD] 3D points from the same LoD layer if at least one of the following conditions is met: (1) the LoDN of the object being processed is the highest layer (the layer that becomes the highest layer as a result of generating the LoD) and (2) N >= maxLoD - ReferringSameLoDNumLayer (S11602 "Yes"), and updates the N 3D points used for generating the prediction value using the selected 3D points (S11603). Here, SearchNumPoint[SameLoD] represents the number of searches for the layer to which the object's 3D points belong.

[0639] On the other hand, if (1) the LoDN of the object being processed is the highest level and (2) N >= maxLoD - ReferringSameLoDNumLayer are not satisfied ("No" in S11602), no update is performed, and the N three-dimensional points calculated in S11601 are determined as the three-dimensional points used for generating the prediction value.

[0640] In addition, Figure 79In the example shown, after the 3D data encoding device calculates N 3D points for generating prediction values ​​based on the upper LoD layer, it selects and updates the 3D points for generating prediction values ​​from the same LoD. However, it can also select and update the 3D points for generating prediction values ​​from the upper LoD after calculating N 3D points for generating prediction values ​​based on the same LoD layer.

[0641] Furthermore, the 3D data encoding device calculates, for example, a weighted average of the attribute information values ​​of N 3D points (surrounding points) obtained through the above processing, and sets the resulting value as the predicted value P. Next, the 3D data encoding device calculates the difference between the attribute information of the object's 3D points and the predicted value, i.e., the prediction residual. Then, the 3D data encoding device calculates the quantized value by quantizing the prediction residual. Finally, the 3D data encoding device performs arithmetic encoding on the quantized value.

[0642] Furthermore, the same surrounding point search process is performed in the 3D data decoding device. In this case, the weighted average of the attribute information values ​​of N 3D points (surrounding points) calculated by the 3D data decoding device is used as the predicted value P. Next, the 3D data decoding device arithmetically decodes the quantized value from the bitstream. Furthermore, the 3D data decoding device calculates the inverse quantized value by inverse quantization of the decoded quantized value. Finally, the 3D data decoding device generates the decoded value (attribute information) by adding the predicted value to the inverse quantized value.

[0643] As described above, the three-dimensional data encoding device of this embodiment performs... Figure 80 The process is shown below. The 3D data encoding device, based on multiple location information of multiple 3D points, assigns multiple 3D points contained in the point group data to one of more than one levels (e.g., LoD) (S11611). When the number of more than one levels is one, the 3D data encoding device permits same-level references and generates encoded attribute information by encoding multiple attribute information of multiple 3D points. The same-level references refer to the attribute information of other 3D points contained in the same level as the 3D points of the processing object to generate predicted values ​​of the attribute information of the 3D points of the processing object (S11612). For example, when the number of more than one levels is one, the 3D data encoding device always permits same-level references. Next, the 3D data encoding device generates a bitstream containing encoded attribute information (S11613).

[0644] Therefore, when the number of layers is 1, the 3D data encoding device can improve encoding efficiency by allowing same-layer references.

[0645] For example, when there are two or more levels in a three-dimensional data encoding device, it determines which levels allow and prohibit intra-level referencing among the two or more levels. Based on the determination results, it allows or prohibits intra-level referencing and encodes multiple attribute information to generate encoded attribute information.

[0646] Therefore, when the number of layers is two or more, the 3D data encoding device can choose to allow or prohibit intra-layer references. This allows for a proper balance between parallel processing and encoding efficiency.

[0647] For example, a three-dimensional data encoding device allows same-level references for levels from the topmost level to the Nth level (N is a natural number) in two or more levels, and prohibits same-level references for levels below the Nth level, thereby encoding multiple attribute information to generate encoded attribute information.

[0648] Therefore, the three-dimensional data encoding device can efficiently improve encoding efficiency by allowing references to the same layer from the upper layer.

[0649] For example, the bitstream contains first information (e.g., ReferringSameLoDNumLayer_minus1), which indicates the layer that allows or prohibits same-layer referencing. That is, the 3D data encoding device appends first information to the bitstream.

[0650] For example, the bitstream contains a second piece of information (e.g., maxLoD) indicating the number of more than one level. That is, the three-dimensional data encoding device appends the second piece of information to the bitstream.

[0651] For example, a three-dimensional data encoding device has a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0652] Furthermore, the three-dimensional data decoding device of this embodiment performs... Figure 81 The process is illustrated below. The 3D data decoding device obtains encoded attribute information from the bitstream. This encoded attribute information is information obtained by encoding multiple attribute information of multiple 3D points contained in the point group data. These multiple 3D points are assigned to one or more levels (e.g., LoD) based on their multiple positional information (S11621). When the number of levels is one, the 3D data decoding device permits same-level references to decode the encoded attribute information. These same-level references refer to the attribute information of other 3D points contained in the same level as the 3D points of the processing object to generate predicted values ​​for the attribute information of the 3D points of the processing object (S11622). For example, when the number of levels is one, the 3D data decoding device always permits same-level references.

[0653] Therefore, the 3D data decoding device can improve encoding efficiency by allowing same-layer references when the number of layers is 1.

[0654] For example, when there are two or more levels in a three-dimensional data decoding device, it determines which levels allow and prohibit intra-level referencing among the two or more levels, and decodes the encoded attribute information by allowing or prohibiting intra-level referencing based on the decision result.

[0655] For example, a 3D data decoding device allows same-layer referencing from the topmost layer to the Nth layer (N is a natural number) in two or more layers, while prohibiting same-layer referencing from layers below the Nth layer, thereby decoding the encoded attribute information.

[0656] For example, a 3D data decoding device obtains first information (e.g., ReferringSameLoDNumLayer_minus1) from the bitstream, which indicates the layer that allows or prohibits same-layer referencing. Based on this first information, the 3D data decoding device determines the layers that allow or prohibit same-layer referencing, and based on the determination result, allows or prohibits same-layer referencing, thereby decoding the encoded attribute information.

[0657] For example, a 3D data decoding device obtains second information (e.g., maxLoD) representing the number of more than one level from the bitstream. For example, the 3D data decoding device determines whether the number of more than one level is 1 based on the second information.

[0658] For example, a three-dimensional data decoding device obtains (1) first information indicating whether the same-layer reference is permitted or prohibited, and (2) second information indicating the number of more than one layer from the bitstream. If the number of more than one layer indicated by the second information is 1, or if the first information does not indicate that same-layer reference is permitted in one of the more than one layers, the bitstream is determined to be non-compliant with the standard.

[0659] For example, a 3D data decoding device has a processor and a memory, with the processor using the memory to perform the aforementioned processing.

[0660] (Implementation Method 7)

[0661] Next, the configuration of the three-dimensional data production apparatus 810 of this embodiment will be described. Figure 82 This is a block diagram illustrating a structural example of the three-dimensional data creation apparatus 810 according to this embodiment. The three-dimensional data creation apparatus 810 is, for example, mounted on a vehicle. The three-dimensional data creation apparatus 810 transmits and receives three-dimensional data from an external traffic monitoring cloud, vehicles ahead, or vehicles behind, and creates and accumulates three-dimensional data.

[0662] The 3D data production apparatus 810 includes: a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a 3D data production unit 816, a 3D data synthesis unit 817, a 3D data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0663] The data receiving unit 811 receives 3D data 831 from the traffic monitoring cloud or the vehicle ahead. The 3D data 831 includes, for example, information such as point clouds, visible light images, depth information, sensor position information, or speed information, including areas that cannot be detected by the vehicle's sensors 815.

[0664] The 812 communication unit communicates with the traffic monitoring cloud or the vehicle ahead, sending data transmission requests and other information to the traffic monitoring cloud or the vehicle ahead.

[0665] The receiving control unit 813 exchanges information such as the corresponding format with the communicating party via the communication unit 812 to establish communication with the communicating party.

[0666] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion on the three-dimensional data 831 received by the data receiving unit 811. Furthermore, if the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0667] Multiple sensors 815 are a group of sensors, such as LiDAR, visible light cameras, or infrared cameras, that acquire information about the exterior of the vehicle, generating sensor information 833. For example, if the sensors 815 are laser sensors such as LiDAR, the sensor information 833 is three-dimensional data such as point clouds (point group data). Alternatively, there may be more than one sensor 815.

[0668] The 3D data production unit 816 generates 3D data 834 based on sensor information 833. 3D data 834 may include, for example, point clouds, visible light images, depth information, sensor position information, or velocity information.

[0669] The 3D data synthesis unit 817 synthesizes 3D data 832 generated from traffic monitoring cloud or vehicles ahead with 3D data 834 generated based on sensor information 833 of the vehicle itself, thereby constructing 3D data 835 that includes the space in front of the vehicles ahead that cannot be detected by the vehicle's sensors 815.

[0670] The three-dimensional data accumulation unit 818 accumulates the generated three-dimensional data 835, etc.

[0671] The 819 communication unit communicates with the traffic monitoring cloud or vehicles behind it, sending data transmission requests and other information to the traffic monitoring cloud or vehicles behind it.

[0672] The transmission control unit 820 exchanges information such as the corresponding format with the communication unit 819 to establish communication with the communication party. Furthermore, the transmission control unit 820 determines the transmission area of ​​the spatial dimension of the dimension data to be transmitted based on the dimension data construction information of the dimension data 832 generated by the dimension data synthesis unit 817 and the data transmission request from the communication party.

[0673] Specifically, the transmission control unit 820 determines the transmission area based on data transmission requests from the traffic monitoring cloud or rear vehicles. This transmission area includes the space in front of the vehicle that cannot be detected by the sensors of the rear vehicles. Furthermore, the transmission control unit 820 determines the transmission area based on 3D data construction information, judging whether the space that can be transmitted or the space that has already been transmitted has been updated. For example, the transmission control unit 820 determines the area specified by the data transmission request and for which corresponding 3D data 835 exists as the transmission area. The transmission control unit 820 also notifies the format conversion unit 821 of the communication format and the transmission area.

[0674] The format conversion unit 821 generates three-dimensional data 837 by converting the three-dimensional data 836 of the transmitting area in the three-dimensional data 835 stored in the three-dimensional data storage unit 818 into a format corresponding to the receiving side. In addition, the format conversion unit 821 can also reduce the data volume by compressing or encoding the three-dimensional data 837.

[0675] The data transmission unit 822 transmits the 3D data 837 to the traffic monitoring cloud or to vehicles behind. The 3D data 837 may include, for example, information such as: point clouds of areas that become blind spots for vehicles behind, visible light images of the area in front of the vehicle, depth information, or sensor location information.

[0676] Furthermore, this explanation uses the format conversion units 814 and 821 as an example, but format conversion may not be necessary.

[0677] With this configuration, the 3D data generation apparatus 810 acquires 3D data 831 of an area that cannot be detected by the vehicle's sensors 815 from the outside, and generates 3D data 835 by combining the 3D data 831 with 3D data 834 based on sensor information 833 detected by the vehicle's sensors 815. Thus, the 3D data generation apparatus 810 is capable of generating 3D data for a range that cannot be detected by the vehicle's sensors 815.

[0678] Furthermore, the 3D data generation device 810 can send 3D data, including the space in front of the vehicle that cannot be detected by the sensors of the vehicle behind, to the traffic monitoring cloud or the vehicle behind, according to data transmission requests from the traffic monitoring cloud or the vehicle behind.

[0679] Next, the sequence of transmitting three-dimensional data to the vehicle behind in the three-dimensional data production device 810 will be explained. Figure 83 This is a flowchart illustrating an example of the sequence in which the 3D data production device 810 sends 3D data to a traffic monitoring cloud or a vehicle behind it.

[0680] First, the 3D data creation device 810 generates and updates 3D data 835 (S801) of the space including the space in front of the vehicle on the road ahead. Specifically, the 3D data creation device 810 constructs 3D data 835 that includes the space in front of the vehicle ahead, which cannot be detected by the vehicle's sensors 815, by combining 3D data 831 created from traffic monitoring cloud or vehicles ahead with 3D data 834 created based on the vehicle's sensor information 833.

[0681] Next, the 3D data production device 810 determines whether the 3D data 835 contained in the transmitted space has changed (S802).

[0682] If the three-dimensional data 835 contained in the space changes due to a vehicle or person entering the space from the outside (S802 "Yes"), the three-dimensional data production device 810 sends the three-dimensional data containing the three-dimensional data 835 of the changed space to the traffic monitoring cloud or the vehicle behind (S803).

[0683] Furthermore, the 3D data creation apparatus 810 can transmit 3D data of a changed space in accordance with the transmission timing of transmitting 3D data at predetermined intervals, but it can also transmit immediately after detecting a change. That is, the 3D data creation apparatus 810 can transmit 3D data of a changed space before 3D data transmitted at predetermined intervals.

[0684] Furthermore, the 3D data production device 810 can transmit all the 3D data of the changed space as 3D data of the changed space, or it can transmit only the differences in the 3D data (such as information on the appearance or disappearance of 3D points, or displacement information of 3D points, etc.).

[0685] Furthermore, the 3D data generation device 810 may send metadata related to the vehicle's hazard avoidance actions, such as emergency braking warnings, to vehicles behind it before sending the 3D data of the changed space. This allows vehicles behind to detect the emergency braking of the vehicle ahead earlier and to initiate hazard avoidance actions such as deceleration as quickly as possible.

[0686] If the three-dimensional data 835 contained in the space that has been sent has not changed ("No" in S802) or after step S803, the three-dimensional data production device 810 sends the three-dimensional data contained in the space of a predetermined shape located at a distance L in front of the vehicle to the traffic monitoring cloud or the vehicle behind (S804).

[0687] Furthermore, for example, the processes in steps S801 to S804 are repeatedly executed at specified time intervals.

[0688] Furthermore, the 3D data production device 810 may not send the 3D spatial data 837 if there is no difference between the 3D spatial data 835 of the current sending object and the 3D map.

[0689] In this embodiment, the client device sends sensor information obtained by the sensor to the server or other client devices.

[0690] First, the system configuration of this embodiment will be explained. Figure 84 This diagram illustrates the structure of the three-dimensional map and sensor information transceiver system according to this embodiment. The system includes a server 901, client devices 902A and 902B. Alternatively, without special distinction between client devices 902A and 902B, it may also be referred to as client device 902.

[0691] The client device 902 is, for example, an in-vehicle device mounted on a moving body such as a vehicle. The server 901 is, for example, a traffic monitoring cloud, capable of communicating with multiple client devices 902.

[0692] Server 901 sends a 3D map composed of point clouds to client device 902. Furthermore, the composition of the 3D map is not limited to point clouds; it can also be represented using other 3D data such as grid structures.

[0693] The client device 902 sends sensor information obtained by the client device 902 to the server 901. The sensor information includes, for example, at least one of the following: LiDAR-acquired information, visible light image, infrared image, depth image, sensor position information, and velocity information.

[0694] Regarding the data transmitted and received between server 901 and client device 902, compression can be performed to reduce data volume, while compression can be omitted to maintain data accuracy. When data compression is used, for example, an octree-based 3D compression method can be employed for point clouds. Furthermore, 2D image compression methods can be used for visible light images, infrared images, and depth images. 2D image compression methods include, for example, MPEG-4 AVC or HEVC standardized by MPEG.

[0695] Furthermore, server 901 sends the 3D map managed by server 901 to client device 902 according to the 3D map sending request from client device 902. Alternatively, server 901 may send the 3D map without waiting for a sending request from client device 902. For example, server 901 may broadcast the 3D map to more than one client device 902 in a predefined space. Server 901 may also send a 3D map appropriate to the location of client device 902 at regular intervals to client devices 902 that have received a sending request once. Furthermore, server 901 may send the 3D map to client device 902 whenever the 3D map managed by server 901 is updated.

[0696] Client device 902 sends a request to server 901 to send a 3D map. For example, if client device 902 wants to estimate its own position while driving, it sends the request to server 901.

[0697] Additionally, client device 902 may also send a request to server 901 to transmit a 3D map in the following situations: If the 3D map held by client device 902 is relatively old, client device 902 may also send a request to server 901 to transmit the 3D map. For example, if client device 902 has obtained the 3D map and a certain period of time has passed, client device 902 may also send a request to server 901 to transmit the 3D map.

[0698] Alternatively, the client device 902 may send a request to the server 901 to transmit the 3D map a certain time before it is about to leave the space shown on the 3D map held by the client device 902. For example, the client device 902 may send the request to the server 901 when it is within a pre-defined distance from the boundary of the space shown on the 3D map held by the client device 902. Furthermore, knowing the movement path and speed of the client device 902, the time when the client device 902 will leave the space shown on the 3D map held by the client device 902 can be predicted based on the known movement path and speed.

[0699] When the error between the three-dimensional data generated by the client device 902 based on sensor information and the position of the three-dimensional map exceeds a certain range, the client device 902 can send a request to the server 901 to send the three-dimensional map.

[0700] Client device 902 sends sensor information to server 901 according to the sensor information transmission request sent from server 901. Alternatively, client device 902 may send sensor information to server 901 without waiting for a transmission request from server 901. For example, if client device 902 has received a sensor information transmission request from server 901 once, it may periodically send sensor information to server 901 within a certain period. Furthermore, if the error between the 3D data generated by client device 902 based on sensor information and the 3D map obtained from server 901 exceeds a certain range, client device 902 determines that the 3D map around client device 902 may have changed, and sends this determination along with the sensor information to server 901.

[0701] Server 901 sends a sensor information transmission request to client device 902. For example, server 901 receives location information of client device 902, such as GPS data, from client device 902. Based on the location information of client device 902, if server 901 determines that client device 902 is approaching an area with limited information in the 3D map managed by server 901, it sends a sensor information transmission request to client device 902 in order to regenerate the 3D map. Alternatively, server 901 may also send sensor information transmission requests when it wants to update the 3D map, check road conditions such as snow accumulation or disasters, or check traffic congestion or accident situations.

[0702] Alternatively, the client device 902 can set the amount of sensor information to be sent to the server 901 based on the communication state or frequency band at the time of receiving the sensor information transmission request from the server 901. Setting the amount of sensor information to be sent to the server 901 can mean, for example, increasing or decreasing the amount of data itself, or selecting an appropriate compression method.

[0703] Figure 85 This is a block diagram illustrating a structural example of client device 902. Client device 902 receives a 3D map, such as a point cloud, from server 901, and estimates its own position based on 3D data generated from sensor information of client device 902. Furthermore, client device 902 sends the obtained sensor information to server 901.

[0704] The client device 902 includes: a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a 3D data production unit 1016, a 3D image processing unit 1017, a 3D data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0705] The data receiving unit 1011 receives the 3D map 1031 from the server 901. The 3D map 1031 includes point cloud data such as WLD or SWLD. The 3D map 1031 may also include either compressed data or uncompressed data.

[0706] The communication unit 1012 communicates with the server 901 and sends data transmission requests (e.g., a 3D map transmission request) to the server 901.

[0707] The receiving control unit 1013 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1012 to establish communication with the communication counterpart.

[0708] The format conversion unit 1014 generates a 3D map 1032 by performing format conversion and other operations on the 3D map 1031 received by the data receiving unit 1011. Furthermore, the format conversion unit 1014 performs decompression or decoding processing when the 3D map 1031 is compressed or encoded. Conversely, the format conversion unit 1014 does not perform decompression or decoding processing when the 3D map 1031 is uncompressed data.

[0709] Multiple sensors 1015 are a group of sensors mounted on the client device 902, such as LiDAR, visible light cameras, infrared cameras, or depth sensors, used to obtain information about the exterior of the vehicle, generating sensor information 1033. For example, if the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as point cloud (point group data). Alternatively, there may be more than one sensor 1015.

[0710] The 3D data production unit 1016 produces 3D data 1034 of the vehicle's surroundings based on sensor information 1033. For example, the 3D data production unit 1016 uses information obtained by LiDAR and visible light images obtained by a visible light camera to produce point cloud data with color information of the vehicle's surroundings.

[0711] The 3D image processing unit 1017 uses the received point cloud and other 3D map 1032, as well as the 3D surrounding data 1034 generated based on sensor information 1033, to perform self-position estimation processing for the vehicle. Alternatively, the 3D image processing unit 1017 can synthesize the 3D map 1032 and the 3D data 1034 to create 3D surrounding data 1035 for the vehicle, and use the created 3D data 1035 to perform self-position estimation processing.

[0712] The 3D data accumulation unit 1018 accumulates 3D maps 1032, 3D data 1034, and 3D data 1035.

[0713] The format conversion unit 1019 generates sensor information 1037 by converting sensor information 1033 into a format corresponding to the receiving side. Furthermore, the format conversion unit 1019 can reduce the data volume by compressing or encoding the sensor information 1037. Moreover, the format conversion unit 1019 can omit processing when format conversion is not required. Furthermore, the format conversion unit 1019 can control the amount of data transmitted according to a specified transmission range.

[0714] The communication unit 1020 communicates with the server 901 and receives data transmission requests (sensor information transmission requests) from the server 901.

[0715] The transmission control unit 1021 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1020, thereby establishing communication.

[0716] The data transmission unit 1022 sends the sensor information 1037 to the server 901. The sensor information 1037 includes, for example, information obtained by LiDAR, brightness images (visible light images) obtained by a visible light camera, infrared images obtained by an infrared camera, depth images obtained by a depth sensor, sensor position information, and velocity information, etc., obtained by multiple sensors 1015.

[0717] Next, the structure of server 901 will be explained. Figure 86 This is a block diagram illustrating an example of the structure of server 901. Server 901 receives sensor information sent from client device 902 and creates 3D data based on the received sensor information. Server 901 uses the created 3D data to update the 3D map managed by server 901. Furthermore, server 901 sends the updated 3D map to client device 902 according to a 3D map sending request from client device 902.

[0718] Server 901 includes: a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a 3D data production unit 1116, a 3D data synthesis unit 1117, a 3D data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0719] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information obtained by LiDAR, brightness images (visible light images) obtained by a visible light camera, infrared images obtained by an infrared camera, depth images obtained by a depth sensor, sensor position information, and speed information.

[0720] The communication unit 1112 communicates with the client device 902 and sends data transmission requests (e.g., sensor information transmission requests) to the client device 902.

[0721] The receiving control unit 1113 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1112, thereby establishing communication.

[0722] When the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 generates sensor information 1132 by performing decompression or decoding processing. Conversely, when the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0723] The 3D data production unit 1116 produces 3D data 1134 of the surrounding area of ​​the client device 902 based on sensor information 1132. For example, the 3D data production unit 1116 uses information obtained by LiDAR and visible light images obtained by a visible light camera to produce point cloud data with color information of the surrounding area of ​​the client device 902.

[0724] The 3D data synthesis unit 1117 synthesizes the 3D data 1134 produced based on sensor information 1132 with the 3D map 1135 managed by the server 901, thereby updating the 3D map 1135.

[0725] The 3D data accumulation unit 1118 accumulates 3D maps 1135, etc.

[0726] The format conversion unit 1119 generates the 3D map 1031 by converting the 3D map 1135 into a format corresponding to the receiving side. Alternatively, the format conversion unit 1119 can reduce the data volume by compressing or encoding the 3D map 1135. Furthermore, the format conversion unit 1119 can omit this process when format conversion is not required. The format conversion unit 1119 can also control the amount of data transmitted according to a specified transmission range.

[0727] The communication unit 1120 communicates with the client device 902 and receives data transmission requests (such as 3D map transmission requests) from the client device 902.

[0728] The transmission control unit 1121 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1120, thereby establishing communication.

[0729] The data transmission unit 1122 sends the 3D map 1031 to the client device 902. The 3D map 1031 includes point cloud data such as WLD or SWLD. The 3D map 1031 may also include either compressed data or uncompressed data.

[0730] Next, the workflow of the client device 902 will be explained. Figure 87 This is a flowchart illustrating the process of acquiring a 3D map using the client device 902.

[0731] First, the client device 902 requests the server 901 to send a 3D map (point cloud, etc.) (S1001). At this time, the client device 902 also sends its location information obtained through GPS, etc., and can then request the server 901 to send a 3D map related to that location information.

[0732] Next, the client device 902 receives a 3D map from the server 901 (S1002). If the received 3D map is compressed data, the client device 902 decodes the received 3D map to generate an uncompressed 3D map (S1003).

[0733] Next, the client device 902 generates 3D data 1034 of its surroundings based on sensor information 1033 obtained from multiple sensors 1015 (S1004). Then, the client device 902 estimates its own position using the 3D map 1032 received from the server 901 and the 3D data 1034 generated from the sensor information 1033 (S1005).

[0734] Figure 88 This is a flowchart illustrating the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a sensor information transmission request from the server 901 (S1011). Having received the transmission request, the client device 902 sends the sensor information 1037 to the server 901 (S1012). Furthermore, if the sensor information 1033 includes multiple pieces of information obtained from multiple sensors 1015, the client device 902 compresses each piece of information using a compression method suitable for that information, thereby generating the sensor information 1037.

[0735] Next, the workflow of server 901 will be explained. Figure 89 This is a flowchart illustrating the operation of server 901 when acquiring sensor information. First, server 901 requests the transmission of sensor information from client device 902 (S1021). Next, server 901 receives sensor information 1037 transmitted from client device 902 according to the request (S1022). Then, server 901 uses the received sensor information 1037 to create 3D data 1134 (S1023). Finally, server 901 reflects the created 3D data 1134 onto a 3D map 1135 (S1024).

[0736] Figure 90This is a flowchart illustrating the operation of server 901 when sending a 3D map. First, server 901 receives a 3D map transmission request from client device 902 (S1031). Having received the 3D map transmission request, server 901 sends the 3D map 1031 to client device 902 (S1032). At this time, server 901 can extract a nearby 3D map corresponding to the location information of client device 902 and send the extracted 3D map. Alternatively, server 901 can compress the 3D map composed of point clouds, for example, using an octree compression method, and send the compressed 3D map.

[0737] Hereinafter, variations of this embodiment will be described.

[0738] Server 901 uses sensor information 1037 received from client device 902 to create 3D data 1134 of the vicinity of client device 902's location. Next, server 901 matches the created 3D data 1134 with a 3D map 1135 of the same area managed by server 901, calculating the difference between the 3D data 1134 and the 3D map 1135. If the difference exceeds a predetermined threshold, server 901 determines that an anomaly has occurred in the vicinity of client device 902. For example, in the event of ground subsidence due to natural disasters such as earthquakes, a significant difference can be considered between the 3D map 1135 managed by server 901 and the 3D data 1134 created based on sensor information 1037.

[0739] Sensor information 1037 may include at least one of the following: sensor type, sensor performance, and sensor model. It may also include a category ID corresponding to the sensor's performance. For example, if sensor information 1037 is information obtained from LiDAR, it's possible to assign identifiers based on sensor performance; for instance, assigning category 1 to sensors capable of obtaining information with an accuracy of a few millimeters, category 2 to sensors capable of obtaining information with an accuracy of a few centimeters, and category 3 to sensors capable of obtaining information with an accuracy of a few meters. Furthermore, server 901 may estimate sensor performance information from the model of client device 902. For example, if client device 902 is mounted on a vehicle, server 901 can determine the sensor's standard information based on the vehicle's model. In this case, server 901 can obtain the vehicle model information beforehand and include that information in the sensor information. It's also possible that server 901 uses the obtained sensor information 1037 to switch the degree of correction for the 3D data 1134 created using sensor information 1037. For example, when the sensor performance is high accuracy (Category 1), server 901 does not perform calibration for the 3D data 1134. When the sensor performance is low accuracy (Category 3), server 901 applies calibration appropriate to the sensor's accuracy to the 3D data 1134. For example, the lower the sensor's accuracy, the greater the degree (intensity) of the calibration.

[0740] Server 901 can also simultaneously send sensor information transmission requests to multiple client devices 902 existing in a certain space. When server 901 receives multiple sensor information from multiple client devices 902, it is not necessary to utilize all the sensor information in the creation of 3D data 1134. For example, it can select the sensor information to be used based on the sensor's performance. For instance, when updating the 3D map 1135, server 901 can select high-precision sensor information (category 1) from the received sensor information and use the selected sensor information to create 3D data 1134.

[0741] Server 901 is not limited to servers such as traffic monitoring cloud servers; it can also be other client devices (vehicle-mounted). Figure 91 The system architecture in this case is shown.

[0742] For example, client device 902C sends a sensor information transmission request to a nearby client device 902A and obtains the sensor information from client device 902A. Then, client device 902C uses the obtained sensor information from client device 902A to create 3D data and update its own 3D map. In this way, client device 902C can utilize its performance to generate a 3D map of the space obtained from client device 902A. This scenario is possible, for example, when client device 902C has high performance.

[0743] Furthermore, in this scenario, the client device 902A, which provides sensor information, is granted the right to obtain a high-precision 3D map generated by the client device 902C. The client device 902A receives the high-precision 3D map from the client device 902C in accordance with this right.

[0744] Alternatively, client device 902C may send sensor information transmission requests to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B is high-performance, client device 902C can use the sensor information obtained through the high-performance sensor to create three-dimensional data.

[0745] Figure 92 This is a block diagram illustrating the functional configuration of server 901 and client device 902. Server 901 includes, for example, a 3D map compression / decoding processing unit 1201 for compressing and decoding 3D maps, and a sensor information compression / decoding processing unit 1202 for compressing and decoding sensor information.

[0746] The client device 902 includes a 3D map decoding processing unit 1211 and a sensor information compression processing unit 1212. The 3D map decoding processing unit 1211 receives compressed coded data of a 3D map, decodes the coded data, and obtains a 3D map. The sensor information compression processing unit 1212 does not compress the 3D data created from the obtained sensor information; instead, it compresses the sensor information itself and sends the compressed coded data of the sensor information to the server 901. With this configuration, the client device 902 can internally house the processing unit (device or LSI) used for decoding 3D maps (point clouds, etc.), without needing to internally house the processing unit used for compressing 3D data (point clouds, etc.). This reduces the cost and power consumption of the client device 902.

[0747] As described above, the client device 902 of this embodiment is mounted on a mobile body. Based on sensor information 1033 obtained by the sensor 1015 mounted on the mobile body, which indicates the surrounding conditions of the mobile body, three-dimensional data 1034 of the mobile body's surroundings is generated. The client device 902 uses the generated three-dimensional data 1034 to estimate the mobile body's own position. The client device 902 sends the obtained sensor information 1033 to the server 901 or other client devices 902.

[0748] Accordingly, the client device 902 sends the sensor information 1033 to the server 901, etc. This potentially reduces the amount of data transmitted compared to transmitting 3D data. Furthermore, since it is unnecessary for the client device 902 to perform 3D data compression or encoding, the processing load on the client device 902 can be reduced. Therefore, the client device 902 can achieve a reduction in the amount of data transmitted or a simplification of the device structure.

[0749] Furthermore, the client device 902 sends a request to the server 901 to send a 3D map, and receives the 3D map 1031 from the server 901. In estimating its own position, the client device 902 uses the 3D data 1034 and the 3D map 1032 to estimate its own position.

[0750] Furthermore, the sensor information 1033 includes at least one of the following: information obtained through the laser sensor, brightness image (visible light image), infrared image, depth image, sensor position information, and sensor speed information.

[0751] Furthermore, sensor information 1033 includes information indicating the performance of the sensor.

[0752] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and during the transmission of sensor information, sends the encoded or compressed sensor information 1037 to the server 901 or other client devices 902. Accordingly, the client device 902 can reduce the amount of data transmitted.

[0753] For example, the client device 902 has a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0754] Furthermore, the server 901 according to this embodiment can communicate with the client device 902 mounted on the mobile body, and receive sensor information 1037 from the client device 902, which is obtained by the sensor 1015 mounted on the mobile body and shows the surrounding conditions of the mobile body. The server 901 generates three-dimensional data 1134 of the surroundings of the mobile body based on the received sensor information 1037.

[0755] Accordingly, server 901 uses sensor information 1037 sent from client device 902 to create 3D data 1134. This potentially reduces the amount of data transmitted compared to when client device 902 sends 3D data directly. Furthermore, since 3D data compression or encoding is unnecessary on client device 902, its processing load can be reduced. Thus, server 901 can achieve a reduction in the amount of data transmitted or a simplification of the device structure.

[0756] Furthermore, server 901 sends a request to the client device 902 to transmit sensor information.

[0757] Furthermore, server 901 uses the generated 3D data 1134 to update 3D map 1135, and sends 3D map 1135 to client device 902 according to the sending request of 3D map 1135 from client device 902.

[0758] Furthermore, the sensor information 1037 includes at least one of the following: information obtained through the laser sensor, brightness image (visible light image), infrared image, depth image, sensor position information, and sensor speed information.

[0759] Furthermore, sensor information 1037 includes information indicating the performance of the sensor.

[0760] Furthermore, server 901 further corrects the 3D data according to the sensor's performance. Therefore, this 3D data creation method can improve the quality of 3D data.

[0761] Furthermore, during the reception of sensor information, server 901 receives multiple sensor information 1037 from multiple client devices 902, and selects the sensor information 1037 to be used in the production of 3D data 1134 based on multiple pieces of information including sensor performance included in the multiple sensor information 1037. Accordingly, server 901 can improve the quality of 3D data 1134.

[0762] Furthermore, server 901 decodes or decompresses the received sensor information 1037, and generates 3D data 1134 based on the decoded or decompressed sensor information 1132. In this way, server 901 can reduce the amount of data transmitted.

[0763] For example, server 901 has a processor and memory, and the processor uses the memory to perform the above-mentioned processing.

[0764] The following is an explanation of the variations. Figure 93 This is a diagram showing the structure of the system in this embodiment. Figure 93The system shown includes server 2001, client device 2002A, and client device 2002B.

[0765] Client devices 2002A and 2002B are mounted on mobile bodies such as vehicles and send sensor information to server 2001. Server 2001 then sends a 3D map (point cloud) to client devices 2002A and 2002B.

[0766] Client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission capability determination unit 2013. Client device 2002B has the same structure. Furthermore, unless otherwise specified, client device 2002A and client device 2002B will also be referred to as client device 2002.

[0767] Figure 94 This is a flowchart illustrating the operation of the client device 2002 according to this embodiment.

[0768] The sensor information acquisition unit 2011 acquires various sensor information using sensors (sensor array) mounted on the moving body. Specifically, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding conditions of the moving body obtained by the sensors (sensor array) mounted on the moving body. Furthermore, the sensor information acquisition unit 2011 stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, visible light images, infrared images, and depth images. Additionally, the sensor information may also include at least one of sensor position information, velocity information, acquisition time information, and acquisition location information. Sensor position information indicates the position of the sensor that acquired the sensor information. Velocity information indicates the velocity of the moving body when the sensor acquired the sensor information. Acquisition time information indicates the time when the sensor acquired the sensor information. Acquisition location information indicates the position of the moving body or the sensor when the sensor acquired the sensor information.

[0769] Next, the data transmission capability determination unit 2013 determines whether the mobile device (client device 2002) exists in an environment capable of sending sensor information to the server 2001 (S2002). For example, the data transmission capability determination unit 2013 may also use information such as GPS to determine the location and time of the client device 2002 and determine whether data can be transmitted. In addition, the data transmission capability determination unit 2013 may also determine whether data can be transmitted by checking whether a connection to a specific access point is possible.

[0770] When the client device 2002 determines that a moving body exists in an environment capable of sending sensor information to the server 2001 (S2002 indicates "Yes"), it sends the sensor information to the server 2001 (S2003). That is, at the point in time when the client device 2002 becomes capable of sending sensor information to the server 2001, the client device 2002 sends the sensor information it holds to the server 2001. For example, a millimeter-wave access point capable of high-speed communication is set up at an intersection. At the point in time the client device 2002 enters the intersection, it uses millimeter-wave communication to send the sensor information held by the client device 2002 to the server 2001 at high speed.

[0771] Next, the client device 2002 deletes the sensor information that has been sent to the server 2001 from the storage unit 2012 (S2004). Alternatively, the client device 2002 may delete the sensor information if no sensor information sent to the server 2001 meets predetermined conditions. For example, the client device 2002 may delete the sensor information from the storage unit 2012 at a time earlier than a certain time prior to the current time when the sensor information was acquired. That is, the client device 2002 may delete the sensor information from the storage unit 2012 if the difference between the time when the sensor information was acquired and the current time exceeds a preset time. Furthermore, the client device 2002 may delete the sensor information from the storage unit 2012 at a time when the distance between the location where the sensor information was acquired and the current location is greater than a certain distance. That is, the client device 2002 may delete the sensor information from the storage unit 2012 if the difference between the position of the moving body or sensor when the sensor information was acquired and the current position of the moving body or sensor exceeds a preset distance. Therefore, the capacity of the storage unit 2012 of the client device 2002 can be suppressed.

[0772] If the acquisition of sensor information by the client device 2002 has not ended ("No" in S2005), the client device 2002 will repeat the processing after step S2001. Alternatively, if the acquisition of sensor information by the client device 2002 has ended ("Yes" in S2005), the client device 2002 will terminate the processing.

[0773] Furthermore, the client device 2002 can also select which sensor information to send to the server 2001 based on the communication conditions. For example, when high-speed communication is possible, the client device 2002 will prioritize sending large-sized sensor information (such as LiDAR acquisition information) stored in the storage unit 2012. Conversely, when high-speed communication is difficult, the client device 2002 will send small-sized, high-priority sensor information (such as visible light images) stored in the storage unit 2012. Thus, the client device 2002 can efficiently send sensor information stored in the storage unit 2012 to the server 2001 according to network conditions.

[0774] Furthermore, the client device 2002 can also obtain time information indicating the current time and location information indicating the current location from the server 2001. Additionally, the client device 2002 can determine the time and location for obtaining the sensor information based on the obtained time information and location information. That is, the client device 2002 can also obtain time information from the server 2001 and use the obtained time information to generate obtained time information. Furthermore, the client device 2002 can also obtain location information from the server 2001 and use the obtained location information to generate obtained location information.

[0775] For example, regarding time information, server 2001 and client device 2002 synchronize their times using mechanisms such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows client device 2002 to obtain accurate time information. Furthermore, since time synchronization is possible between server 2001 and multiple client devices, time synchronization can be achieved within the sensor information obtained by different client devices 2002. Therefore, server 2001 can process the sensor information representing the synchronized time. Alternatively, the time synchronization mechanism can be any method other than NTP or PTP. Furthermore, GPS information can also be used as the aforementioned time and location information.

[0776] Server 2001 can also specify a time or location to obtain sensor information from multiple client devices 2002. For example, in the event of an accident, to locate nearby clients, server 2001 specifies the time and location of the accident and broadcasts a sensor information transmission request to multiple client devices 2002. The client devices 2002, possessing sensor information at the corresponding time and location, then transmit the sensor information to server 2001. That is, client devices 2002 receive sensor information transmission requests from server 2001 containing specified information including the specified location and time. If client devices 2002 store sensor information obtained at the specified location and time in storage unit 2012, and determine that a moving object exists in an environment capable of transmitting sensor information to server 2001, they will transmit the sensor information obtained at the specified location and time to server 2001. Thus, server 2001 can obtain sensor information related to the occurrence of an accident from multiple client devices 2002 for accident analysis, etc.

[0777] Additionally, the client device 2002 can also refuse to send sensor information upon receiving a sensor information transmission request from the server 2001. Furthermore, the client device 2002 can pre-configure which sensor information among multiple sensor information requests can be sent. Alternatively, the server 2001 can query the client device 2002 each time whether sensor information can be sent.

[0778] Furthermore, a score can be assigned to the client device 2002 that sent sensor information to the server 2001. This score can be used for payments such as gasoline purchase fees, EV (Electric Vehicle) charging fees, highway tolls, or car rental fees. Additionally, after receiving the sensor information, the server 2001 can delete information about the client device 2002 used to identify the source of the sensor information. This information could include, for example, the network address of the client device 2002. This anonymizes the sensor information, allowing the user of the client device 2002 to confidently send sensor information from the client device 2002 to the server 2001. Furthermore, the server 2001 can be composed of multiple servers. For example, by having multiple servers share the sensor information, even if one server fails, the others can still communicate with the client device 2002. This avoids service interruptions due to server failures.

[0779] Furthermore, the location specified in the sensor information transmission request, indicating the location of the accident, may sometimes differ from the location of the client device 2002 at the specified time specified in the sensor information transmission request. Therefore, the server 2001 can, for example, specify a range such as within XX meters of the specified location to request information from client devices 2002 located within that range. Similarly, the server 2001 can specify a range such as within N seconds before or after a certain time. Thus, the server 2001 can obtain sensor information from client devices 2002 located "within XX meters of absolute position S" at times "t-N to t+N". The client device 2002 can also transmit data generated immediately following time t when transmitting three-dimensional data such as LiDAR.

[0780] Furthermore, server 2001 can also separately designate the location of client device 2002, which is the object of sensor information acquisition, and the location where sensor information is desired, as designated locations. For example, server 2001 may designate to obtain sensor information from client device 2002, which exists within XXm of absolute position S, including at least a range of YYm from absolute position S. When selecting 3D data to be transmitted, client device 2002 selects at least one randomly accessible unit of 3D data in a manner that includes at least the sensor information within the designated range. Furthermore, when transmitting visible light images, client device 2002 may also transmit multiple temporally consecutive image data, including at least the frame immediately preceding or immediately following time t.

[0781] When the client device 2002 can utilize multiple physical networks, such as 5G, WiFi, or multiple modes of 5G, in transmitting sensor information, the client device 2002 can select the network to use according to the priority order notified by the server 2001. Alternatively, the client device 2002 can select a network that ensures appropriate bandwidth based on the size of the data to be transmitted. Alternatively, the client device 2002 can select the network to use based on factors such as the cost incurred in data transmission. Furthermore, the transmission request from the server 2001 may include information indicating the transmission period, such as whether the client device 2002 can start transmitting before time T. If the server 2001 cannot obtain sufficient sensor information within the period, it can reissue the transmission request.

[0782] Sensor information may also include header information representing characteristics of the sensor data, along with compressed or uncompressed sensor data. Client device 2002 may also send the header information to server 2001 via a different physical network or communication protocol than the sensor data. For example, client device 2002 may send the header information to server 2001 before sending the sensor data. Server 2001 determines whether to obtain the sensor data from client device 2002 based on the parsing results of the header information. For example, the header information may also include information representing LiDAR point group acquisition density, elevation angle, or frame rate, or the resolution, SN ratio, or frame rate of the visible light image. Thus, server 2001 can obtain sensor information from client device 2002 with sensor data of a determined quality.

[0783] As described above, the client device 2002 is mounted on the mobile body, obtains sensor information representing the surrounding conditions of the mobile body from sensors mounted on the mobile body, and stores the sensor information in the storage unit 2012. The client device 2002 determines whether the mobile body exists in an environment capable of sending sensor information to the server 2001. If it determines that the mobile body exists in an environment capable of sending sensor information to the server, it sends the sensor information to the server 2001.

[0784] Furthermore, the client device 2002 further generates three-dimensional data of the surroundings of the moving object based on sensor information, and uses the generated three-dimensional data to estimate the position of the moving object itself.

[0785] Furthermore, the client device 2002 sends a request to the server 2001 to send a 3D map and receives the 3D map from the server 2001. In estimating its own position, the client device 2002 uses 3D data and the 3D map to estimate its own position.

[0786] In addition, the processing performed by the aforementioned client device 2002 can also be implemented as an information sending method in the client device 2002.

[0787] In addition, the client device 2002 may also have a processor and a memory, with the processor using the memory to perform the aforementioned processing.

[0788] Next, the sensor information collection system involved in this embodiment will be described. Figure 95 This is a diagram illustrating the structure of the sensor information collection system according to this embodiment. For example... Figure 95As shown, the sensor information collection system according to this embodiment includes terminal 2021A, terminal 2021B, communication device 2022A, communication device 2022B, network 2023, data collection server 2024, map server 2025, and client device 2026. Furthermore, if terminal 2021A and terminal 2021B are not specifically distinguished, they are also referred to as terminal 2021. If communication device 2022A and communication device 2022B are not specifically distinguished, they are also referred to as communication device 2022.

[0789] The data collection server 2024 collects sensor data and other data obtained by the sensors of the terminal 2021 as location association data corresponding to the location in three-dimensional space.

[0790] Sensor data refers to data obtained using sensors on terminal 2021, such as the surrounding state of terminal 2021 or the internal state of terminal 2021. Terminal 2021 sends sensor data collected from one or more sensor devices located in a position that can communicate directly with terminal 2021 or through one or more relay devices via the same communication method to data collection server 2024.

[0791] The location-associated data may include information such as the operational status of the terminal itself or the devices it possesses, operational logs, and service utilization. Furthermore, the location-associated data may also include information that establishes a correspondence between the terminal 2021's identifier and its location or movement path.

[0792] The location-related data contains location information that corresponds to location information in 3D data, such as 3D map data. Details regarding location information will be described later.

[0793] In addition to location information representing location, location-related data may also include at least one of the following: time information, and information representing the attributes of the data contained in the location-related data or the type (e.g., model) of the sensor that generated the data. Location and time information may also be stored in the header area of ​​the location-related data or the header area of ​​the frame storing the location-related data. Furthermore, location and time information may also be transmitted and / or stored separately from the location-related data as metadata corresponding to it.

[0794] Map server 2025, for example, is connected to network 2023 and sends 3D data, such as 3D map data, according to requests from other devices such as terminal 2021. Furthermore, as described in the various embodiments, map server 2025 may also have the function of updating 3D data using sensor information sent from terminal 2021.

[0795] The data collection server 2024 is connected to the network 2023, for example, and collects location-related data from other devices such as the terminal 2021, and saves the collected location-related data to internal storage devices or storage devices on other servers. In addition, the data collection server 2024 sends the collected location-related data or metadata of the 3D map data generated based on the location-related data to the terminal 2021 according to the request from the terminal 2021.

[0796] Network 2023 is a communication network such as the Internet. Terminal 2021 is connected to network 2023 via communication device 2022. Communication device 2022 communicates with terminal 2021 through one communication method or by switching between multiple communication methods. Communication device 2022 is, for example, (1) a base station of LTE (Long Term Evolution), (2) an access point (AP) of WiFi or millimeter wave communication, (3) a gateway of LPWA (Low Power Wide Area) network such as SIGFOX, LoRaWAN or Wi-SUN, or (4) a communication satellite that communicates using satellite communication methods such as DVB-S2.

[0797] In addition, the base station can communicate with the terminal 2021 either through LPWA, which is classified as NB-IoT (Narrow Band-IoT) or LTE-M, or it can switch between these methods while communicating with the terminal 2021.

[0798] Here, an example is given of a terminal 2021 that has the function of communicating with a communication device 2022 using two communication methods, communicating with a map server 2025 or a data collection server 2024 while using one of these communication methods or switching between multiple communication methods and the communication device 2022 as the direct communication counterpart. However, the structure of the sensor information collection system and the terminal 2021 is not limited to this. For example, the terminal 2021 may not have the communication function under multiple communication methods, but may only have the function of communicating using a single communication method. In addition, the terminal 2021 may correspond to more than three communication methods. Furthermore, each terminal 2021 may correspond to a different communication method.

[0799] Terminal 2021, for example, has Figure 85 The structure of the client device 902 shown is illustrated. The terminal 2021 uses the received 3D data to perform position estimation, including its own position. Furthermore, the terminal 2021 establishes a correspondence between sensor data obtained from sensors and position information obtained from the position estimation process, generating position-related data.

[0800] Location information attached to location-related data can represent, for example, the position in the coordinate system used in the three-dimensional data. For instance, location information can be coordinate values ​​expressed as latitude and longitude. In this case, the terminal 2021 can also include, along with the coordinate values, information about the coordinate system used as the reference for those coordinate values ​​and information about the three-dimensional data used in location estimation. Furthermore, the coordinate values ​​can also include altitude information.

[0801] Furthermore, location information can also be correlated with units of data or space that can be used in the encoding of the aforementioned 3D data. These units may be, for example, WLD, GOS, SPC, VLM, or VXL. In this case, location information is represented, for example, by an identifier used to determine the data unit such as the SPC corresponding to the location-associated data. In addition to the identifier used to determine the data unit such as the SPC, location information may also include information indicating the 3D data obtained by encoding the 3D space including the data unit such as the SPC, or information indicating the detailed location within the SPC. Information representing the 3D data may be, for example, the filename of the 3D data.

[0802] In this way, the syst...

Claims

1. A three-dimensional data encoding method, wherein, The attribute information of multiple 3D points, which are assigned to a certain level in more than one hierarchy based on their respective location information, is encoded. Generate a bitstream containing the encoded attribute information. In the encoding of the attribute information of the three-dimensional points of the processing object contained in the plurality of three-dimensional points, when the number of the more than one level is 1, same-level reference is performed. In the same-level reference, the attribute information of other three-dimensional points contained in the same level as the three-dimensional points of the processing object is referenced to generate the predicted value of the attribute information of the three-dimensional points of the processing object.

2. The three-dimensional data encoding method according to claim 1, wherein, When the number of the above-mentioned levels is 1, the above-mentioned levels do not include levels that do not perform the same-level reference.

3. The three-dimensional data encoding method according to claim 1, wherein, In the encoding of the attribute information of the three-dimensional points of the processed object, if the number of the one or more levels is 1, no upper-level reference is performed; if the number of the one or more levels is 2 or more, the upper-level reference is performed. The upper-level reference refers to the attribute information of other three-dimensional points contained in a higher level than the three-dimensional points of the processed object to generate the predicted value of the attribute information of the three-dimensional points of the processed object.

4. The three-dimensional data encoding method according to claim 1, wherein, When the number of the above-mentioned levels is two or more, determine the levels that perform the same-level reference and the levels that do not perform the same-level reference. Based on the decision, the attribute information of the plurality of three-dimensional points is encoded.

5. The three-dimensional data encoding method according to claim 4, wherein, In the encoding of the attribute information of the multiple three-dimensional points, the same-level reference is performed on the layers from the topmost layer to the Nth layer in the two or more levels, and the same-level reference is not performed on the layers below the Nth layer, where N is a natural number.

6. The three-dimensional data encoding method according to claim 1, wherein, The bitstream contains first information, which indicates whether the same-level reference is performed or not.

7. The three-dimensional data encoding method according to claim 1, wherein, The bitstream contains a second piece of information representing the number of the one or more levels.

8. A three-dimensional data decoding method, wherein, Encoded attribute information of multiple 3D points is obtained from the bitstream. These multiple 3D points are assigned to one or more levels based on their respective position information. When the number of the above-mentioned levels is 1, same-level reference is performed, and the encoded attribute information is decoded. The same-level reference refers to the attribute information of other three-dimensional points in the same level as the three-dimensional points of the processing object contained in the multiple three-dimensional points to generate the predicted value of the attribute information of the three-dimensional points of the processing object.

9. The three-dimensional data decoding method according to claim 8, wherein, When the number of the above-mentioned levels is 1, the above-mentioned levels do not include levels that do not perform the same-level reference.

10. The three-dimensional data decoding method according to claim 8, wherein, In the decoding of the encoded attribute information of the three-dimensional points of the processed object, if the number of the above-one or more levels is 1, no upper-level reference is performed; if the number of the above-one or more levels is 2 or more, the upper-level reference is performed. The upper-level reference refers to the attribute information of other three-dimensional points contained in a higher level than the three-dimensional points of the processed object to generate the predicted value of the attribute information of the three-dimensional points of the processed object.

11. The three-dimensional data decoding method according to claim 8, wherein, When the number of the above-mentioned levels is two or more, the levels that perform the same-level reference and the levels that do not perform the same-level reference are determined, and based on the determination, the encoded attribute information is decoded.

12. The three-dimensional data decoding method according to claim 11, wherein, In the decoding of the encoded attribute information, the same-level reference is performed on the levels from the topmost level to the Nth level in the two or more levels, and the same-level reference is not performed on the levels below the Nth level, where N is a natural number.

13. The three-dimensional data decoding method according to claim 8, wherein, The first information is obtained from the bit stream, the first information representing the level that performs the same-level reference or the level that does not perform the same-level reference.

14. The three-dimensional data decoding method according to claim 8, wherein, Second information representing the number of the one or more levels is obtained from the bit stream.

15. The three-dimensional data decoding method according to claim 8, wherein, From the bitstream, obtain (1) first information indicating whether the same-level reference is performed or not, and (2) second information indicating the number of the one or more levels. If the number of the more than one levels shown in the second information is 1, and if the first information does not indicate that the same-level reference is performed in one of the more than one levels, the bitstream is determined to be non-compliant with the standard.

16. A three-dimensional data encoding device, wherein, have: processor; as well as memory, The processor uses the memory. The attribute information of multiple 3D points, which are assigned to a certain level in more than one hierarchy based on their respective location information, is encoded. Generate a bitstream containing the encoded attribute information. In the encoding of the attribute information of the three-dimensional points of the processing object contained in the plurality of three-dimensional points, when the number of the more than one level is 1, same-level reference is performed. In the same-level reference, the attribute information of other three-dimensional points contained in the same level as the three-dimensional points of the processing object is referenced to generate the predicted value of the attribute information of the three-dimensional points of the processing object.

17. A three-dimensional data decoding device, wherein, have: Processor; and memory, The processor uses the memory. Encoded attribute information of multiple 3D points is obtained from the bitstream. These multiple 3D points are assigned to one or more levels based on their respective position information. When the number of the above-mentioned levels is 1, same-level reference is performed, and the encoded attribute information is decoded. In the same-level reference, the attribute information of the three-dimensional points of the processing object contained in the multiple three-dimensional points is referenced to generate the predicted value of the attribute information of the three-dimensional points of the processing object.

Images