Three-dimensional data coding method, three-dimensional data decoding method, three-dimensional data coding device, and three-dimensional data decoding device

Abstract

A three-dimensional data encoding method capable of providing a random access function in encoded three-dimensional data is provided. The three-dimensional data encoding method is a three-dimensional data encoding method for encoding three-dimensional data (111), and includes a division step (S103) for dividing the three-dimensional data (111) into first processing units, each of which is a random access unit and corresponds to a three-dimensional coordinate, and an encoding step (S104) for generating encoded three-dimensional data (112) by encoding each of a plurality of first processing units.

Description

Technology Field

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Technology

[0002] In a wide range of fields, such as computer vision for the autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, or image transmission, the widespread adoption of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired through various methods, including distance sensors such as rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0003] One of the methods for representing three-dimensional data is a method called a point cloud, which represents the shape of a three-dimensional structure using a point cloud in three-dimensional space (see, for example, Non-patent Literature 1). In a point cloud, the location and color of the point cloud are stored. The point cloud is expected to become the mainstream method for representing three-dimensional data, but the amount of data in the point cloud is very large. Therefore, in the storage or transmission of three-dimensional data, compression of the amount of data through encoding is essential, just as with two-dimensional video (for example, MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, point cloud compression is partially supported by open libraries (Point Cloud Libraries) that perform point cloud-related processing. Prior art literature

[0005] “Octree-Based Progressive Geometry Coding of Point Clouds”, Eurographics Symposium on Point-Based Graphics (2006)

[0006] However, when using encoded data such as point clouds in actual devices or services, random access to desired spatial locations or objects is essential, but until now, random access in three-dimensional encoded data has not existed as a function, nor has there been an encoding method for it.

[0007] The present disclosure aims to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device capable of providing a random access function in encoded three-dimensional data.

[0008] In addition, the present disclosure enables spatial quantization, prediction, etc. through a configuration that divides and encodes space, and is valid even when random access is not necessarily performed.

[0009] A point cloud three-dimensional data encoding method according to one aspect of the present disclosure is a point cloud three-dimensional data encoding method for encoding point cloud three-dimensional data, comprising: a dividing step of dividing the point cloud three-dimensional data into first processing units, each of which is a random access unit and each corresponds to a three-dimensional coordinate; and an encoding step of generating encoded data by encoding each of the plurality of first processing units.

[0010] A point cloud three-dimensional data decoding method according to one aspect of the present disclosure is a point cloud three-dimensional data decoding method for decoding point cloud three-dimensional data, comprising a decoding step of generating three-dimensional data of a first processing unit by decoding each of the encoded data of a first processing unit, each of which is a random access unit and each corresponds to a three-dimensional coordinate.

[0011] In addition, the general or specific modes thereof may be realized as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0012] The present disclosure may provide a point cloud three-dimensional data encoding method, a point cloud three-dimensional data decoding method, a point cloud three-dimensional data encoding device, or a point cloud three-dimensional data decoding device capable of providing a random access function in encoded three-dimensional data. Brief explanation of the drawing

[0013] FIG. 1 is a diagram illustrating the configuration of encoded three-dimensional data according to an embodiment. FIG. 2 is a diagram illustrating an example of a prediction structure between SPCs belonging to the lowest layer of GOS according to an embodiment. FIG. 3 is a drawing illustrating an example of a predicted structure between layers according to an embodiment. FIG. 4 is a diagram illustrating an example of the encoding sequence of GOS according to an embodiment. FIG. 5 is a diagram illustrating an example of the encoding sequence of GOS according to an embodiment. FIG. 6 is a block diagram of a three-dimensional data encoding device according to an embodiment. FIG. 7 is a flowchart of encoding processing according to an embodiment. FIG. 8 is a block diagram of a three-dimensional data decoding device according to an embodiment. FIG. 9 is a flowchart of decoding processing according to an embodiment. FIG. 10 is a drawing illustrating an example of meta-information according to an embodiment. Specific details for implementing the invention

[0014] A point cloud three-dimensional data encoding method according to one aspect of the present disclosure is a point cloud three-dimensional data encoding method for encoding point cloud three-dimensional data, comprising: a dividing step of dividing the point cloud three-dimensional data into first processing units, each of which is a random access unit and each corresponds to a three-dimensional coordinate; and an encoding step of generating encoded data by encoding each of the plurality of first processing units.

[0015] According to this, random access is possible for each first processing unit. In this way, the three-dimensional data encoding method can provide a random access function in the encoded three-dimensional data.

[0016] For example, the above three-dimensional data encoding method includes a generation step of generating first information representing three-dimensional coordinates corresponding to each of the plurality of first processing units, and the encoded data may include the first information.

[0017] For example, the first information may also represent at least one of an object, a time, and a data storage location corresponding to each of the plurality of first processing units.

[0018] For example, in the above-mentioned splitting step, the first processing unit may also be divided into a plurality of second processing units, and in the above-mentioned encoding step, each of the plurality of second processing units may be encoded.

[0019] For example, in the above encoding step, a second processing unit of a processing target included in a first processing unit of a processing target may be encoded by referring to another second processing unit included in the first processing unit of the processing target.

[0020] According to this, encoding efficiency can be improved by referring to another second processing unit.

[0021] For example, in the encoding step, any one of a first type that does not refer to another second processing unit, a second type that refers to another second processing unit, and a third type that refers to two other second processing units may be selected as the type of the second processing unit of the processing target, and the second processing unit of the processing target may be encoded according to the selected type.

[0022] For example, in the above encoding step, the frequency of selecting the first type may be changed depending on the number or density of objects included in the three-dimensional data.

[0023] According to this, random accessibility and encoding efficiency, which are in a trade-off relationship, can be appropriately set.

[0024] For example, in the above encoding step, the size of the first processing unit may be determined according to the number or density of objects or dynamic objects included in the three-dimensional data.

[0025] According to this, random accessibility and encoding efficiency, which are in a trade-off relationship, can be appropriately set.

[0026] For example, the first processing unit may include a plurality of layers, each of which is spatially divided in a predetermined direction and includes one or more of the second processing units, and in the encoding step, the second processing unit may be encoded by referring to the second processing unit included in the same layer as the second processing unit or in a layer lower than the second processing unit.

[0027] According to this, for example, the random accessibility of important layers in the system can be improved, and the degradation of encoding efficiency can be suppressed.

[0028] For example, in the above-mentioned splitting step, a second processing unit containing only static objects and a second processing unit containing only dynamic objects may be assigned to different first processing units.

[0029] According to this, control of dynamic and static objects can be easily performed.

[0030] For example, in the above encoding step, a plurality of dynamic objects may be encoded individually, and the encoded data of the plurality of dynamic objects may correspond to a second processing unit containing only static objects.

[0031] According to this, control of dynamic and static objects can be easily performed.

[0032] For example, in the above-mentioned splitting step, the second processing unit may also be divided into a plurality of third processing units, and in the above-mentioned encoding step, each of the plurality of third processing units may be encoded.

[0033] For example, the third processing unit may include one or more voxels, which are the minimum units to which location information corresponds.

[0034] For example, the second processing unit may include a group of feature points derived from information obtained from a sensor.

[0035] For example, the above-mentioned encoding data may include information indicating the encoding order of the plurality of first processing units.

[0036] For example, the above-mentioned encoding data may include information indicating the size of the plurality of first processing units.

[0037] For example, in the above encoding step, the plurality of first processing units may be encoded in parallel.

[0038] In addition, a point cloud three-dimensional data decoding method according to one aspect of the present disclosure is a point cloud three-dimensional data decoding method for decoding point cloud three-dimensional data, and includes a decoding step of generating point cloud three-dimensional data of a first processing unit by decoding each of the encoded data of a first processing unit, each of which is a random access unit and each corresponds to a three-dimensional coordinate.

[0039] According to this, random access is possible for each first processing unit. In this way, the three-dimensional data decoding method can provide a random access function in encoded three-dimensional data.

[0040] In addition, a point cloud three-dimensional data encoding device according to one embodiment of the present disclosure may include a dividing unit that divides the point cloud three-dimensional data into first processing units, each of which is a random access unit and each corresponds to a three-dimensional coordinate, and an encoding unit that generates encoded data by encoding each of the plurality of first processing units.

[0041] According to this, random access is possible for each first processing unit. In this way, the three-dimensional data encoding device can provide a random access function in the encoded three-dimensional data.

[0042] In addition, a point cloud three-dimensional data decoding device according to one aspect of the present disclosure may include a decoding unit that generates point cloud three-dimensional data of a first processing unit by decoding each of the encoded data of a first processing unit, each of which is a random access unit and each corresponds to a three-dimensional coordinate.

[0043] According to this, random access is possible for each first processing unit. In this way, the three-dimensional data decoding device can provide a random access function in encoded three-dimensional data.

[0044] In addition, comprehensive or specific embodiments thereof may be realized as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0045] Hereinafter, embodiments will be described in detail with reference to the drawings. Furthermore, the embodiments described below represent specific embodiments of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection types of components, steps, and sequence of steps shown in the following embodiments are examples and are not well known facts that limit the present disclosure. Additionally, regarding the components in the following embodiments, any component not described in the independent claim representing the highest concept is described as any component.

[0046] First, the data structure of the encoded three-dimensional data (hereinafter referred to as encoded data) according to the present embodiment will be described. FIG. 1 is a diagram illustrating the configuration of the encoded three-dimensional data according to the present embodiment.

[0047] In the present embodiment, the three-dimensional space is divided into spaces (SPCs) corresponding to pictures in video encoding, and three-dimensional data is encoded using the spaces as units. The spaces are also divided into volumes (VLMs) corresponding to macroblocks, etc. in video encoding, and prediction and transformation are performed using the VLMs as units. The volumes include a plurality of voxels (VXLs), which are the smallest units to which position coordinates correspond. Furthermore, prediction involves generating predicted three-dimensional data similar to the processing unit to be processed by referencing another processing unit, just as prediction is performed on a two-dimensional image, and encoding the difference between said predicted three-dimensional data and the processing unit to be processed. In addition, this prediction includes not only spatial prediction referencing another prediction unit at the same time, but also temporal prediction referencing a prediction unit at a different time.

[0048] For example, when a three-dimensional data encoding device (hereinafter referred to as the encoding device) encodes a three-dimensional space represented by point cloud data, such as a point cloud, it groups each point of the point cloud or multiple points contained within a voxel and encodes them according to the size of the voxel. Subdividing the voxel allows the three-dimensional shape of the point cloud to be represented with high precision, while increasing the size of the voxel allows the three-dimensional shape of the point cloud to be represented approximately.

[0049] In addition, the following explanation uses the case where the three-dimensional data is a point cloud as an example; however, the three-dimensional data is not limited to a point cloud and can be any format of three-dimensional data.

[0050] Alternatively, a hierarchical voxel structure may be used. In this case, for the n-th order layer, whether a sample point exists in the layer of order (n-1) or lower (the lower layer of the n-th order layer) may be indicated sequentially. For example, when decoding only the n-th order layer, if a sample point exists in the layer of order (n-1) or lower, decoding can be performed by assuming that the sample point exists at the center of the voxel of the n-th order layer.

[0051] In addition, the encoding device acquires point cloud data using a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor, etc.

[0052] A space is classified into at least three prediction structures, including an intra-space (I-SPC) that can be decoded independently, a predictive space (P-SPC) that can only be referenced unidirectionally, and a bidirectional space (B-SPC) that can be referenced bidirectionally, just like video encoding. Additionally, the space has two types of time information: a decoding time and a display time.

[0053] Also, as illustrated in FIG. 1, there exists a Group of Space (GOS), which is a random access unit, as a processing unit including multiple spaces. In addition, there exists a World (WLD) as a processing unit including multiple GOSs.

[0054] The spatial area occupied by the world corresponds to an absolute location on Earth using GPS or latitude and longitude information. This location information is stored as metadata. Additionally, the metadata may be included in the encoded data or transmitted separately from the encoded data.

[0055] Additionally, within GOS, all SPCs may be three-dimensionally adjacent, and there may exist SPCs that are not three-dimensionally adjacent to other SPCs.

[0056] Furthermore, hereinafter, processing such as encoding, decoding, or referencing of three-dimensional data included in a processing unit such as a GOS, SPC, or VLM is simply described as encoding, decoding, or referencing the processing unit. Also, the three-dimensional data included in the processing unit includes, for example, at least one set of spatial positions such as three-dimensional coordinates and characteristic values ​​such as color information.

[0057] Next, the prediction structure of the SPC in the GOS is explained. Multiple SPCs within the same GOS, or multiple VLMs within the same SPC, occupy different spaces but possess the same time information (decoding time and display time).

[0058] Also, the SPC that is the leading in the decoding order within the GOS is the I-SPC. Additionally, there are two types of GOS: closed GOS and open GOS. A closed GOS is a GOS that can decode all SPCs within the GOS when decoding is started from the leading I-SPC. In an open GOS, some SPCs within the GOS whose display time is earlier than the leading I-SPC refer to a different GOS, and decoding cannot be performed using only that GOS.

[0059] Furthermore, in encoded data such as map information, WLDs are sometimes decoded in the reverse order of encoding, and reverse reconstruction is difficult if there are dependencies between GOSs. Therefore, in such cases, a closed GOS is basically used.

[0060] In addition, the GOS has a layered structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the layer below.

[0061] Figure 2 is a diagram illustrating an example of a prediction structure between SPCs belonging to the lowest layer of GOS. Figure 3 is a diagram illustrating an example of a prediction structure between layers.

[0062] One or more I-SPCs exist within the GOS. In three-dimensional space, there are objects such as people, animals, cars, bicycles, signals, or landmark buildings, and particularly small objects are valid when encoded as I-SPCs. For example, when a three-dimensional data decoder (hereinafter referred to as the decoder) decodes the GOS at low throughput or high speed, it decodes only the I-SPCs within the GOS.

[0063] Additionally, the encoding device may change the encoding interval or occurrence frequency of I-SPC according to the density of objects within the WLD.

[0064] In addition, in the configuration illustrated in FIG. 3, the encoding device or the decoding device sequentially encodes or decodes a plurality of layers starting from the lower layer (Layer 1). By doing so, for example, in the case of an autonomous vehicle, The priority of data near a surface with more information can be increased.

[0065] In addition, for encoded data used in drones, etc., encoding or decoding may be performed sequentially from the SPC of the upper layer in the height direction within the GOS.

[0066] In addition, the encoding device or the decoding device may encode or decode multiple layers so that the decoding device can roughly determine the GOS and gradually increase the resolution. For example, the encoding device or the decoding device may encode or decode in the order of layers 3, 8, 1, 9…

[0067] Next, the handling methods for static and dynamic objects will be explained.

[0068] In three-dimensional space, there exist static objects or scenes, such as buildings or roads (hereinafter referred to as bundled static objects), and dynamic objects, such as cars or people (hereinafter referred to as dynamic objects). Object detection is performed separately, such as by extracting feature points from point cloud data or camera images, such as those from stereo cameras. Here, an example of an encoding method for dynamic objects is described.

[0069] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects based on identification information.

[0070] For example, a GOS is used as an identification unit. In this case, a GOS containing only SPCs constituting static objects and a GOS containing SPCs constituting dynamic objects are distinguished by identification information stored within the encoded data or separately from the encoded data.

[0071] Alternatively, an SPC may be used as an identification unit. In this case, an SPC containing only VLMs constituting static objects and an SPC containing VLMs constituting dynamic objects are distinguished by the identification information.

[0072] Alternatively, a VLM or VXL may be used as an identification unit. In this case, a VLM or VXL containing only static objects and a VLM or VXL containing dynamic objects are distinguished by the identification information.

[0073] In addition, the encoding device may encode a dynamic object as one or more VLMs or SPCs, and may encode a VLM or SPC containing only static objects and an SPC containing dynamic objects as different GOSs. In addition, if the size of the GOS varies depending on the size of the dynamic object, the encoding device stores the size of the GOS separately as metadata.

[0074] In addition, the encoding device may encode static objects and dynamic objects independently of each other, and may superimpose dynamic objects onto a world composed of static objects. In this case, the dynamic objects are composed of one or more SPCs, and each SPC corresponds to one or more SPCs that constitute the static objects to which the said SPC is superimposed. In addition, dynamic objects may be represented by one or more VLMs or VXLs instead of SPCs.

[0075] Additionally, the encoding device may encode static objects and dynamic objects as different streams.

[0076] Additionally, the encoding device may generate a GOS containing one or more SPCs that constitute a dynamic object. Furthermore, the encoding device may set the GOS containing the dynamic object (GOS_M) and the GOS of the static object corresponding to the spatial area of ​​GOS_M to be of the same size (occupy the same spatial area). By doing so, overlapping processing can be performed on a GOS basis.

[0077] P-SPCs or B-SPCs constituting a dynamic object may reference SPCs included in different GOSs that have finished encoding. In cases where the location of a dynamic object changes over time and the same dynamic object is encoded as a GOS at a different time, referencing across GOSs becomes valid from the perspective of compression ratio.

[0078] Additionally, depending on the intended use of the encoded data, the first and second methods described above may be interchanged. For example, when the encoded three-dimensional data is used as a map, the encoding device uses the second method because it is desirable to be able to separate dynamic objects. On the other hand, when the encoding device is encoding three-dimensional data for events such as concerts or sports, if there is no need to separate dynamic objects, the first method is used.

[0079] In addition, the decoding time and display time of the GOS or SPC may be stored within the encoded data or as metadata. Also, the time information of the static object may all be identical. In this case, the actual decoding time and display time may be determined by the decoder. Alternatively, different values ​​may be assigned to each GOS or SPC as the decoding time, and the same value may be assigned to all as the display time. Furthermore, a model may be introduced that guarantees that the decoder can decode without corruption by having a buffer of a predetermined size and reading a bit stream at a predetermined bit rate according to the decoding time, such as the decoder model in motion encoding like HEVC’s HRD (Hypothetical Reference Decoder).

[0080] Next, the arrangement of GOS within the world is described. The coordinates of the three-dimensional space within the world are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, and z-axis). By establishing a predetermined rule for the encoding order of the GOS, encoding can be performed such that spatially adjacent GOS are continuous within the encoded data. For example, in the example illustrated in Fig. 4, GOS within the xz plane are encoded continuously. After the encoding of all GOS within a given xz plane is completed, the value of the y-axis is updated. That is, as encoding proceeds, the world expands in the y-axis direction. In addition, the index numbers of the GOS are set in the encoding order.

[0081] Here, the three-dimensional space of the world corresponds one-to-one with geographical absolute coordinates such as GPS, latitude, and longitude. Alternatively, the three-dimensional space may be represented by a relative position from a pre-set reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, and said direction vectors are stored as meta-information along with encoded data.

[0082] In addition, the size of the GOS is fixed, and the encoding device stores the said size as metadata. Furthermore, the size of the GOS may be changed depending, for example, whether it is a city or not, or whether it is indoors or outdoors. That is, the size of the GOS may be changed according to the quantity or nature of objects that have informational value. Alternatively, the encoding device may adaptively change the size of the GOS or the interval of the I-SPC within the GOS according to the density of objects within the same world. For example, the encoding device reduces the size of the GOS and shortens the interval of the I-SPC within the GOS as the density of objects increases.

[0083] In the example of Fig. 5, the GOS is subdivided to realize random access at fine particle size because the object density is high in the regions of the 3rd to 10th GOS. Additionally, the 7th to 10th GOS are each located on the opposite side of the 3rd to 6th GOS.

[0084] Next, the configuration and operation flow of the three-dimensional data encoding device according to the present embodiment will be described. FIG. 6 is a block diagram of the three-dimensional data encoding device (100) according to the present embodiment. FIG. 7 is a flowchart showing an example of the operation of the three-dimensional data encoding device (100).

[0085] The three-dimensional data encoding device (100) illustrated in FIG. 6 generates encoded three-dimensional data (112) by encoding three-dimensional data (111). The three-dimensional data encoding device (100) comprises an acquisition unit (101), an encoding area determination unit (102), a division unit (103), and an encoding unit (104).

[0086] As illustrated in FIG. 7, first, the acquisition unit (101) acquires three-dimensional data (111), which is point cloud data (S101).

[0087] Next, the encoding area determination unit (102) determines the area to be encoded among the spatial areas corresponding to the acquired point cloud data (S102). For example, the encoding area determination unit (102) determines the spatial area surrounding the location of the user or vehicle as the area to be encoded according to the location of the user or vehicle.

[0088] Next, the dividing unit (103) divides the point cloud data included in the area of ​​the encoding target into each processing unit. Here, the processing unit is the GOS and SPC described above. Also, the area of ​​the encoding target corresponds, for example, to the world described above. Specifically, the dividing unit (103) divides the point cloud data into processing units based on the size of the GOS set in advance, or the presence or size of dynamic objects (S103). Also, the dividing unit (103) determines the starting position of the SPC that is the leading in the encoding order for each GOS.

[0089] Next, the encoding unit (104) generates encoded three-dimensional data (112) by sequentially encoding a plurality of SPCs within each GOS (S104).

[0090] In addition, the above example illustrates dividing the region to be encoded into GOS and SPC and then encoding each GOS, but the order of processing is not limited to the above. For example, a sequence such as determining the configuration of one GOS, encoding that GOS, and then determining the configuration of the next GOS may be used.

[0091] In this way, the three-dimensional data encoding device (100) generates encoded three-dimensional data (112) by encoding three-dimensional data (111). Specifically, the three-dimensional data encoding device (100) divides the three-dimensional data into first processing units (GOS), each of which is a random access unit and corresponds to a three-dimensional coordinate, divides the first processing units (GOS) into a plurality of second processing units (SPC), and divides the second processing units (SPC) into a plurality of third processing units (VLM). In addition, the third processing units (VLM) include one or more voxels (VXL), which are the minimum units to which position information corresponds.

[0092] Next, the three-dimensional data encoding device (100) generates encoded three-dimensional data (112) by encoding each of a plurality of first processing units (GOS). Specifically, the three-dimensional data encoding device (100) encodes each of a plurality of second processing units (SPC) in each first processing unit (GOS). In addition, the three-dimensional data encoding device (100) encodes each of a plurality of third processing units (VLM) in each second processing unit (SPC).

[0093] For example, when the first processing unit (GOS) of the processing target is a closed GOS, the three-dimensional data encoding device (100) encodes the second processing unit (SPC) of the processing target included in the first processing unit (GOS) of the processing target by referring to another second processing unit (SPC) included in the first processing unit (GOS) of the processing target. That is, the three-dimensional data encoding device (100) does not refer to the second processing unit (SPC) included in the first processing unit (GOS) that is different from the first processing unit (GOS) of the processing target.

[0094] Meanwhile, if the first processing unit (GOS) of the processing target is an open GOS, the second processing unit (SPC) of the processing target included in the first processing unit (GOS) of the processing target is encoded by referring to a different second processing unit (SPC) included in the first processing unit (GOS) of the processing target, or a second processing unit (SPC) included in a first processing unit (GOS) that is different from the first processing unit (GOS) of the processing target.

[0095] Additionally, the three-dimensional data encoding device (100) selects one of a first type (I-SPC) that does not refer to another second processing unit (SPC), a second type (P-SPC) that refers to another second processing unit (SPC), and a third type that refers to two other second processing units (SPC), as a type of the second processing unit (SPC) of the processing target, and encodes the second processing unit (SPC) of the processing target according to the selected type.

[0096] Next, the configuration and operation flow of the three-dimensional data decoding device according to the present embodiment will be described. FIG. 8 is a block diagram of the blocks of the three-dimensional data decoding device (200) according to the present embodiment. FIG. 9 is a flowchart showing an example of the operation of the three-dimensional data decoding device (200).

[0097] The three-dimensional data decoding device (200) illustrated in FIG. 8 generates decoded three-dimensional data (212) by decoding encoded three-dimensional data (211). Here, the encoded three-dimensional data (211) is, for example, encoded three-dimensional data (112) generated by a three-dimensional data encoding device (100). The three-dimensional data decoding device (200) comprises an acquisition unit (201), a decoding start GOS determination unit (202), a decoding SPC determination unit (203), and a decoding unit (204).

[0098] First, the acquisition unit (201) acquires encoded three-dimensional data (211) (S201). Next, the decoding start GOS determination unit (202) determines the GOS of the decoding target (S202). Specifically, the decoding start GOS determination unit (202) determines the GOS of the decoding target by referring to metadata stored within the encoded three-dimensional data (211) or separately from the encoded three-dimensional data, and includes a GOS that includes an SPC corresponding to a spatial location, object, or time to start decoding.

[0099] Next, the decoding SPC determination unit (203) determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the decoding SPC determination unit (203) determines whether to (1) decode only the I-SPC, (2) decode the I-SPC and P-SPC, or (3) decode all types. Additionally, if the type of SPC to be decoded is determined in advance, such as decoding all SPCs, this step does not need to be performed.

[0100] Next, the decoding unit (204) obtains an address position that the SPC leading in the decoding order (same as the encoding order) within the GOS starts in the encoded three-dimensional data (211), obtains the encoded data of the leading SPC from the said address position, and sequentially decodes each SPC in order starting from the said leading SPC (S204). In addition, the said address position is stored in meta information, etc.

[0101] In this way, the three-dimensional data decoding device (200) decodes the decoded three-dimensional data (212). Specifically, the three-dimensional data decoding device (200) generates the decoded three-dimensional data (212) of the first processing unit (GOS) by decoding each of the encoded three-dimensional data (211) of the first processing unit (GOS), each of which is a random access unit and corresponds to a three-dimensional coordinate. More specifically, the three-dimensional data decoding device (200) decodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Also, the three-dimensional data decoding device (200) decodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0102] Hereinafter, meta-information for random access is described. This meta-information is generated by a three-dimensional data encoding device (100) and is included in encoded three-dimensional data (112 (211)).

[0103] In the case of random access in conventional two-dimensional video images, decoding was initiated from the leading frame of a random access unit that is near a specified time. On the other hand, in the case of a world, random access to space (coordinates or objects, etc.) is assumed in addition to time.

[0104] Therefore, in order to realize random access to at least three elements—coordinates, objects, and time—a table is prepared that corresponds each element to an index number of the GOS. Additionally, an index number of the GOS is corresponded to the address of the I-SPC that is the leader of the GOS. FIG. 10 is a diagram illustrating an example of a table included in meta-information. Furthermore, it is not necessary to use all the tables illustrated in FIG. 10, and at least one table needs to be used.

[0105] Below, as an example, random access based on coordinates is described. When accessing coordinates (x2, y2, z2), first, the coordinate-GOS table is referenced to determine that the point at coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is referenced to determine that the address of the leading I-SPC in the second GOS is addr (2), so the decoding unit (204) obtains data from this address and starts decoding.

[0106] In addition, the address may be an address in a logical format, or a physical address in an HDD or memory. Also, information identifying a file segment may be used instead of an address. For example, a file segment is a unit segmented from one or more GOSs.

[0107] Additionally, if an object spans multiple GOSs, the object-GOS table may list multiple GOSs to which the object belongs. If the multiple GOSs are closed GOSs, the encoding device and the decoding device can perform encoding or decoding in parallel. On the other hand, if the multiple GOSs are open GOSs, compression efficiency can be further improved by the multiple GOSs referencing each other.

[0108] Examples of objects include people, animals, cars, bicycles, signals, or buildings that serve as landmarks. For example, a three-dimensional data encoding device (100) can extract characteristic points unique to an object from a three-dimensional point cloud, etc., during world encoding, detect an object based on said characteristic points, and set the detected object as a random access point.

[0109] In this way, the three-dimensional data encoding device (100) generates first information representing three-dimensional coordinates corresponding to each of the plurality of first processing units (GOS). Also, the encoded three-dimensional data (112 (211)) includes this first information. Also, the first information represents at least one of an object, a time, and a data storage location corresponding to each of the plurality of first processing units (GOS).

[0110] The three-dimensional data decoding device (200) obtains first information from encoded three-dimensional data (211), uses the first information to identify the encoded three-dimensional data (211) of the first processing unit corresponding to a specified three-dimensional coordinate, object, or time, and decodes the encoded three-dimensional data (211).

[0111] Examples of other meta-information are described below. In addition to meta-information for random access, the three-dimensional data encoding device (100) may generate and store meta-information as follows. Also, the three-dimensional data decoding device (200) may use this meta-information during decoding.

[0113] *

[0114] In cases where three-dimensional data is used as map information, a profile is defined according to the purpose, and information representing said profile may be included in the metadata. For example, a profile dedicated to urban areas or suburbs, or dedicated to flying objects, is defined, and for each, the maximum or minimum sizes of the World, SPC, or VLM are defined. For instance, since more detailed information is required for urban areas than for suburbs, the minimum size of the VLM is set smaller.

[0115] Meta information may include tag values ​​indicating the type of object. These tag values ​​correspond to the VLM, SPC, or GOS that constitute the object. For example, tag values ​​may be set for each type of object, such as tag value "0" representing a "person," tag value "1" representing a "vehicle," and tag value "2" representing a "signal." Alternatively, if it is difficult to determine the type of object or if determination is not necessary, tag values ​​indicating size, or properties such as dynamic or static objects, may be used.

[0116] Additionally, the meta-information may include information indicating the range of the spatial area occupied by the world.

[0117] Additionally, the metadata may store the size of the SPC or VXL as common header information for the entire stream of encoded data or for multiple SPCs, such as the SPC within the GOS.

[0118] Additionally, the metadata may include identification information, such as distance sensors or cameras used to generate the point cloud, or information indicating the positional precision of the point cloud.

[0119] Additionally, the meta-information may include information indicating whether the world consists only of static objects or includes dynamic objects.

[0120] Variations of the present embodiment will be described below.

[0121] The encoding device or the decoding device may encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel may be determined based on meta-information indicating the spatial location of the GOS.

[0122] In cases where three-dimensional data is used as a spatial map when a vehicle or flying object moves, or when such a spatial map is generated, the encoding device or the decoding device may encode or decode the GOS or SPC included in the space specified based on GPS, route information, or zoom magnification.

[0123] Additionally, the decoder may perform decoding sequentially starting from the space close to its own location or the driving path. The encoding device or the decoder may encode or decode the space far from its own location or the driving path with lower priority compared to the space close to it. Here, lowering priority means lowering the processing order, lowering the resolution (filtering out processing), or lowering the image quality (increasing encoding efficiency; for example, increasing the quantization stage), etc.

[0124] In addition, when the decoding device decodes encoded data that is hierarchically encoded within a space, it may decode only the lower layer.

[0125] In addition, the decoding device may prioritize decoding from lower layers depending on the map's zoom level or application.

[0126] In addition, for applications such as self-position estimation or object recognition during autonomous driving of a vehicle or robot, the encoding device or the decoding device may perform encoding or decoding with reduced resolution in areas other than the area within a specific height from the road surface (the area where recognition is performed).

[0127] In addition, the encoding device may encode point clouds representing indoor and outdoor spatial shapes separately. For example, by separating the GOS representing the indoor area (Indoor GOS) and the GOS representing the outdoor area (Outdoor GOS), the decoding device can select the GOS to decode according to the viewpoint position when using the encoded data.

[0128] Additionally, the encoding device may encode an indoor GOS and an outdoor GOS with close coordinates so that they are adjacent within the encoding stream. For example, the encoding device corresponds the identifiers of both and stores information representing the corresponded identifiers within the encoding stream or in metadata stored separately. By doing so, the decoding device can identify an indoor GOS and an outdoor GOS with close coordinates by referring to the information in the metadata.

[0129] Additionally, the encoding device may change the size of the GOS or SPC in the indoor GOS and outdoor GOS. For example, the encoding device sets the size of the GOS to be smaller indoors compared to outdoors. Furthermore, the encoding device may change the precision when extracting feature points from a point cloud or the precision of object detection, etc., in the indoor GOS and outdoor GOS.

[0130] Additionally, the encoding device may add information to the encoded data to enable the decoding device to distinguish dynamic objects from static objects. By doing so, the decoding device can match dynamic objects with red borders or explanatory characters. Alternatively, the decoding device may display only red borders or explanatory characters instead of dynamic objects. Furthermore, the decoding device may display more detailed object types. For example, a red border may be used for cars, and a yellow border for people.

[0131] Additionally, the encoding device or the decoding device may determine whether to encode or decode dynamic objects and static objects as different SPCs or GOSs based on the frequency of occurrence of dynamic objects, or the ratio of static objects to dynamic objects, etc. For example, if the frequency or ratio of occurrence of dynamic objects exceeds a threshold, an SPC or GOS containing a mixture of dynamic and static objects is permitted, and if the frequency or ratio of occurrence of dynamic objects does not exceed a threshold, an SPC or GOS containing a mixture of dynamic and static objects is not permitted.

[0132] When detecting dynamic objects from two-dimensional image information of a camera rather than a point cloud, the encoding device may separately acquire information (such as borders or characters) for identifying the detection result and the object location, and encode this information as part of three-dimensional encoded data. In this case, the decoder displays auxiliary information (borders or characters) representing the dynamic object by superimposing it on the decoding result of the static object.

[0133] Additionally, the encoding device may change the density of VXL or VLM in SPC according to the complexity of the shape of the static object, etc. For example, the encoding device sets VXL or VLM to be denser as the shape of the static object becomes more complex. Furthermore, the encoding device may determine the quantization step when quantizing spatial position or color information according to the density of VXL or VLM. For example, the encoding device sets the quantization step to be smaller as VXL or VLM becomes denser.

[0134] As described above, the encoding device or decoding device according to the present embodiment performs encoding or decoding of space in space units having coordinate information.

[0135] In addition, the encoding device and the decoding device perform encoding or decoding on a volume basis within the space. The volume includes a voxel, which is the smallest unit to which position information corresponds.

[0136] Additionally, the encoding device and the decoding device perform encoding or decoding by mapping arbitrary elements using a table mapping each element of spatial information, including coordinates, objects, and time, to a GOP, or a table mapping the elements to each other. Additionally, the decoding device determines coordinates using the value of a selected element, specifies a volume, voxel, or space from the coordinates, and decodes the space containing the said volume or voxel, or the specified space.

[0137] In addition, the encoding device determines a volume, voxel, or space selectable by an element by feature point extraction or object recognition, and encodes it as a randomly accessible volume, voxel, or space.

[0138] Spaces are classified into three types: I-SPC, which can be encoded or decoded as a space group; P-SPC, which is encoded or decoded by referencing any one processed space; and B-SPC, which is encoded or decoded by referencing any two processed spaces.

[0139] One or more volumes correspond to static or dynamic objects. Spaces containing only static objects and spaces containing only dynamic objects are encoded or decoded as different GOS. That is, an SPC containing only static objects and an SPC containing only dynamic objects are assigned to different GOS.

[0140] Dynamic objects are encoded or decoded for each object and corresponded to one or more spaces containing only static objects. That is, multiple dynamic objects are encoded individually, and the encoded data of the multiple dynamic objects obtained is corresponded to an SPC containing only static objects.

[0141] The encoding device and the decoding device increase the priority of the I-SPC within the GOS and perform encoding or decoding. For example, the encoding device performs encoding to minimize I-SPC degradation (so that the original three-dimensional data is more faithfully reproduced after decoding). Also, the decoding device decodes, for example, only the I-SPC.

[0142] The encoding device may perform encoding by changing the frequency of using I-SPC depending on the density or number (quantity) of objects in the world. That is, the encoding device changes the frequency of selecting I-SPC according to the number or density of objects included in the three-dimensional data. For example, the encoding device increases the frequency of using I-space as the density of objects in the world increases.

[0143] In addition, the encoding device sets random access points in units of GOS and stores information indicating a spatial area corresponding to the GOS in header information.

[0144] The encoding device uses, for example, a default value as the space size of the GOS. Additionally, the encoding device may change the size of the GOS based on the number (quantity) or density of objects or dynamic objects. For example, the encoding device reduces the space size of the GOS as the objects or dynamic objects become denser or more numerous.

[0145] In addition, the space or volume includes a group of feature points derived using information obtained from sensors such as a depth sensor, a gyroscope, or a camera. The coordinates of the feature points are set at the center position of the voxel. In addition, high precision of position information can be realized through the subdivision of the voxel.

[0146] A feature point group is derived using multiple pictures. The multiple pictures have at least two types of time information, including actual time information and time information identical to that of the multiple pictures corresponding to the space (e.g., encoding time used for rate control, etc.).

[0147] In addition, encoding or decoding is performed in GOS units containing one or more spaces.

[0148] The encoding device and the decoding device refer to the space within the GOS that has finished processing and predict the P-space or B-space within the GOS of the processing target.

[0149] Alternatively, the encoding device and the decoding device perform prediction of the P-space or B-space within the GOS of the processing target using the finished space within the GOS of the processing target without referring to different GOSs.

[0150] In addition, the encoding device and the decoding device transmit or receive an encoded stream in world units including one or more GOS.

[0151] In addition, the GOS has a layer structure in at least one direction within the world, and the encoding device and the decoding device perform encoding or decoding from the lower layer. For example, a randomly accessible GOS belongs to the lowest layer. A GOS belonging to an upper layer refers only to GOS belonging to the same layer or lower. That is, the GOS includes a plurality of layers that are spatially divided in a predetermined direction and each includes one or more SPCs. The encoding device and the decoding device encode or decode each SPC by referring to an SPC included in the same layer as the said SPC or in a layer lower than the said SPC.

[0152] Additionally, the encoding device and the decoding device sequentially encode or decode GOS within a world unit containing a plurality of GOS. The encoding device and the decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. That is, the encoding data includes information indicating the encoding order of a plurality of GOS.

[0153] In addition, the encoding device and the decoding device encode or decode two or more different spaces or GOS in parallel.

[0154] In addition, the encoding device and the decoding device encode or decode spatial information (coordinates, size, etc.) of space or GOS.

[0155] In addition, the encoding device and the decoding device encode or decode a space or GOS included in a specific space specified based on external information regarding its location or / and area size, such as GPS, route information, or scale.

[0156] An encoding device or a decoding device encodes or decodes spaces far from its location with lower priority compared to spaces close by.

[0157] The encoding device sets one direction of the world according to the scale or purpose and encodes a GOS having a layer structure in that direction. In addition, the decoding device decodes the GOS having a layer structure in one direction of the world set according to the scale or purpose, starting from the lower layer.

[0158] The encoding device changes the extraction of feature points included in the space, the precision of object recognition, or the size of the spatial region in indoor and outdoor environments. However, the encoding device and the decoding device encode or decode indoor GOS and outdoor GOS with close coordinates adjacently within the world, and encode or decode by corresponding their identifiers.

[0159] For the above, a three-dimensional data encoding device and a three-dimensional data decoding device according to embodiments of the present disclosure have been described, but the present disclosure is not limited to these embodiments.

[0160] In addition, each processing unit included in the three-dimensional data encoding device or three-dimensional data decoding device according to the above embodiment is typically realized as an LSI, which is an integrated circuit. These may be individually integrated into a single chip, or they may be integrated into a single chip to include some or all of them.

[0161] In addition, integrated circuits are not limited to LSIs and may be realized using dedicated circuits or general-purpose processors. A Field Programmable Gate Array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections or settings of circuit cells within the LSI may also be used.

[0162] In addition, in each of the above embodiments, each component may be realized by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be realized by a program execution unit, such as a CPU or processor, reading and executing a software program recorded on a recording medium, such as a hard disk or semiconductor memory.

[0163] Additionally, the present disclosure may be realized as a three-dimensional data encoding method or a three-dimensional data decoding method executed by a three-dimensional data encoding device or a three-dimensional data decoding device.

[0164] Also, the division of function blocks in the block diagram is an example, and multiple function blocks may be realized as a single function block, a single function block may be divided into multiple blocks, or some functions may be moved to other function blocks. In addition, the functions of multiple function blocks having similar functions may be processed in parallel or by time-sharing by a single piece of hardware or software.

[0165] Furthermore, the order in which each step in the flowchart is executed is for illustrative purposes to specifically explain the present disclosure, and may be in a different order than that described above. Additionally, some of the steps may be executed simultaneously (in parallel) with other steps.

[0166] The three-dimensional data encoding device and the three-dimensional data decoding device according to one or more embodiments have been described above based on embodiments, but the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceived by those skilled in the art may be implemented in the embodiments, or forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments. Industrial applicability

[0167] The present disclosure can be applied to a three-dimensional data encoding device and a three-dimensional data decoding device. Explanation of the symbols

[0168] 100: Three-dimensional data encoding device 101, 201: Acquisition unit 102: Encoding region determination unit 103: Division unit 104: Encoding section 111: Three-dimensional data 112, 211: Encoded three-dimensional data 200: Three-dimensional data decoder 202: Decoding Initiation GOS Decision Unit 203: Decoding SPC Decision Unit 204: Decoding section 212: Decoding three-dimensional data

Claims

Claim 1 A three-dimensional data encoding method for encoding three-dimensional data, executed by a three-dimensional data encoding device, wherein the three-dimensional data is divided into a plurality of first processing units, each corresponding to a three-dimensional coordinate, each of the plurality of first processing units is divided into a plurality of second processing units, and each of the plurality of second processing units is encoded to generate encoded data, and in generating the encoded data, the second processing unit of the processing target is encoded by referring to another second processing unit included in the first processing unit that includes the second processing unit of the processing target. Claim 2 A three-dimensional data encoding method according to claim 1, wherein the encoded data comprises first information representing three-dimensional coordinates corresponding to each of the plurality of first processing units. Claim 3 A three-dimensional data encoding method according to claim 1, wherein the encoded data comprises first information representing at least one of an object, a time, and a data storage location corresponding to the first processing unit. Claim 4 A three-dimensional data encoding method according to claim 1, wherein, in generating the encoded data, one of a first type that does not refer to another second processing unit, a second type that refers to another second processing unit, and a third type that refers to two other second processing units is selected as the type of the second processing unit of the processing target, and the second processing unit of the processing target is encoded according to the selected type. Claim 5 A three-dimensional data encoding method according to claim 4, wherein the frequency of selecting the first type is changed according to the number or density of objects included in the three-dimensional data. Claim 6 A three-dimensional data encoding method according to claim 4, wherein the size of the first processing unit is determined according to the number or density of objects or dynamic objects included in the three-dimensional data. Claim 7 A three-dimensional data encoding method according to claim 1, further comprising determining whether to refer to the other second processing unit. Claim 8 A three-dimensional data encoding method according to claim 1, wherein the first processing unit comprises a plurality of layers that are spatially partitioned and each comprises one or more of the second processing units, and wherein, in generating the encoded data, the second processing unit is encoded by referring to the second processing unit included in the same layer as the second processing unit or in a layer lower than the second processing unit. Claim 9 A three-dimensional data encoding method according to claim 1, wherein a second processing unit comprising only static objects and a second processing unit comprising only dynamic objects are assigned to different first processing units. Claim 10 A three-dimensional data encoding method according to claim 1, wherein in generating the encoded data, a plurality of dynamic objects are encoded individually, and the encoded data of the plurality of dynamic objects corresponds to a second processing unit comprising only static objects. Claim 11 A three-dimensional data encoding method according to claim 1, wherein the second processing unit is divided into a plurality of third processing units, and each of the plurality of third processing units is encoded in the generation of the encoded data. Claim 12 A three-dimensional data encoding method according to claim 11, wherein the third processing unit comprises one or more voxels, which are the minimum units to which position information corresponds. Claim 13 A three-dimensional data encoding method according to any one of claims 1 to 12, wherein the second processing unit comprises a group of feature points derived from information obtained from a sensor. Claim 14 A three-dimensional data encoding method according to any one of claims 1 to 12, wherein the encoded data includes information indicating the encoding order of the plurality of first processing units. Claim 15 A three-dimensional data encoding method according to any one of claims 1 to 12, wherein the encoded data includes information indicating the size of the plurality of first processing units. Claim 16 A three-dimensional data encoding method according to any one of claims 1 to 12, wherein the plurality of first processing units are encoded in parallel. Claim 17 A three-dimensional data decoding method for decoding three-dimensional data executed by a three-dimensional data decoding device, wherein a first processing unit, each corresponding to a three-dimensional coordinate, decodes the three-dimensional data of the first processing unit by decoding each of the encoded data of a plurality of second processing units divided therefrom, and in decoding the three-dimensional data, the second processing unit of the processing target is decoded by referring to another second processing unit included in the first processing unit that includes the second processing unit of the processing target. Claim 18 In claim 17, a three-dimensional data decoding method that also determines whether to refer to the other second processing unit. Claim 19 A three-dimensional data encoding device for encoding three-dimensional data, comprising: a dividing unit that divides the three-dimensional data into a plurality of first processing units, each corresponding to a three-dimensional coordinate, and divides each of the plurality of first processing units into a plurality of second processing units; and an encoding unit that generates encoded data by encoding each of the plurality of second processing units, wherein the encoding unit encodes a second processing unit of a processing target by referring to another second processing unit included in the first processing unit that includes the second processing unit of the processing target. Claim 20 A three-dimensional data decoding device for decoding three-dimensional data, comprising a decoding unit that decodes the three-dimensional data of a first processing unit by decoding each of a plurality of second processing units, each of which is divided into a first processing unit corresponding to a three-dimensional coordinate, and wherein the decoding unit decodes the second processing unit of the processing target by referring to another second processing unit included in the first processing unit including the second processing unit of the processing target.

Images