Three dimensional data encoding method, three dimensional data decoding method, three dimensional data encoding device, and three dimensional data decoding device

Abstract

To provide a three dimensional data encoding method capable of improving encoding efficiency.SOLUTION: A three dimensional data encoding method for encoding three dimensional data 111 includes dividing three dimensional data 111 into a plurality of first processing units each associated with three dimensional coordinates, and encoding each of the plurality of first processing units to generate encoded data, the encoded data including information indicating a profile defined in accordance with an application of the three dimensional data.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space (see, for example, Non-Patent Document 1). A point cloud stores the position and color of the point cloud. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds involve an extremely large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, compression of point clouds is partially supported by public libraries that perform point cloud-related processing (Point Cloud Library). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] “Octree-Based Progressive Geometry Coding of Point Clouds”, Eurographics Symposium on Point-Based Graphics (2006) Summary of the Invention [Problem to be solved by the invention]

[0006] However, when using encoded data such as point clouds in actual devices or services, random access to desired spatial positions or objects is essential, but until now, random access to three-dimensional encoded data has not existed as a function, and no encoding method for this purpose has existed.

[0007] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can provide a random access function in encoded three-dimensional data. [Means for solving the problem]

[0008] Note that the present disclosure enables spatial quantization, prediction, and the like by using a configuration in which space is divided and coded, and is effective even when random access is not necessarily performed.

[0009] A three-dimensional data encoding method according to one embodiment of the present disclosure is a three-dimensional data encoding method for encoding three-dimensional data, which divides the three-dimensional data into a plurality of first processing units, each of which corresponds to a three-dimensional coordinate, and generates encoded data by encoding each of the plurality of first processing units, and the encoded data includes information indicating a profile defined according to the use of the three-dimensional data.

[0010] A three-dimensional data decoding method according to one embodiment of the present disclosure is a three-dimensional data decoding method for decoding three-dimensional data, which obtains each piece of encoded data in which a plurality of first processing units, each of which corresponds to a three-dimensional coordinate, are encoded, obtains information indicating a profile defined according to the use of the three-dimensional data, and decodes each piece of encoded data using the information.

[0011] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0012] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can provide a random access function in encoded three-dimensional data. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing the structure of encoded three-dimensional data according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a prediction structure between SPCs belonging to the lowest layer of a GOS according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of an inter-layer prediction structure according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the coding order of the GOS according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the coding order of the GOS according to the embodiment. [Figure 6] FIG. 6 is a block diagram of a three-dimensional data encoding device according to an embodiment. [Figure 7] FIG. 7 is a flowchart of the encoding process according to the embodiment. [Figure 8]FIG. 8 is a block diagram of a three-dimensional data decoding device according to an embodiment. [Figure 9] FIG. 9 is a flowchart of the decoding process according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of meta information according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] A three-dimensional data encoding method according to one embodiment of the present disclosure is a three-dimensional data encoding method for encoding three-dimensional data, and includes a division step of dividing the three-dimensional data into first processing units, which are random access units and each of which corresponds to a three-dimensional coordinate, and an encoding step of generating encoded data by encoding each of a plurality of the first processing units.

[0015] This enables random access for each first processing unit. In this way, the three-dimensional data encoding method can provide a random access function for encoded three-dimensional data.

[0016] For example, the three-dimensional data encoding method may include a generation step of generating first information indicating the plurality of first processing units and three-dimensional coordinates corresponding to each of the plurality of first processing units, and the encoded data may include the first information.

[0017] For example, the first information may further indicate at least one of an object, a time, and a data storage destination associated with each of the plurality of first processing units.

[0018] For example, the dividing step may further divide the first processing unit into a plurality of second processing units, and the encoding step may encode each of the plurality of second processing units.

[0019] For example, in the encoding step, a second processing unit to be processed that is included in a first processing unit to be processed may be encoded by referring to another second processing unit that is included in the first processing unit to be processed.

[0020] According to this, by referring to other second processing units, it is possible to improve coding efficiency.

[0021] For example, in the encoding step, the type of the second processing unit to be processed may be selected from among a first type that does not reference other second processing units, a second type that references one other second processing unit, and a third type that references two other second processing units, and the second processing unit to be processed may be encoded according to the selected type.

[0022] For example, in the encoding step, the frequency of selecting the first type may be changed depending on the number or density of objects included in the three-dimensional data.

[0023] This allows the trade-off between random accessibility and coding efficiency to be appropriately set.

[0024] For example, in the encoding step, the size of the first processing unit may be determined according to the number or density of objects or dynamic objects included in the three-dimensional data.

[0025] This allows the trade-off between random accessibility and coding efficiency to be appropriately set.

[0026] For example, the first processing unit may be spatially divided in a predetermined direction and include multiple layers, each including one or more of the second processing units, and in the encoding step, the second processing unit may be encoded by referring to a second processing unit included in the same layer as the second processing unit or in a layer lower than the second processing unit.

[0027] This makes it possible to improve the random accessibility of important layers in the system and to suppress a decrease in coding efficiency, for example.

[0028] For example, in the dividing step, a second processing unit including only static objects and a second processing unit including only dynamic objects may be assigned to different first processing units.

[0029] This allows easy control of dynamic and static objects.

[0030] For example, in the encoding step, a plurality of dynamic objects may be individually encoded, and the encoded data of the plurality of dynamic objects may be associated with a second processing unit including only static objects.

[0031] This allows easy control of dynamic and static objects.

[0032] For example, the dividing step may further divide the second processing unit into a plurality of third processing units, and the encoding step may encode each of the plurality of third processing units.

[0033] For example, the third processing unit may include one or more voxels, which are the smallest units to which position information can be associated.

[0034] For example, the second processing unit may include a group of feature points derived from information obtained by a sensor.

[0035] For example, the coded data may include information indicating the coding order of the first processing units.

[0036] For example, the encoded data may include information indicating the sizes of the plurality of first processing units.

[0037] For example, in the encoding step, the plurality of first processing units may be encoded in parallel.

[0038] Furthermore, a three-dimensional data decoding method according to one embodiment of the present disclosure is a three-dimensional data decoding method for decoding three-dimensional data, and includes a decoding step of generating three-dimensional data of a first processing unit by decoding each piece of encoded data of the first processing unit, which is a random access unit and each piece of encoded data corresponds to a three-dimensional coordinate.

[0039] This enables random access for each first processing unit. In this way, the three-dimensional data decoding method can provide a random access function for encoded three-dimensional data.

[0040] Furthermore, a three-dimensional data encoding device according to one embodiment of the present disclosure may be a three-dimensional data encoding device that encodes three-dimensional data, and may include a dividing unit that divides the three-dimensional data into first processing units that are random access units and each of which corresponds to a three-dimensional coordinate, and an encoding unit that generates encoded data by encoding each of a plurality of the first processing units.

[0041] This allows random access for each first processing unit. In this way, the three-dimensional data encoding device can provide a random access function for encoded three-dimensional data.

[0042] Furthermore, a three-dimensional data decoding device according to one embodiment of the present disclosure may be a three-dimensional data decoding device that decodes three-dimensional data, and may include a decoding unit that generates three-dimensional data of a first processing unit by decoding each piece of encoded data of the first processing unit, which is a random access unit and each piece of encoded data corresponds to a three-dimensional coordinate.

[0043] This enables random access for each first processing unit. In this way, the three-dimensional data decoding device can provide a random access function for encoded three-dimensional data.

[0044] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0045] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a superordinate concept will be described as optional components.

[0046] First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to this embodiment will be described. Fig. 1 is a diagram showing the structure of encoded three-dimensional data according to this embodiment.

[0047] In this embodiment, a three-dimensional space is divided into spaces (SPCs) corresponding to pictures in video encoding, and three-dimensional data is encoded using the spaces as units. The spaces are further divided into volumes (VLMs) corresponding to macroblocks or the like in video encoding, and prediction and conversion are performed using the VLMs as units. A volume includes a plurality of voxels (VXLs), which are the smallest units to which position coordinates can be associated. Note that prediction, like prediction performed for two-dimensional images, refers to generating predicted three-dimensional data similar to the processing unit to be processed by referring to other processing units, and encoding the difference between the predicted three-dimensional data and the processing unit to be processed. Furthermore, this prediction includes not only spatial prediction that refers to other prediction units at the same time, but also temporal prediction that refers to a prediction unit at a different time.

[0048] For example, when a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes a three-dimensional space represented by point cloud data such as a point cloud, it encodes each point of the point cloud or multiple points contained in a voxel collectively according to the size of the voxel. By subdividing the voxels, the three-dimensional shape of the point cloud can be expressed with high precision, and by increasing the voxel size, the three-dimensional shape of the point cloud can be expressed roughly.

[0049] In the following, an example will be described in which the three-dimensional data is a point cloud, but the three-dimensional data is not limited to a point cloud and may be three-dimensional data in any format.

[0050] Alternatively, voxels with a hierarchical structure may be used. In this case, the nth layer may indicate in order whether a sample point exists in the n-1th layer or lower (a layer below the nth layer). For example, when decoding only the nth layer, if a sample point exists in the n-1th layer or lower, the sample point can be decoded by assuming that the sample point exists at the center of the voxel in the nth layer.

[0051] The encoding device also acquires point cloud data using a distance sensor, a stereo camera, a monocular camera, a gyro, an inertial sensor, or the like.

[0052] Similar to video coding, spaces are classified into at least three prediction structures, including independently decodable intra-space (I-SPC), predictive space (P-SPC), and bidirectional space (B-SPC). Spaces also have two types of time information: decoding time and display time.

[0053] As shown in Figure 1, there is a random access unit called a Group Of Space (GOS), which is a processing unit that includes multiple spaces. There is also a World (WLD), which is a processing unit that includes multiple GOS.

[0054] The spatial region occupied by the world is associated with an absolute position on Earth using GPS or latitude and longitude information. This position information is stored as meta information. Note that the meta information may be included in the encoded data or may be transmitted separately from the encoded data.

[0055] Furthermore, within a GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0056] In the following, the process of encoding, decoding, referencing, etc. of three-dimensional data included in a processing unit such as a GOS, SPC, or VLM will also be simply referred to as encoding, decoding, or referencing the processing unit, etc. The three-dimensional data included in the processing unit includes, for example, at least one pair of a spatial position such as three-dimensional coordinates and a characteristic value such as color information.

[0057] Next, the prediction structure of SPCs in a GOS will be explained. Multiple SPCs in the same GOS or multiple VLMs in the same SPC occupy different spaces, but have the same time information (decoding time and display time).

[0058] Furthermore, the first SPC in a GOS in decoding order is the I-SPC. There are two types of GOS: closed GOS and open GOS. A closed GOS is a GOS that can decode all SPCs in the GOS when decoding starts from the first I-SPC. In an open GOS, some SPCs that appear earlier in the GOS than the first I-SPC refer to a different GOS, and cannot be decoded using only that GOS.

[0059] In addition, in coded data such as map information, WLDs are sometimes decoded in the reverse order of coding, and if there is dependency between GOSs, reverse playback is difficult. Therefore, in such cases, closed GOSs are generally used.

[0060] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed in order from the SPC in the lower layer.

[0061] Fig. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of a GOS, and Fig. 3 is a diagram showing an example of a prediction structure between layers.

[0062] A GOS contains one or more I-SPCs. Objects such as people, animals, cars, bicycles, traffic lights, and landmark buildings exist in three-dimensional space, and it is particularly effective to encode small objects as I-SPCs. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPCs in the GOS when decoding the GOS with low processing load or at high speed.

[0063] The encoding device may also switch the encoding interval or frequency of occurrence of I-SPC depending on the density of objects in the WLD.

[0064] 3, the encoding device or decoding device encodes or decodes multiple layers in order from the lowest layer (layer 1). This allows, for example, an autonomous vehicle to prioritize data near the ground, which contains more information.

[0065] In addition, encoded data used by drones, etc. may be encoded or decoded in order from the SPC of the highest layer in the height direction within the GOS.

[0066] Alternatively, the encoding or decoding device may encode or decode multiple layers so that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding or decoding device may encode or decode layers 3, 8, 1, 9, etc. in that order.

[0067] Next, we will explain how to handle static and dynamic objects.

[0068] In a three-dimensional space, there exist static objects or scenes such as buildings or roads (hereinafter collectively referred to as static objects), and dynamic objects such as cars or people (hereinafter referred to as dynamic objects). Object detection is performed separately by extracting feature points from point cloud data or camera images such as a stereo camera. Here, an example of a method for encoding dynamic objects will be described.

[0069] The first method is to encode static objects without distinguishing between static and dynamic objects, and the second method is to distinguish between static and dynamic objects using identification information.

[0070] For example, the GOS is used as the identification unit. In this case, a GOS containing only SPCs that constitute static objects is distinguished from a GOS containing SPCs that constitute dynamic objects by identification information stored within the coded data or separately from the coded data.

[0071] Alternatively, the SPC may be used as the identification unit. In this case, the above-mentioned identification information distinguishes between an SPC that includes only a VLM that constitutes a static object and an SPC that includes a VLM that constitutes a dynamic object.

[0072] Alternatively, the VLM or VXL may be used as the identification unit, in which case the VLM or VXL including only static objects is distinguished from the VLM or VXL including dynamic objects by the above-mentioned identification information.

[0073] The encoding device may also encode dynamic objects as one or more VLMs or SPCs, and encode a VLM or SPC containing only static objects and an SPC containing dynamic objects as different GOSs. If the size of the GOS varies depending on the size of the dynamic objects, the encoding device stores the size of the GOS separately as meta information.

[0074] The encoding device may also encode static objects and dynamic objects independently of each other, and overlay the dynamic objects on a world made up of static objects. In this case, the dynamic object is made up of one or more SPCs, and each SPC corresponds to one or more SPCs that make up the static object on which the SPC is overlaid. Note that the dynamic object may be represented by one or more VLMs or VXLs instead of SPCs.

[0075] The encoding device may also encode static objects and dynamic objects as different streams.

[0076] The encoding device may also generate a GOS that includes one or more SPCs that make up a dynamic object. Furthermore, the encoding device may set the GOS (GOS_M) that includes the dynamic object and the GOS of the static object that corresponds to the spatial region of GOS_M to the same size (occupy the same spatial region). This allows superimposition processing to be performed on a GOS-by-GOS basis.

[0077] A P-SPC or B-SPC that configures a dynamic object may refer to an SPC included in a different GOS that has already been coded. In cases where the position of a dynamic object changes over time and the same dynamic object is coded as a GOS at different times, referencing across GOSs is effective from the viewpoint of compression ratio.

[0078] The encoding device may switch between the first and second methods depending on the intended use of the encoded data. For example, when the encoded three-dimensional data is used as a map, it is desirable to be able to separate dynamic objects, so the encoding device uses the second method. On the other hand, when encoding three-dimensional data of an event such as a concert or sporting event, the encoding device uses the first method if there is no need to separate dynamic objects.

[0079] The decode time and display time of a GOS or SPC can be stored in the coded data or as meta information. The time information of all static objects may be the same. In this case, the actual decode time and display time may be determined by the decoding device. Alternatively, a different value may be assigned as the decode time for each GOS or SPC, and the same value may be assigned as the display time for all. Furthermore, a model may be introduced that ensures that the decoder has a buffer of a predetermined size and can decode without failure if it reads a bitstream at a predetermined bit rate according to the decode time, as in a decoder model used in video coding, such as the HEVC HRD (Hypothetical Reference Decoder).

[0080] Next, we will explain the arrangement of GOS within a world. The coordinates of the three-dimensional space in a world are expressed by three mutually orthogonal coordinate axes (x-axis, y-axis, and z-axis). By establishing a predetermined rule for the encoding order of GOS, encoding can be performed so that spatially adjacent GOS are continuous within the encoded data. For example, in the example shown in Figure 4, GOS within the xz plane are encoded continuously. After encoding of all GOS within a certain xz plane is completed, the value of the y-axis is updated. In other words, as encoding progresses, the world extends in the y-axis direction. Furthermore, the index numbers of GOS are set in the encoding order.

[0081] Here, the three-dimensional space of the world is associated one-to-one with absolute geographical coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space may be expressed by relative positions from a preset reference position. The directions of the x-, y-, and z-axes of the three-dimensional space are expressed as direction vectors determined based on the latitude and longitude, and the direction vectors are stored as meta information together with the encoded data.

[0082] The size of the GOS is fixed, and the encoding device stores the size as meta information. The size of the GOS may be changed depending on, for example, whether the location is an urban area or whether the location is indoors or outdoors. That is, the size of the GOS may be changed depending on the quantity or nature of objects that have information value. Alternatively, the encoding device may adaptively change the size of the GOS or the spacing between I-SPCs within the GOS depending on, for example, the density of objects within the same world. For example, the higher the object density, the smaller the GOS size and the shorter the spacing between I-SPCs within the GOS.

[0083] In the example shown in Figure 5, the third to tenth GOS regions have a high density of objects, so the GOS are subdivided to allow finer granularity for random access. Note that the seventh to tenth GOS regions are located behind the third to sixth GOS regions, respectively.

[0084] Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be described. Fig. 6 is a block diagram of the three-dimensional data encoding device 100 according to this embodiment. Fig. 7 is a flowchart showing an example of the operation of the three-dimensional data encoding device 100.

[0085] 6 generates encoded three-dimensional data 112 by encoding three-dimensional data 111. This three-dimensional data encoding device 100 includes an acquisition unit 101, an encoding region determination unit 102, a division unit 103, and an encoding unit 104.

[0086] As shown in FIG. 7, first, the acquisition unit 101 acquires three-dimensional data 111, which is point cloud data (S101).

[0087] Next, the coding area determination unit 102 determines an area to be coded from the spatial area corresponding to the acquired point cloud data (S102). For example, depending on the position of the user or vehicle, the coding area determination unit 102 determines a spatial area around the position as the area to be coded.

[0088] Next, the dividing unit 103 divides the point cloud data included in the region to be coded into processing units. Here, the processing units are the above-mentioned GOS and SPC, etc. Furthermore, this region to be coded corresponds to, for example, the above-mentioned world. Specifically, the dividing unit 103 divides the point cloud data into processing units based on the size of a preset GOS or the presence or size of a dynamic object (S103). Furthermore, the dividing unit 103 determines the start position of the SPC that is the first in coding order in each GOS.

[0089] Next, the encoding unit 104 generates encoded three-dimensional data 112 by sequentially encoding the plurality of SPCs in each GOS (S104).

[0090] Although an example has been shown in which the area to be coded is divided into GOSs and SPCs and then each GOS is coded, the processing procedure is not limited to the above. For example, a procedure may be used in which the configuration of one GOS is determined, the GOS is coded, and then the configuration of the next GOS is determined.

[0091] In this way, the three-dimensional data encoding device 100 generates encoded three-dimensional data 112 by encoding three-dimensional data 111. Specifically, the three-dimensional data encoding device 100 divides the three-dimensional data into first processing units (GOS), which are random access units and each correspond to a three-dimensional coordinate, divides the first processing units (GOS) into a plurality of second processing units (SPC), and divides the second processing units (SPC) into a plurality of third processing units (VLM). Furthermore, the third processing units (VLM) include one or more voxels (VXL), which are the smallest units to which position information can be associated.

[0092] Next, the three-dimensional data encoding device 100 generates encoded three-dimensional data 112 by encoding each of the plurality of first processing units (GOS). Specifically, the three-dimensional data encoding device 100 encodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the three-dimensional data encoding device 100 encodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0093] For example, when the first processing unit (GOS) to be processed is a closed GOS, the three-dimensional data encoding device 100 encodes the second processing unit (SPC) to be processed included in the first processing unit (GOS) to be processed by referring to other second processing units (SPC) included in the first processing unit (GOS) to be processed. In other words, the three-dimensional data encoding device 100 does not refer to second processing units (SPC) included in first processing units (GOS) different from the first processing unit (GOS) to be processed.

[0094] On the other hand, if the first processing unit (GOS) to be processed is an open GOS, the second processing unit (SPC) to be processed included in the first processing unit (GOS) to be processed is encoded by referring to other second processing units (SPC) included in the first processing unit (GOS) to be processed, or second processing units (SPC) included in a first processing unit (GOS) different from the first processing unit (GOS) to be processed.

[0095] In addition, the three-dimensional data encoding device 100 selects, as the type of the second processing unit (SPC) to be processed, one of the following: a first type (I-SPC) that does not reference other second processing units (SPCs), a second type (P-SPC) that references one other second processing unit (SPC), or a third type that references two other second processing units (SPCs), and encodes the second processing unit (SPC) to be processed according to the selected type.

[0096] Next, the configuration and operation flow of the three-dimensional data decoding device according to this embodiment will be described. Fig. 8 is a block diagram of the blocks of the three-dimensional data decoding device 200 according to this embodiment. Fig. 9 is a flowchart showing an example of the operation of the three-dimensional data decoding device 200.

[0097] 8 generates decoded three-dimensional data 212 by decoding encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. This three-dimensional data decoding device 200 includes an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0098] First, the acquisition unit 201 acquires the encoded 3D data 211 (S201). Next, the decoding start GOS determination unit 202 determines a GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to meta information stored in the encoded 3D data 211 or separately from the encoded 3D data, and determines a GOS including an SPC corresponding to a spatial position, object, or time at which decoding starts as the GOS to be decoded.

[0099] Next, the decoding SPC determination unit 203 determines the type (I, P, B) of SPC to be decoded in the GOS (S203). For example, the decoding SPC determination unit 203 determines whether to (1) decode only I-SPC, (2) decode I-SPC and P-SPC, or (3) decode all types. Note that if the type of SPC to be decoded has been determined in advance, such as when all SPCs are to be decoded, this step may not be performed.

[0100] Next, the decoding unit 204 acquires the address position in the encoded 3D data 211 where the first SPC in the GOS in decoding order (the same as the encoding order) starts, acquires the encoded data of the first SPC from the address position, and sequentially decodes each SPC in order starting from the first SPC (S204). Note that the address position is stored in meta information or the like.

[0101] In this way, the three-dimensional data decoding device 200 decodes the decoded three-dimensional data 212. Specifically, the three-dimensional data decoding device 200 generates the decoded three-dimensional data 212 of the first processing units (GOS) by decoding each of the encoded three-dimensional data 211 of the first processing units (GOS), which are random access units and each of which is associated with a three-dimensional coordinate. More specifically, the three-dimensional data decoding device 200 decodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the three-dimensional data decoding device 200 decodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0102] The following describes the meta information for random access. This meta information is generated by the three-dimensional data encoding device 100 and is included in the encoded three-dimensional data 112 (211).

[0103] In conventional random access for 2D video, decoding starts from the first frame of the random access unit that is close to the specified time. On the other hand, in the world, random access to space (coordinates, objects, etc.) is assumed in addition to time.

[0104] Therefore, to realize random access to at least three elements, coordinates, objects, and time, a table is prepared that associates each element with a GOS index number. Furthermore, the GOS index number is associated with the address of the I-SPC at the beginning of the GOS. Figure 10 shows an example of a table included in the meta information. Note that it is not necessary to use all of the tables shown in Figure 10; it is sufficient to use at least one table.

[0105] Hereinafter, as an example, random access starting from a coordinate will be described. When accessing coordinates (x2, y2, z2), first, the coordinate-GOS table is referenced and it is found that the point with coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is referenced and it is found that the address of the first I-SPC in the second GOS is addr(2). Therefore, the decoding unit 204 obtains data from this address and starts decoding.

[0106] The address may be an address in a logical format or a physical address on a hard disk drive or in memory. Information identifying a file segment may be used instead of the address. For example, a file segment is a unit obtained by segmenting one or more GOSs.

[0107] Furthermore, if an object spans multiple GOSs, the object-GOS table may indicate multiple GOSs to which the object belongs. If the multiple GOSs are closed GOSs, the encoding device and decoding device can encode or decode in parallel. On the other hand, if the multiple GOSs are open GOSs, the multiple GOSs can reference each other, thereby improving compression efficiency.

[0108] Examples of objects include people, animals, cars, bicycles, traffic lights, landmark buildings, etc. For example, when encoding a world, the three-dimensional data encoding device 100 can extract feature points specific to objects from a three-dimensional point cloud or the like, detect objects based on the feature points, and set the detected objects as random access points.

[0109] In this way, the three-dimensional data encoding device 100 generates first information indicating a plurality of first processing units (GOS) and three-dimensional coordinates associated with each of the plurality of first processing units (GOS). The encoded three-dimensional data 112 (211) includes this first information. The first information further indicates at least one of an object, a time, and a data storage destination associated with each of the plurality of first processing units (GOS).

[0110] The three-dimensional data decoding device 200 acquires first information from the encoded three-dimensional data 211, and uses the first information to identify the encoded three-dimensional data 211 of the first processing unit corresponding to the specified three-dimensional coordinates, object, or time, and decodes the encoded three-dimensional data 211.

[0111] Other examples of meta information will be described below. In addition to the meta information for random access, the three-dimensional data encoding device 100 may generate and store the following meta information. Furthermore, the three-dimensional data decoding device 200 may use this meta information during decoding.

[0112] When using three-dimensional data as map information, a profile may be defined depending on the application, and information indicating the profile may be included in the meta information. For example, profiles for urban areas, suburban areas, or flying objects may be defined, and the maximum or minimum size of the world, SPC, or VLM may be defined for each. For example, for urban areas, more detailed information is required than for suburban areas, so the minimum size of the VLM may be set smaller.

[0113] The meta information may include a tag value indicating the type of object. This tag value is associated with the VLM, SPC, or GOS that constitutes the object. For example, a tag value may be set for each type of object, such as a tag value of "0" indicating a "person," a tag value of "1" indicating a "car," and a tag value of "2" indicating a "traffic light." Alternatively, if the type of object is difficult to determine or does not need to be determined, a tag value indicating a property such as size or whether the object is dynamic or static may be used.

[0114] The meta information may also include information indicating the range of the spatial region occupied by the world.

[0115] The meta information may also store the size of the SPC or VXL as header information common to a plurality of SPCs, such as the entire stream of coded data or an SPC in a GOS.

[0116] The meta information may also include identification information for the range sensor or camera used to generate the point cloud, or information indicating the positional accuracy of the points in the point cloud.

[0117] The meta information may also include information indicating whether the world is made up of only static objects or whether it also includes dynamic objects.

[0118] A modification of this embodiment will now be described.

[0119] The encoding device or decoding device may encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel can be determined based on meta-information indicating the spatial positions of the GOSs.

[0120] In cases where three-dimensional data is used as a spatial map for vehicles or flying objects moving around, or where such a spatial map is to be generated, the encoding device or decoding device may encode or decode a GOS or SPC contained in a space identified based on GPS, route information, zoom magnification, etc.

[0121] Furthermore, the decoding device may perform decoding in order from the space closest to the current location or the travel route. The encoding device or decoding device may encode or decode a space farther from the current location or the travel route by lowering the priority compared to a closer space. Here, lowering the priority means lowering the processing order, lowering the resolution (thinning out the data before processing), or lowering the image quality (increasing the encoding efficiency, for example, by increasing the quantization step), etc.

[0122] Furthermore, when decoding coded data that has been coded hierarchically in space, the decoding device may decode only the lower layers.

[0123] The decoding device may also decode data preferentially from the lowest layer depending on the zoom factor or purpose of the map.

[0124] In addition, for applications such as self-position estimation or object recognition performed when a car or robot is driving autonomously, the encoding device or decoding device may encode or decode with reduced resolution except for areas within a specific height from the road surface (area where recognition is performed).

[0125] The encoding device may also encode point clouds representing indoor and outdoor spatial shapes separately. For example, by separating the GOS representing the indoor space (indoor GOS) from the GOS representing the outdoor space (outdoor GOS), the decoding device can select the GOS to decode depending on the viewpoint position when using the encoded data.

[0126] The encoding device may also encode indoor and outdoor GOS with nearby coordinates so that they are adjacent in the encoded stream. For example, the encoding device may associate identifiers for the two and store information indicating the associated identifiers in the encoded stream or in separately stored meta information. This allows the decoding device to identify indoor and outdoor GOS with nearby coordinates by referring to the information in the meta information.

[0127] The encoding device may also switch the size of the GOS or SPC between indoor and outdoor GOS. For example, the encoding device may set a smaller GOS size indoors than outdoors. The encoding device may also change the accuracy of extracting feature points from the point cloud or the accuracy of object detection between indoor and outdoor GOS.

[0128] The encoding device may also add information to the encoded data that enables the decoding device to distinguish dynamic objects from static objects. This allows the decoding device to display dynamic objects together with red frames or explanatory text. The decoding device may also display only red frames or explanatory text instead of dynamic objects. The decoding device may also display more specific object types. For example, a red frame may be used for cars and a yellow frame for people.

[0129] Furthermore, the encoding device or decoding device may determine whether to encode or decode dynamic objects and static objects as different SPCs or GOSs depending on the frequency of appearance of dynamic objects, the ratio of static objects to dynamic objects, etc. For example, if the frequency of appearance or ratio of dynamic objects exceeds a threshold, an SPC or GOS in which dynamic objects and static objects are mixed is permitted, and if the frequency of appearance or ratio of dynamic objects does not exceed the threshold, an SPC or GOS in which dynamic objects and static objects are mixed is not permitted.

[0130] When detecting dynamic objects from two-dimensional camera image information rather than from a point cloud, the encoding device may separately acquire information for identifying the detection result (such as a frame or text) and the object position, and encode this information as part of the three-dimensional encoded data. In this case, the decoding device displays auxiliary information (such as a frame or text) indicating the dynamic object by superimposing it on the decoding result of the static object.

[0131] The encoding device may also change the density of the VXL or VLM in the SPC depending on factors such as the complexity of the shape of the static object. For example, the encoding device may set the VXL or VLM to a higher density the more complex the shape of the static object. Furthermore, the encoding device may determine the quantization step, etc., used when quantizing spatial position or color information depending on the density of the VXL or VLM. For example, the encoding device may set a smaller quantization step the denser the VXL or VLM.

[0132] As described above, the encoding device or decoding device according to this embodiment encodes or decodes space in units of spaces each having coordinate information.

[0133] Furthermore, the encoding device and the decoding device perform encoding and decoding in units of volumes within a space. A volume includes voxels, which are the smallest units to which position information can be associated.

[0134] The encoding device and decoding device perform encoding or decoding by associating any elements using a table that associates each element of spatial information, including coordinates, objects, and time, with a GOP, or a table that associates each element with another element. The decoding device determines coordinates using the value of a selected element, identifies a volume, voxel, or space from the coordinates, and decodes the space including the volume or voxel, or the identified space.

[0135] The encoding device also determines a volume, voxel, or space that can be selected by an element through feature point extraction or object recognition, and encodes it as a randomly accessible volume, voxel, or space.

[0136] Spaces are classified into three types: I-SPC, which can be encoded or decoded by itself; P-SPC, which is encoded or decoded by referring to any one processed space; and B-SPC, which is encoded or decoded by referring to any two processed spaces.

[0137] One or more volumes correspond to static objects or dynamic objects. A space containing only static objects and a space containing only dynamic objects are coded or decoded as different GOSs. That is, an SPC containing only static objects and an SPC containing only dynamic objects are assigned to different GOSs.

[0138] Dynamic objects are coded or decoded on an object-by-object basis and associated with one or more spaces containing only static objects, i.e., multiple dynamic objects are coded individually, and the resulting coded data for the multiple dynamic objects is associated with an SPC containing only static objects.

[0139] The encoding device and the decoding device perform encoding or decoding by increasing the priority of the I-SPC in the GOS. For example, the encoding device performs encoding so as to minimize degradation of the I-SPC (so that the original 3D data is reproduced more faithfully after decoding). Also, the decoding device decodes only the I-SPC, for example.

[0140] The encoding device may perform encoding by changing the frequency of using I-SPCs depending on the density or number (quantity) of objects in the world. In other words, the encoding device changes the frequency of selecting I-SPCs depending on the number or density of objects included in the three-dimensional data. For example, the encoding device may use I-spaces more frequently as the density of objects in the world increases.

[0141] Furthermore, the encoding device sets random access points in units of GOS, and stores information indicating the spatial region corresponding to the GOS in the header information.

[0142] The encoding device uses, for example, a default value as the spatial size of the GOS. Note that the encoding device may change the size of the GOS depending on the number (quantity) or density of objects or dynamic objects. For example, the encoding device reduces the spatial size of the GOS as the density or number of objects or dynamic objects increases.

[0143] The space or volume also includes a set of feature points derived using information obtained by sensors such as a depth sensor, a gyroscope, or a camera. The coordinates of the feature points are set at the center positions of the voxels. Furthermore, by subdividing the voxels, it is possible to achieve high accuracy of the position information.

[0144] The feature point group is derived using multiple pictures, each of which has at least two types of time information: actual time information and time information that is the same for multiple pictures associated with the space (e.g., encoding time used for rate control, etc.).

[0145] Also, encoding or decoding is performed in units of GOS, each GOS including one or more spaces.

[0146] The encoding device and the decoding device refer to the spaces in the processed GOS to predict the P space or the B space in the GOS to be processed.

[0147] Alternatively, the encoding device and the decoding device do not refer to a different GOS, but predict the P space or the B space in the GOS to be processed using the processed space in the GOS to be processed.

[0148] Furthermore, the encoding device and the decoding device transmit or receive the encoded stream in units of worlds each including one or more GOSs.

[0149] Furthermore, the GOS has a layer structure in at least one direction within a world, and the encoding device and decoding device encode or decode from the lower layer. For example, a randomly accessible GOS belongs to the lowest layer. A GOS belonging to a higher layer references only GOS belonging to the same layer or lower. In other words, the GOS is spatially divided in a predetermined direction and includes multiple layers, each containing one or more SPCs. The encoding device and decoding device encode or decode each SPC by referring to SPCs in the same layer as the SPC or in a layer lower than the SPC.

[0150] Furthermore, the encoding device and the decoding device encode or decode consecutive GOSs within a world unit including multiple GOSs. The encoding device and the decoding device write or read information indicating the encoding or decoding order (direction) as metadata. In other words, the encoded data includes information indicating the encoding order of multiple GOSs.

[0151] Furthermore, the encoding device and the decoding device encode or decode two or more different spaces or GOSs in parallel.

[0152] The encoding device and decoding device also encode and decode spatial information (coordinates, size, etc.) of the space or GOS.

[0153] Furthermore, the encoding device and decoding device encode or decode a space or GOS included in a specific space that is specified based on external information related to its own position and / or area size, such as GPS, route information, or magnification.

[0154] The encoding device or decoding device encodes or decodes spaces farther from its own position with lower priority than spaces closer to its own position.

[0155] The encoding device sets one direction of the world according to the magnification or use, and encodes the GOS having a layer structure in that direction. The decoding device decodes the GOS having a layer structure in one direction of the world set according to the magnification or use, preferentially from the lower layer.

[0156] The encoding device varies the feature point extraction, object recognition accuracy, spatial region size, etc., included in the indoor and outdoor spaces. However, the encoding device and decoding device encode or decode the indoor GOS and outdoor GOS that are close in coordinates as adjacent in the world, and also associate their identifiers and encode or decode them.

[0157] Although the three-dimensional data encoding device and the three-dimensional data decoding device according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0158] Furthermore, each processing unit included in the three-dimensional data encoding device or three-dimensional data decoding device according to the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0159] Furthermore, the integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured, may also be used.

[0160] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0161] The present disclosure may also be realized as a three-dimensional data encoding method or a three-dimensional data decoding method executed by a three-dimensional data encoding device or a three-dimensional data decoding device.

[0162] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0163] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be executed simultaneously (in parallel) with other steps.

[0164] Although the three-dimensional data encoding device and the three-dimensional data decoding device according to one or more aspects have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments, and configurations constructed by combining components of different embodiments, may also be included within the scope of one or more aspects. [Industrial Applicability]

[0165] The present disclosure is applicable to a three-dimensional data encoding device and a three-dimensional data decoding device. [Explanation of symbols]

[0166] 100 3D data encoding device 101,201 Acquisition Department 102 Coding area determination unit 103 Split part 104 Encoding section 111 Three-dimensional data 112,211 encoded three-dimensional data 200 Three-dimensional data decoding device 202 Decoding start GOS determination unit 203 Decoding SPC Determination Unit 204 Decoding Unit 212 Decoded 3D Data

Claims

1. A three-dimensional data encoding method for encoding three-dimensional data, comprising: Dividing the three-dimensional data into a plurality of first processing units, each of which corresponds to a three-dimensional coordinate; generating coded data by encoding each of the plurality of first processing units; The encoded data includes information indicating a profile defined according to the use of the three-dimensional data. Three-dimensional data encoding method.

2. The encoded data includes information indicating three-dimensional coordinates associated with each of the plurality of first processing units.

2. The three-dimensional data encoding method according to claim 1.

3. The size of the first processing unit is determined according to the number or density of objects or dynamic objects included in the three-dimensional data.

2. The three-dimensional data encoding method according to claim 1.

4. The encoded data includes information indicating an encoding order of the plurality of first processing units. The three-dimensional data encoding method according to any one of claims 1 to 3.

5. The encoded data includes information indicating the sizes of the first processing units. The three-dimensional data encoding method according to any one of claims 1 to 3.

6. Encoding the plurality of first processing units in parallel The three-dimensional data encoding method according to any one of claims 1 to 3.

7. A three-dimensional data decoding method for decoding three-dimensional data, comprising: acquiring each piece of coded data obtained by coding a plurality of first processing units, each of which is associated with a three-dimensional coordinate; Acquire information indicating a profile defined according to the use of the three-dimensional data; decoding each of the encoded data using the information; Three-dimensional data decoding method.

8. The encoded data includes information indicating three-dimensional coordinates associated with each of the plurality of first processing units.

8. The three-dimensional data decoding method according to claim 7.

9. The size of the first processing unit varies depending on the number or density of objects or dynamic objects included in the three-dimensional data.

8. The three-dimensional data decoding method according to claim 7.

10. The encoded data includes information indicating a decoding order of the plurality of first processing units. The three-dimensional data decoding method according to any one of claims 7 to 9.

11. The encoded data includes information indicating the sizes of the first processing units. The three-dimensional data decoding method according to any one of claims 7 to 9.

12. The plurality of first processing units are decodable in parallel. The three-dimensional data decoding method according to any one of claims 7 to 9.

13. A three-dimensional data encoding device that encodes three-dimensional data, a processor; a memory; The processor uses the memory to: Dividing the three-dimensional data into a plurality of first processing units, each of which corresponds to a three-dimensional coordinate; generating coded data by encoding each of the plurality of first processing units; The encoded data includes information indicating a profile defined according to the use of the three-dimensional data. Three-dimensional data encoding device.

14. A three-dimensional data decoding device for decoding three-dimensional data, comprising: a processor; a memory; The processor uses the memory to: acquiring each piece of coded data obtained by coding a plurality of first processing units, each of which is associated with a three-dimensional coordinate; Acquire information indicating a profile defined according to the use of the three-dimensional data; decoding each of the encoded data using the information; Three-dimensional data decoding device.

Images