Three-dimensional data processing method and three-dimensional data processing device

Abstract

The invention relates to a three-dimensional data processing method and a three-dimensional data processing device. A three-dimensional data processing method of the present disclosure includes generating an N-way tree structure of a plurality of three-dimensional points, where N is an integer of 2 or more, and generating information indicating whether 8 bits corresponding to nodes included in the N-way tree structure include 4 or more bits each corresponding to a fixed value.

Description

[0001] This application is a divisional case of the invention patent application with the application date of October 11, 2019, application number 201980066145.2, and invention name “Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device”. Technical Field

[0002] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Art

[0003] Devices and services that utilize three-dimensional data will become increasingly common in a wide range of fields, including computer vision for autonomous vehicle and robot operations, mapping, surveillance, infrastructure inspection, and image distribution. Three-dimensional data is acquired using a variety of methods, including distance sensors such as rangefinders, stereo cameras, and combinations of multiple single-lens reflex cameras.

[0004] One method of representing three-dimensional data is a point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space (for example, see Non-Patent Document 1). The point cloud stores the position and color of the point group. Although point clouds are expected to become the mainstream method of representing three-dimensional data, the amount of point group data is very large. Therefore, in the storage or transmission of three-dimensional data, as with two-dimensional dynamic images (for example, MPEG-4AVC or HEVC standardized by MPEG), data compression through encoding is required.

[0005] Furthermore, compression of point clouds is partially supported by a public library (PointCloud Library) that performs point cloud association processing.

[0006] Furthermore, there is a known technique for searching for facilities around a vehicle using three-dimensional map data and displaying the facilities (for example, see Patent Document 1).

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1 International Publication No. 2014 / 020663 Summary of the Invention

[0010] Problems to be solved by the invention

[0011] It is hoped that the coding efficiency can be improved in the coding process of three-dimensional data.

[0012] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

[0013] Means used to solve problems

[0014] A three-dimensional data processing method in one form of the present disclosure includes: generating an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and generating information indicating whether the 8 bits corresponding to the nodes included in the N-ary tree structure include more than 4 bits corresponding to fixed values, respectively.

[0015] A three-dimensional data processing method in one form of the present disclosure includes: obtaining an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and obtaining information indicating whether the 8 bits corresponding to the nodes included in the N-ary tree structure include more than 4 bits corresponding to fixed values.

[0016] A three-dimensional data processing device in one form of the present disclosure includes a processor and a memory. The processor uses the memory to generate an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and generates information indicating whether the 8 bits corresponding to the nodes included in the N-ary tree structure include more than 4 bits corresponding to fixed values.

[0017] A three-dimensional data processing device in one form of the present disclosure includes a processor and a memory. The processor uses the memory to obtain an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and obtains information indicating whether the 8 bits corresponding to the nodes included in the N-ary tree structure include more than 4 bits corresponding to fixed values.

[0018] Effects of the Invention

[0019] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a diagram showing the structure of encoded three-dimensional data according to the first embodiment.

[0021] Figure 2 This is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of the GOS according to the first embodiment.

[0022] Figure 3 This is a diagram showing an example of an inter-layer prediction structure involved in Implementation 1.

[0023] Figure 4 This is a diagram showing an example of the coding order of the GOS according to the first embodiment.

[0024] Figure 5 This is a diagram showing an example of the coding order of the GOS according to the first embodiment.

[0025] Figure 6 This is a block diagram of the three-dimensional data encoding device involved in Embodiment 1.

[0026] Figure 7 This is a flowchart of the encoding process involved in Implementation 1.

[0027] Figure 8 This is a block diagram of the three-dimensional data decoding device according to the first embodiment.

[0028] Figure 9 This is a flowchart of the decoding process involved in Implementation 1.

[0029] Figure 10 This is a diagram showing an example of meta-information according to the first embodiment.

[0030] Figure 11 This is a diagram showing a configuration example of a SWLD according to the second embodiment.

[0031] Figure 12 This is a diagram showing an example of the operation of the server and the client according to the second embodiment.

[0032] Figure 13 This is a diagram showing an example of the operation of the server and the client according to the second embodiment.

[0033] Figure 14 This is a diagram showing an example of the operation of the server and the client according to the second embodiment.

[0034] Figure 15 This is a diagram showing an example of the operation of the server and the client according to the second embodiment.

[0035] Figure 16 This is a block diagram of a three-dimensional data encoding device according to the second embodiment.

[0036] Figure 17 This is a flowchart of the encoding process involved in Implementation 2.

[0037] Figure 18 This is a block diagram of a three-dimensional data decoding device according to the second embodiment.

[0038] Figure 19 This is a flowchart of the decoding process involved in Implementation 2.

[0039] Figure 20 This is a diagram showing a configuration example of a WLD according to the second embodiment.

[0040] Figure 21 This is a diagram showing an example of an octree structure of a WLD according to the second embodiment.

[0041] Figure 22 This is a diagram showing a configuration example of a SWLD according to the second embodiment.

[0042] Figure 23 This is a diagram showing an example of an octree structure of a SWLD according to the second embodiment.

[0043] Figure 24 This is a schematic diagram showing a state of transmission and reception of three-dimensional data between vehicles according to the third embodiment.

[0044] Figure 25 This is a diagram showing an example of three-dimensional data transmitted between vehicles according to the third embodiment.

[0045] Figure 26 This is a block diagram of a three-dimensional data creation device according to the third embodiment.

[0046] Figure 27 This is a flowchart of the three-dimensional data creation process involved in the third embodiment.

[0047] Figure 28 This is a block diagram of a three-dimensional data transmitting device according to Embodiment 3.

[0048] Figure 29 This is a flowchart of the three-dimensional data transmission process involved in the third embodiment.

[0049] Figure 30 This is a block diagram of a three-dimensional data creation device according to the third embodiment.

[0050] Figure 31 This is a flowchart of the three-dimensional data creation process involved in the third embodiment.

[0051] Figure 32 This is a block diagram of a three-dimensional data transmitting device according to Embodiment 3.

[0052] Figure 33 This is a flowchart of the three-dimensional data transmission process involved in the third embodiment.

[0053] Figure 34 This is a block diagram of a three-dimensional information processing device according to the fourth embodiment.

[0054] Figure 35 This is a flowchart of the three-dimensional information processing method involved in the fourth embodiment.

[0055] Figure 36 This is a flowchart of the three-dimensional information processing method involved in the fourth embodiment.

[0056] Figure 37 This is a diagram for explaining the three-dimensional data transmission process involved in the fifth embodiment.

[0057] Figure 38 This is a block diagram of a three-dimensional data creation device according to the fifth embodiment.

[0058] Figure 39 This is a flowchart of the three-dimensional data creation method involved in the fifth embodiment.

[0059] Figure 40 This is a flowchart of the three-dimensional data creation method involved in the fifth embodiment.

[0060] Figure 41 This is a flowchart of the display method involved in embodiment 6.

[0061] Figure 42 This is a diagram showing an example of the surrounding environment viewed through the windshield according to the sixth embodiment.

[0062] Figure 43 A diagram showing a display example of a head-up display according to the sixth embodiment.

[0063] Figure 44 It is a diagram showing a display example of the head-up display after adjustment according to the sixth embodiment.

[0064] Figure 45 This is a diagram showing the configuration of a system according to the seventh embodiment.

[0065] Figure 46 This is a block diagram of a client device according to the seventh embodiment.

[0066] Figure 47 This is a block diagram of the server involved in Implementation Method 7.

[0067] Figure 48 This is a flowchart of the three-dimensional data creation process performed by the client device involved in the seventh embodiment.

[0068] Figure 49 This is a flowchart of sensor information transmission processing performed by the client device according to the seventh embodiment.

[0069] Figure 50 This is a flowchart of the three-dimensional data creation process performed by the server involved in the seventh embodiment.

[0070] Figure 51This is a flowchart of the three-dimensional map transmission process performed by the server involved in the seventh embodiment.

[0071] Figure 52 This is a diagram showing the configuration of a modified example of the system according to the seventh embodiment.

[0072] Figure 53 This is a diagram showing the configuration of a server and a client device according to the seventh embodiment.

[0073] Figure 54 This is a block diagram of a three-dimensional data encoding device according to the eighth embodiment.

[0074] Figure 55 This is a diagram showing an example of the prediction residual involved in the eighth embodiment.

[0075] Figure 56 This is a diagram showing an example of volume according to the eighth embodiment.

[0076] Figure 57 This is a diagram showing an example of octree representation of volumes according to the eighth embodiment.

[0077] Figure 58 This is a diagram showing an example of a bit string of volume involved in Implementation Example 8.

[0078] Figure 59 This is a diagram showing an example of octree representation of volumes according to the eighth embodiment.

[0079] Figure 60 This is a diagram showing an example of volume according to the eighth embodiment.

[0080] Figure 61 This is a diagram for explaining the intra-frame prediction processing involved in embodiment 8.

[0081] Figure 62 This is a diagram used to illustrate the rotation and translation processing involved in embodiment 8.

[0082] Figure 63 This is a diagram showing a syntax example of the RT application flag and RT information involved in Implementation 8.

[0083] Figure 64 This is a diagram for explaining the inter-frame prediction processing involved in embodiment 8.

[0084] Figure 65 This is a block diagram of a three-dimensional data decoding device according to the eighth embodiment.

[0085] Figure 66 This is a flowchart of a three-dimensional data encoding process performed by the three-dimensional data encoding device according to the eighth embodiment.

[0086] Figure 67 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to the eighth embodiment.

[0087] Figure 68 This is a diagram showing an example of a tree structure according to the ninth embodiment.

[0088] Figure 69 This is a diagram showing an example of occupancy rate coding according to the ninth embodiment.

[0089] Figure 70 This is a diagram schematically showing the operation of the three-dimensional data encoding device involved in the ninth embodiment.

[0090] Figure 71 This is a diagram showing an example of geometric information involved in Implementation 9.

[0091] Figure 72 This is a diagram showing an example of selection of a coding table using geometric information involved in Implementation 9.

[0092] Figure 73 This is a diagram showing an example of selecting a coding table using structural information according to the ninth embodiment.

[0093] Figure 74 This is a diagram showing an example of selecting a coding table using attribute information according to the ninth embodiment.

[0094] Figure 75 This is a diagram showing an example of selecting a coding table using attribute information according to the ninth embodiment.

[0095] Figure 76 This is a diagram showing a structural example of a bit stream involved in Implementation 9.

[0096] Figure 77 This is a diagram showing an example of a coding table involved in Implementation 9.

[0097] Figure 78 This is a diagram showing an example of a coding table involved in Implementation 9.

[0098] Figure 79 This is a diagram showing a structural example of a bit stream involved in Implementation 9.

[0099] Figure 80 This is a diagram showing an example of a coding table involved in Implementation 9.

[0100] Figure 81 This is a diagram showing an example of a coding table involved in Implementation 9.

[0101] Figure 82This is a diagram showing an example of the bit number of the occupancy coding involved in the ninth embodiment.

[0102] Figure 83 This is a flowchart of the encoding process using geometric information involved in Implementation 9.

[0103] Figure 84 This is a flowchart of the decoding process using geometric information involved in Implementation 9.

[0104] Figure 85 This is a flowchart of the encoding process using structural information involved in Implementation 9.

[0105] Figure 86 This is a flowchart of the decoding process using structural information involved in Implementation 9.

[0106] Figure 87 This is a flowchart of the encoding process of usage attribute information involved in Implementation 9.

[0107] Figure 88 This is a flowchart of the decoding process of usage attribute information involved in the ninth embodiment.

[0108] Figure 89 This is a flowchart of the coding table selection process using geometric information involved in embodiment 9.

[0109] Figure 90 This is a flowchart of the coding table selection process using structural information involved in the ninth embodiment.

[0110] Figure 91 This is a flowchart of the coding table selection process using attribute information involved in the ninth embodiment.

[0111] Figure 92 This is a block diagram of a three-dimensional data encoding device according to the ninth embodiment.

[0112] Figure 93 This is a block diagram of a three-dimensional data decoding device according to the ninth embodiment.

[0113] Figure 94 This is a diagram showing the reference relationship in the octree structure involved in the tenth embodiment.

[0114] Figure 95 This is a diagram showing the reference relationship in the spatial area involved in embodiment 10.

[0115] Figure 96 This is a diagram showing an example of adjacent reference nodes involved in embodiment 10.

[0116] Figure 97This is a diagram showing the relationship between parent nodes and nodes involved in implementation mode 10.

[0117] Figure 98 This is a diagram showing an example of occupancy coding of a parent node according to the tenth embodiment.

[0118] Figure 99 This is a block diagram showing a three-dimensional data encoding device according to the tenth embodiment.

[0119] Figure 100 This is a block diagram showing a three-dimensional data decoding device according to the tenth embodiment.

[0120] Figure 101 This is a flowchart showing the three-dimensional data encoding process involved in the tenth embodiment.

[0121] Figure 102 This is a flowchart showing the three-dimensional data decoding process involved in the tenth embodiment.

[0122] Figure 103 This is a diagram showing an example of switching of the coding table involved in the tenth embodiment.

[0123] Figure 104 This is a diagram showing the reference relationship in the spatial region involved in the first modification of the tenth embodiment.

[0124] Figure 105 This is a diagram showing a syntax example of header information according to the first modification of the tenth embodiment.

[0125] Figure 106 This is a diagram showing a syntax example of header information according to the first modification of the tenth embodiment.

[0126] Figure 107 This is a diagram showing an example of adjacent reference nodes involved in the second variation of the tenth embodiment.

[0127] Figure 108 This is a diagram showing an example of a target node and adjacent nodes according to a second variation of the tenth embodiment.

[0128] Figure 109 This is a diagram showing the reference relationship in the octree structure involved in the third modification of the tenth embodiment.

[0129] Figure 110 This is a diagram showing the reference relationship in the spatial region involved in the third variant of the tenth embodiment.

[0130] Figure 111 This is a diagram for explaining the outline of the three-dimensional data encoding method involved in embodiment 11.

[0131] Figure 112 This is a diagram for explaining a method of converting a plane from which tilt is detected according to the eleventh embodiment into an XY plane.

[0132] Figure 113 This is a diagram showing the relationship between the plane involved in embodiment 11 and the point group selected in each method.

[0133] Figure 114 This is a diagram showing the frequency distribution of quantized distances between a plane detected from a three-dimensional point cloud and a point cloud surrounding the plane (candidates of a plane point cloud) in the first method according to the eleventh embodiment.

[0134] Figure 115 This is a diagram showing the frequency distribution of quantized distances between a plane detected from a three-dimensional point group and a point group surrounding the plane in the second method according to the eleventh embodiment.

[0135] Figure 116 This is a diagram showing an example of dividing the two-dimensional space involved in the eleventh embodiment into four subspaces.

[0136] Figure 117 This is a diagram showing an example of applying the four subspaces in the two-dimensional space according to the eleventh embodiment to the eight subspaces in the three-dimensional space.

[0137] Figure 118 This is a diagram showing the neighbor relationship of three-dimensional points of the first three-dimensional point group arranged on a plane according to the eleventh embodiment.

[0138] Figure 119 This is a diagram showing the neighbor relationship of three-dimensional points of a three-dimensional point group arranged in a three-dimensional space according to the eleventh embodiment.

[0139] Figure 120 This is a block diagram showing the structure of a three-dimensional data encoding device according to embodiment 11.

[0140] Figure 121 This is a block diagram showing the detailed structure of the quadtree encoding unit using the first method involved in the eleventh embodiment.

[0141] Figure 122 This is a block diagram showing the detailed structure of a quadtree encoding unit using the second method according to the eleventh embodiment.

[0142] Figure 123 This is a block diagram showing the structure of a three-dimensional data decoding device according to the eleventh embodiment.

[0143] Figure 124 This is a block diagram showing the detailed structure of a quadtree decoding unit using the first method according to the eleventh embodiment.

[0144] Figure 125 This is a block diagram showing the detailed structure of a quadtree decoding unit using the second method according to the eleventh embodiment.

[0145] Figure 126 This is a flowchart of the three-dimensional data encoding method involved in embodiment 11.

[0146] Figure 127 This is a flowchart of the three-dimensional data decoding method involved in embodiment 11.

[0147] Figure 128 This is a flowchart of the quadtree encoding process involved in embodiment 11.

[0148] Figure 129 This is a flowchart of the octree encoding process involved in implementation mode 11.

[0149] Figure 130 This is a flowchart of the quadtree decoding process involved in embodiment 11.

[0150] Figure 131 This is a flowchart of the octree decoding process involved in embodiment 11. DETAILED DESCRIPTION

[0151] A three-dimensional data encoding method in one form disclosed herein encodes the first information of the first object node contained in the N (N is 2 or 4) octree structure of multiple first three-dimensional points of the first three-dimensional point group, or the second information of the second object node contained in the octree structure of multiple second three-dimensional points of the second three-dimensional point group. In the encoding, the first information is encoded using a first encoding style that is the same as the second encoding style used in the encoding of the second information.

[0152] Thus, the three-dimensional data encoding method can reduce the processing load by encoding the N-ary tree structure information using the same encoding pattern as that for encoding the octree structure information.

[0153] For example, it may also be that the first coding style is a coding style for selecting a coding table used in the coding of the first information, and the second coding style is a coding style for selecting a coding table used in the coding of the second information. In the coding, the first coding style is generated based on the first adjacent information of multiple first adjacent nodes that are spatially adjacent to the first object node in multiple directions, and the second coding style is generated based on the second adjacent information of multiple second adjacent nodes that are spatially adjacent to the second object node in the multiple directions.

[0154] For example, it may also be that, in the generation of the first coding pattern, the first coding pattern including a third bit pattern of 6 bits is generated, the third bit pattern is composed of a first bit pattern and a second bit pattern, the first bit pattern is composed of one or more bits, the one or more bits representing one or more first adjacent nodes spatially adjacent to the first object node in a specified direction among multiple directions, and indicating that each is not occupied by a point group, the second bit pattern is composed of multiple bits, the multiple bits representing multiple second adjacent nodes spatially adjacent to the first object node in directions other than the specified direction among the multiple directions, and in the generation of the second coding pattern, the second coding pattern including a fourth bit pattern of 6 bits is generated, the fourth bit pattern is composed of multiple bits, the multiple bits representing multiple third adjacent nodes spatially adjacent to the second object node in the multiple directions.

[0155] For example, it may also be that, in the encoding, the first encoding table is selected based on the first encoding style, and the first information is entropy encoded using the selected first encoding table; the second encoding table is selected based on the second encoding style, and the second information is entropy encoded using the selected second encoding table.

[0156] For example, it may also be that, in the encoding, a bit stream including a third bit string consisting of 8 bits is generated by encoding the first information, and the first information indicates whether each of the N first subspaces obtained by performing N divisions on the first object node contains the first three-dimensional point, the third bit string is composed of the first bit string and an invalid second bit string, the first bit string is composed of N bits corresponding to the first information, and the second bit string is composed of (8-N) bits.

[0157] For example, a bit stream including identification information indicating whether the encoding target is the first information or the second information may be further generated.

[0158] For example, the first three-dimensional point group may be a point group arranged on a plane, and the second three-dimensional point group may be a point group arranged around the plane.

[0159] A three-dimensional data decoding method involved in one embodiment of the present invention decodes the first information of the first object node contained in the N (N is 2 or 4) octree structure of multiple first three-dimensional points of the first three-dimensional point group, or the second information of the second object node contained in the octree structure of multiple second three-dimensional points of the second three-dimensional point group. In the decoding, the first information is decoded using a first decoding style that is the same as the second decoding style used in the decoding of the second information.

[0160] Thus, the three-dimensional data decoding method can reduce the processing load by decoding the information in the N-ary tree structure using the same decoding pattern as that used for decoding the information in the octree structure.

[0161] For example, it may also be that the first decoding style is a decoding style for selecting a decoding table used in decoding the first information, and the second decoding style is a decoding style for selecting a decoding table used in decoding the second information. In the decoding, the first decoding style is generated based on the first adjacent information of multiple first adjacent nodes that are spatially adjacent to the first object node in multiple directions, and the second decoding style is generated based on the second adjacent information of multiple second adjacent nodes that are spatially adjacent to the second object node in the multiple directions.

[0162] For example, it may also be that, in the generation of the first decoding pattern, the first decoding pattern is generated, which includes a 6-bit third bit pattern, and the third bit pattern is composed of a first bit pattern and a second bit pattern, the first bit pattern is composed of one or more bits, and the one or more bits represent one or more first adjacent nodes that are spatially adjacent to the first object node in a specified direction among multiple directions, and indicate that each is not occupied by a point group, the second bit pattern is composed of multiple bits, and the multiple bits represent multiple second adjacent nodes that are spatially adjacent to the first object node in directions other than the specified direction among the multiple directions, and in the generation of the second decoding pattern, the second decoding pattern is generated, the second decoding pattern includes a 6-bit fourth bit pattern, and the fourth bit pattern is composed of multiple bits, and the multiple bits represent multiple third adjacent nodes that are spatially adjacent to the second object node in the multiple directions.

[0163] For example, in the decoding, a first decoding table is selected based on the first decoding style, and the first information is entropy decoded using the selected first decoding table; a second decoding table is selected based on the second decoding style, and the second information is entropy decoded using the selected second decoding table.

[0164] For example, it may also be that in the decoding, a bit stream including a third bit string consisting of 8 bits is obtained, the third bit string consisting of a first bit string consisting of N bits and an invalid second bit string consisting of (8-N) bits, and the first information is decoded from the first bit string of the bit stream, the first information indicating whether the first three-dimensional point is included in each of the N first subspaces obtained by performing N divisions on the first object node.

[0165] For example, the bit stream may include identification information indicating whether the encoding object is the first information or the second information, and in the decoding, when the identification information indicates that the encoding object is the first information, the first bit string in the bit stream is decoded.

[0166] For example, the first three-dimensional point group may be a point group arranged on a plane, and the second three-dimensional point group may be a point group arranged around the plane.

[0167] In addition, a three-dimensional data encoding device involved in one embodiment of the present invention includes a processor and a memory, and the processor uses the memory to encode the first information of the first object node contained in the N (N is 2 or 4) octree structure of multiple first three-dimensional points of the first three-dimensional point group, or the second information of the second object node contained in the octree structure of multiple second three-dimensional points of the second three-dimensional point group. In the encoding, the first information is encoded using a first encoding style that is the same as the second encoding style used in the encoding of the second information.

[0168] Thus, the three-dimensional data encoding method can reduce the processing load by encoding the N-ary tree structure information using the same encoding pattern as that for encoding the octree structure information.

[0169] A three-dimensional data decoding device involved in one embodiment of the present invention includes a processor and a memory, and the processor uses the memory to decode the first information of the first object node contained in the N (N is 2 or 4) octree structure of multiple first three-dimensional points of the first three-dimensional point group, or the second information of the second object node contained in the octree structure of multiple second three-dimensional points of the second three-dimensional point group. In the decoding, the first information is decoded using a first decoding style that is common to the second decoding style used in the decoding of the second information.

[0170] Thus, the three-dimensional data decoding method can reduce the processing load by decoding the information in the N-ary tree structure using the same decoding pattern as that used for decoding the information in the octree structure.

[0171] In addition, these general or specific forms can be implemented by systems, methods, integrated circuits, computer programs or computer-readable recording media such as CD-ROMs, and can be implemented by any combination of systems, methods, integrated circuits, computer programs and recording media.

[0172] The following detailed description of the embodiments is given with reference to the accompanying drawings. In addition, the embodiments to be described below are all specific examples of the present disclosure. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, order of steps, etc. shown in the following embodiments are all examples, and their purpose is not to limit the present disclosure. Moreover, among the components of the following embodiments, the components that are not recorded in the technical solution showing the highest concept are described as arbitrary components.

[0173] (Implementation 1)

[0174] First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to this embodiment will be described. Figure 1 This is a diagram showing the structure of encoded three-dimensional data according to this embodiment.

[0175] In this embodiment, the three-dimensional space is divided into spaces (SPC) equivalent to the spaces of pictures in the encoding of dynamic images, and the three-dimensional data is encoded in units of space. The space is further divided into volumes (VLM) equivalent to macroblocks in dynamic image encoding, and prediction and transformation are performed in units of VLM. The volume includes a plurality of voxels (VXL), which are the smallest units corresponding to position coordinates. In addition, prediction means generating predicted three-dimensional data similar to the processing unit of the processing object with reference to other processing units, and encoding the difference between the predicted three-dimensional data and the processing unit of the processing object, similar to the prediction performed in the two-dimensional image. Furthermore, the prediction includes not only spatial prediction with reference to other prediction units at the same time, but also temporal prediction with reference to prediction units at different times.

[0176] For example, when encoding a three-dimensional space represented by point cloud data, such as a point cloud, a three-dimensional data encoding device (hereinafter referred to as an encoding device) encodes each point in the point cloud or multiple points contained within a voxel, all at once, according to the size of the voxel. Subdividing the voxels allows for a highly accurate representation of the point cloud's three-dimensional shape, while increasing the voxel size allows for a coarser representation of the point cloud's three-dimensional shape.

[0177] In addition, although the following description uses the case where the three-dimensional data is a point cloud as an example, the three-dimensional data is not limited to point clouds and can also be three-dimensional data in any form.

[0178] Furthermore, voxels with a hierarchical structure can be utilized. In this case, within the nth level, it is possible to sequentially indicate whether a sampling point exists in the n-1th level or lower levels (the levels below the nth level). For example, when decoding only the nth level, if a sampling point exists in the n-1th level or lower levels, decoding can be performed as if the sampling point exists at the center of the voxel in the nth level.

[0179] Furthermore, the encoding device obtains point group data through a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor.

[0180] Similar to video encoding, spaces are classified into at least one of the following three prediction structures: independently decodable intra-frame space (I-SPC), unidirectionally referenced prediction space (P-SPC), and bidirectionally referenced prediction space (B-SPC). Spaces also contain two types of time information: decoding time and display time.

[0181] And, as Figure 1 As shown, as a processing unit including a plurality of spaces, there is a GOS (Group of Space) which is a random access unit. Furthermore, as a processing unit including a plurality of GOS, there is a world space (WLD).

[0182] The spatial area occupied by world space is associated with an absolute position on Earth using GPS or latitude and longitude information. This location information is stored as metadata. Metadata can be included in the encoded data or transmitted separately.

[0183] Furthermore, within the GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0184] In addition, below, the encoding, decoding, or referencing of three-dimensional data contained in a processing unit such as a GOS, SPC, or VLM will also be referred to simply as encoding, decoding, or referencing the processing unit. Furthermore, the three-dimensional data contained in the processing unit includes, for example, at least one pair of spatial positions such as three-dimensional coordinates and characteristic values such as color information.

[0185] Next, the prediction structure of the SPC in the GOS will be described. Although multiple SPCs in the same GOS or multiple VLMs in the same SPC occupy different spaces, they have the same time information (decoding time and display time).

[0186] Furthermore, within a GOS, the first SPC in decoding order is the I-SPC. Furthermore, there are two types of GOS: closed GOS and open GOS. A closed GOS is one that can decode all SPCs within the GOS when decoding starts from the first I-SPC. In an open GOS, some SPCs within the GOS that are earlier than the display time of the first I-SPC refer to a different GOS and can only be decoded in that GOS.

[0187] In addition, in coded data such as map information, the WLD may be decoded in the reverse order of the coding order. If there is a dependency between GOS, it will be difficult to reproduce the data in the reverse order. Therefore, in this case, a closed GOS is basically used.

[0188] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the bottom layer.

[0189] Figure 2 This is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of the GOS. Figure 3 This is a diagram showing an example of an inter-layer prediction structure.

[0190] There are one or more I-SPCs within a GOS. While objects such as people, animals, cars, bicycles, traffic lights, and buildings serving as land landmarks exist within a three-dimensional space, encoding small objects as I-SPCs is particularly effective. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPCs within the GOS when decoding the GOS with low throughput or high speed.

[0191] Furthermore, the encoding device may switch the encoding interval or the appearance frequency of the I-SPC according to the density of objects within the WLD.

[0192] And, in Figure 3 In the configuration shown, the encoding device or decoding device encodes or decodes multiple layers sequentially starting from the lower layer (layer 1). This allows, for example, autonomous vehicles to prioritize data near the ground, which contains a large amount of information.

[0193] In addition, in the coded data used by drones, etc., coding or decoding can be performed sequentially starting from the SPC of the upper layer in the height direction within the GOS.

[0194] Furthermore, the encoding device or decoding device may encode or decode multiple layers in such a manner that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding device or decoding device may encode or decode layers in the order of 3, 8, 1, 9, etc.

[0195] Next, the corresponding method of static objects and dynamic objects is explained.

[0196] In three-dimensional space, there are static objects or scenes such as buildings and roads (hereinafter referred to as static objects), as well as dynamic objects such as vehicles and people (hereinafter referred to as dynamic objects). Object detection can also be performed by extracting feature points from point cloud data or images captured by stereo cameras. Here, an example of a method for encoding dynamic objects is described.

[0197] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects using identification information.

[0198] For example, GOS is used as the identification unit. In this case, GOS including SPCs constituting static objects and GOS including SPCs constituting dynamic objects are distinguished within the coded data or by identification information stored separately from the coded data.

[0199] Alternatively, SPC is used as the identification unit. In this case, the SPC including only the VLM constituting the static object and the SPC including the VLM constituting the dynamic object are distinguished by the above-mentioned identification information.

[0200] Alternatively, VLM or VXL may be used as the identification unit. In this case, a VLM or VXL including a static object and a VLM or VXL including a dynamic object are distinguished by the above-mentioned identification information.

[0201] Furthermore, the encoding device may encode dynamic objects as one or more VLMs or SPCs, and may encode the VLMs or SPCs containing static objects and the SPCs containing dynamic objects as separate GOSs. Furthermore, if the size of the GOS is variable depending on the size of the dynamic object, the encoding device may store the size of the GOS separately as meta-information.

[0202] Furthermore, the encoding device encodes static objects and dynamic objects independently, allowing dynamic objects to be superimposed on the world space composed of static objects. In this case, a dynamic object is composed of one or more SPCs, each of which corresponds to one or more SPCs that constitute the static object with which it is superimposed. Furthermore, dynamic objects can be represented not by SPCs but by one or more VLMs or VXLs.

[0203] Furthermore, the encoding device may encode static objects and dynamic objects as different streams.

[0204] Furthermore, the encoding device may generate a GOS that includes one or more SPCs constituting a dynamic object. Furthermore, the encoding device may set the GOS (GOS_M) including the dynamic object and the GOS of the static object corresponding to the spatial region of the GOS_M to be of the same size (occupying the same spatial region). This enables overlapping processing on a GOS basis.

[0205] The P-SPC or B-SPC that constitutes a dynamic object can also refer to the SPC contained in a different coded GOS. When the position of a dynamic object changes over time and the same dynamic object is coded as a GOS at different times, cross-GOS reference is effective from the perspective of compression rate.

[0206] Furthermore, the first and second methods can be switched depending on the intended use of the encoded data. For example, when the encoded three-dimensional data is used as a map, separation from dynamic objects is desirable, so the encoding device uses the second method. Alternatively, when encoding three-dimensional data of events such as concerts or sports, where separation of dynamic objects is not necessary, the encoding device uses the first method.

[0207] Furthermore, the decoding time and display time of the GOS or SPC can be stored in the encoded data or as meta-information. Furthermore, the time information of static objects can all be the same. In this case, the actual decoding time and display time can be determined by the decoding device. Alternatively, different values can be assigned to each GOS or SPC as the decoding time, and the same value can be assigned to all as the display time. Moreover, as shown in the decoder mode in dynamic image encoding such as HEVC's HRD (Hypothetical Reference Decoder), the decoder has a buffer of a specified size. As long as the bit stream is read at a specified bit rate according to the decoding time, a model that will not be destroyed and is guaranteed to be decodable can be imported.

[0208] Next, the configuration of GOS in the world space is described. The coordinates of the three-dimensional space in the world space are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, z-axis). By setting a prescribed rule in the encoding order of GOS, spatially adjacent GOS can be encoded continuously in the encoded data. For example, Figure 4 In the example shown, the GOS within the xz plane are encoded continuously. After encoding all GOS within a single xz plane, the y-axis value is updated. In other words, as encoding continues, world space extends in the y-axis direction. Furthermore, the GOS index numbers are set to the encoding order.

[0209] Here, the three-dimensional world space corresponds one-to-one to absolute geographic coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space can be represented by relative positions relative to a predetermined reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, and these direction vectors are stored as metadata along with the encoded data.

[0210] Furthermore, the size of the GOS is set to be fixed, and the encoding device stores this size as meta-information. Furthermore, the size of the GOS can be switched, for example, depending on whether the location is urban, indoors, or outdoors. In other words, the size of the GOS can be switched based on the quantity or nature of objects with informational value. Alternatively, within the same world space, the encoding device can appropriately switch the size of the GOS or the spacing of I-SPCs within the GOS based on, for example, the density of objects. For example, the encoding device can set the GOS size to be smaller and the spacing of I-SPCs within the GOS to be shorter when the density of objects is higher.

[0211] exist Figure 5 In the example, in the area from the 3rd to the 10th GOS, the density of objects is high, so the GOS is subdivided to achieve fine-grained random access. In addition, the 7th to the 10th GOS exist on the back of the 3rd to the 6th GOS, respectively.

[0212] Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be described. Figure 6 This is a block diagram of the three-dimensional data encoding device 100 according to this embodiment. Figure 7 3D data encoding apparatus 100 is a flowchart showing an example of its operation.

[0213] Figure 6 The three-dimensional data encoding device 100 shown generates encoded three-dimensional data 112 by encoding three-dimensional data 111. The three-dimensional data encoding device 100 includes an acquisition unit 101, an encoding region determination unit 102, a division unit 103, and an encoding unit 104.

[0214] like Figure 7 As shown, first, the acquisition unit 101 acquires three-dimensional data 111 as point cloud data ( S101 ).

[0215] Next, the coding region determination unit 102 determines a coding target region from the spatial region corresponding to the obtained point cloud data (S102). For example, the coding region determination unit 102 determines a spatial region around the position of the user or vehicle as the coding target region.

[0216] Next, the segmentation unit 103 segments the point cloud data contained in the encoding target region into processing units. Here, the processing units are the aforementioned GOS and SPCs, for example. Furthermore, the encoding target region corresponds to the aforementioned world space, for example. Specifically, the segmentation unit 103 segments the point cloud data into processing units based on the size of the pre-set GOS and the presence or size of dynamic objects (S103). Furthermore, the segmentation unit 103 determines the starting position of the SPC, which is the first in the encoding order, within each GOS.

[0217] Next, the encoding unit 104 generates the encoded three-dimensional data 112 by sequentially encoding the plurality of SPCs in each GOS ( S104 ).

[0218] In addition, although an example is shown here in which each GOS is encoded after the encoding target area is divided into GOS and SPC, the processing order is not limited to the above. For example, the order may be such that the composition of a GOS is determined, the GOS is encoded, and then the composition of the GOS is determined.

[0219] In this manner, the 3D data encoding device 100 generates encoded 3D data 112 by encoding the 3D data 111. Specifically, the 3D data encoding device 100 divides the 3D data into random access units, namely, first processing units (GOS) corresponding to respective 3D coordinates. The first processing units (GOS) are divided into a plurality of second processing units (SPCs), and the second processing units (SPCs) are divided into a plurality of third processing units (VLMs). Furthermore, each third processing unit (VLM) includes one or more voxels (VXLs), which are the smallest units corresponding to positional information.

[0220] Next, the 3D data encoding device 100 generates encoded 3D data 112 by encoding each of the plurality of first processing units (GOS). Specifically, the 3D data encoding device 100 encodes each of the plurality of second processing units (SPC) within each first processing unit (GOS). Furthermore, the 3D data encoding device 100 encodes each of the plurality of third processing units (VLM) within each second processing unit (SPC).

[0221] For example, when the first processing unit (GOS) to be processed is a closed GOS, the three-dimensional data encoding device 100 encodes the second processing unit (SPC) to be processed included in the first processing unit (GOS) to be processed by referring to other second processing units (SPC) included in the first processing unit (GOS) to be processed. In other words, the three-dimensional data encoding device 100 does not refer to the second processing unit (SPC) included in the first processing unit (GOS) different from the first processing unit (GOS) to be processed.

[0222] Furthermore, when the first processing unit (GOS) of the processing object is an open GOS, the second processing unit (SPC) of the processing object included in the first processing unit (GOS) of the processing object is encoded with reference to other second processing units (SPC) included in the first processing unit (GOS) of the processing object, or a second processing unit (SPC) included in a first processing unit (GOS) different from the first processing unit (GOS) of the processing object.

[0223] Furthermore, the three-dimensional data encoding device 100 selects one as the type of the second processing unit (SPC) of the processing object from a first type (I-SPC) that does not refer to other second processing units (SPCs), a second type (P-SPC) that refers to one other second processing unit (SPC), and a third type that refers to two other second processing units (SPCs), and encodes the second processing unit (SPC) of the processing object according to the selected type.

[0224] Next, the configuration and operation flow of the three-dimensional data decoding device according to this embodiment will be described. Figure 8 This is a block diagram of a three-dimensional data decoding device 200 according to this embodiment. Figure 9 3D data decoding apparatus 200 is a flowchart showing an example of its operation.

[0225] Figure 8 The illustrated three-dimensional data decoding device 200 generates decoded three-dimensional data 212 by decoding encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. The three-dimensional data decoding device 200 includes an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0226] First, the acquisition unit 201 acquires the encoded 3D data 211 (S201). Next, the decoding start GOS determination unit 202 determines the GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to metadata stored within or separately from the encoded 3D data 211 and determines the GOS that includes the SPC corresponding to the spatial position, object, or time at which decoding should start as the decoding start GOS.

[0227] Next, the decoded SPC determination unit 203 determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the decoded SPC determination unit 203 determines (1) whether to decode only the I-SPC, (2) whether to decode both the I-SPC and the P-SPC, or (3) whether to decode all types. Alternatively, this step may be omitted if the type of SPC to be decoded is predetermined, such as when all SPCs are to be decoded.

[0228] Next, the decoding unit 204 obtains the first SPC in the decoding order (the same as the encoding order) within the GOS, and the address position at the beginning of the encoded 3D data 211. From this address position, it obtains the encoded data of the first SPC and decodes each SPC sequentially from the first SPC (S204). The address position is stored in meta-information, etc.

[0229] In this manner, the 3D data decoding device 200 decodes the decoded 3D data 212. Specifically, the 3D data decoding device 200 decodes each of the encoded 3D data 211 of the first processing unit (GOS) corresponding to the three-dimensional coordinates, thereby generating the decoded 3D data 212 of the first processing unit (GOS), which is a random access unit. More specifically, the 3D data decoding device 200 decodes each of the plurality of second processing units (SPC) for each first processing unit (GOS). Furthermore, the 3D data decoding device 200 decodes each of the plurality of third processing units (VLM) for each second processing unit (SPC).

[0230] The following describes the meta-information for random access. This meta-information is generated by the three-dimensional data encoding device 100 and included in the encoded three-dimensional data 112 (211).

[0231] In conventional random access of two-dimensional moving images, decoding starts from the first frame of the random access unit near a specified time. However, in world space, random access to (coordinates, objects, etc.) is also possible in addition to time.

[0232] Therefore, in order to realize random access to at least the three elements of coordinates, objects, and time, a table is prepared that associates each element with the index number of the GOS. Furthermore, the index number of the GOS is associated with the address of the I-SPC at the beginning of the GOS. Figure 10 This is a diagram showing an example of a table included in meta information. Figure 10 Of all the tables shown, at least one table may be used.

[0233] The following describes random access using coordinates as the starting point, as an example. When accessing the coordinates (x2, y2, z2), the coordinate-GOS table is first referenced, and it is determined that the location with coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is referenced, and since the address of the first I-SPC in the second GOS is known to be addr(2), the decoding unit 204 obtains data from this address and begins decoding.

[0234] In addition, the address can be an address in a logical format or a physical address of an HDD or memory. Furthermore, information identifying a file segment can be used instead of an address. For example, a file segment is a unit obtained by segmenting one or more GOSs.

[0235] Furthermore, if an object spans multiple GOSs, the object GOS table may also indicate the GOSs to which the multiple objects belong. If the multiple GOSs are closed, the encoding and decoding devices can perform encoding and decoding in parallel. Furthermore, if the multiple GOSs are open, the GOSs can reference each other, further improving compression efficiency.

[0236] Examples of objects include people, animals, cars, bicycles, traffic lights, and buildings that serve as landmarks on land. For example, when encoding in world space, the three-dimensional data encoding device 100 extracts feature points unique to an object from a three-dimensional point cloud, detects the object based on the feature points, and can set the detected object as a random access point.

[0237] In this manner, the three-dimensional data encoding device 100 generates first information indicating a plurality of first processing units (GOS) and the three-dimensional coordinates corresponding to each of the plurality of first processing units (GOS). The encoded three-dimensional data 112 (211) includes the first information. Furthermore, the first information further indicates at least one of an object, a time, and a data storage destination corresponding to each of the plurality of first processing units (GOS).

[0238] The three-dimensional data decoding device 200 obtains first information from the encoded three-dimensional data 211 , uses the first information to identify the first processing unit of encoded three-dimensional data 211 corresponding to the specified three-dimensional coordinates, object or time, and decodes the encoded three-dimensional data 211 .

[0239] Other examples of meta-information are described below. In addition to the meta-information for random access, the 3D data encoding device 100 can also generate and store the following meta-information. The 3D data decoding device 200 can also use this meta-information during decoding.

[0240] When using 3D data as map information, profiles are defined based on the intended use, and information indicating the profiles can be included in the meta-information. For example, profiles for urban areas or suburban areas, or for flying objects, can be defined, with the maximum and minimum sizes of the world space, SPC, or VLM defined for each. For example, in an urban area profile, more detailed information is required than in a suburban area, so the minimum size of the VLM is set smaller.

[0241] Meta-information may also include a tag value indicating the type of object. This tag value corresponds to the VLM, SPC, or GOS that constitutes the object. Tag values can be set based on the object type, for example, a tag value of "0" representing a "person," a tag value of "1" representing a "car," and a tag value of "2" representing a "traffic light." Alternatively, when the object type is difficult or unnecessary to determine, a tag value indicating properties such as size or whether the object is dynamic or static can be used.

[0242] Furthermore, the meta-information may include information indicating the range of the spatial region occupied by the world space.

[0243] Furthermore, the meta-information may be used as header information shared by the entire stream of coded data or a plurality of SPCs such as an SPC within a GOS, and may store the size of the SPC or VXL.

[0244] Furthermore, the meta-information may include identification information of a distance sensor, a camera, or the like used in generating the point cloud, or information indicating the positional accuracy of a point group within the point cloud.

[0245] Also, the meta information may include information showing whether the world space is composed of only static objects or contains dynamic objects.

[0246] Modifications of this embodiment will be described below.

[0247] The encoding device or decoding device can encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel can be determined based on meta-information indicating the spatial position of the GOSs.

[0248] When three-dimensional data is used as a spatial map for a moving vehicle or flying object, or when such a spatial map is generated, the encoding device or decoding device can encode or decode the GOS or SPC contained in a space determined based on GPS, path information, or zoom factor, etc.

[0249] Furthermore, the decoding device may also decode sequentially starting from spaces closest to its own position or path. The encoding device or decoding device may also prioritize spaces farther from its own position or path, lowering the priority of spaces closer to it, and perform encoding or decoding accordingly. Here, lowering the priority means reducing the processing order, reducing the resolution (post-filtering processing), or reducing image quality (to improve coding efficiency, such as increasing the quantization step size).

[0250] Furthermore, when decoding coded data that has been spatially hierarchically coded, the decoding device may decode only the lower layer.

[0251] Furthermore, the decoding device may start decoding from a lower layer according to the zoom ratio or purpose of the map.

[0252] Furthermore, in applications such as estimating the self-position of a car or robot during autonomous driving or identifying objects, the encoding device or decoding device may also reduce the resolution of areas outside the area within a specified height from the road surface (the area for identification) for encoding or decoding.

[0253] Furthermore, the encoding device may independently encode the point clouds representing indoor and outdoor spatial shapes. For example, by separating the GOS representing indoor spaces (indoor GOS) from the GOS representing outdoor spaces (outdoor GOS), the decoding device can select the GOS to decode based on the viewpoint position when using the encoded data.

[0254] Furthermore, the encoding device may encode indoor and outdoor GOSs that are located close together in the coded stream. For example, the encoding device may associate identifiers between the two and store information indicating the associated identifiers within the coded stream or in separately stored meta-information. This allows the decoding device to identify the indoor and outdoor GOSs that are located close together by referring to the meta-information.

[0255] Furthermore, the encoding device may switch the size of the GOS or SPC between the indoor GOS and the outdoor GOS. For example, the encoding device may set the size of the GOS to be smaller indoors than outdoors. Furthermore, the encoding device may change the accuracy of feature point extraction from the point cloud or the accuracy of object detection between the indoor GOS and the outdoor GOS.

[0256] Furthermore, the encoding device can add information to the encoded data that allows the decoding device to distinguish between dynamic and static objects. This allows the decoding device to display dynamic objects in combination with a red frame or explanatory text. Alternatively, the decoding device can replace dynamic objects with a red frame or explanatory text alone. Furthermore, the decoding device can display more detailed object categories. For example, a car can be displayed with a red frame, while a person can be displayed with a yellow frame.

[0257] Furthermore, the encoding or decoding device may determine whether to encode or decode dynamic objects and static objects as different SPCs or GOSs based on the frequency of occurrence of dynamic objects or the ratio of static objects to dynamic objects. For example, if the frequency or ratio of dynamic objects exceeds a threshold, an SPC or GOS containing a mixture of dynamic and static objects is permitted. If the frequency or ratio of dynamic objects does not exceed the threshold, an SPC or GOS containing a mixture of dynamic and static objects is not permitted.

[0258] When dynamic objects are detected from 2D camera image information rather than point clouds, the encoding device can obtain information identifying the detection result (box or text, etc.) and the object position separately, encoding this information as part of the 3D encoded data. In this case, the decoding device displays auxiliary information representing the dynamic object (box or text) overlaid on the decoded result of the static object.

[0259] Furthermore, the encoding device may change the density of VXL or VLM based on factors such as the complexity of the shape of the static object. For example, the encoding device may set VXL or VLM to be denser when the shape of the static object is more complex. Furthermore, the encoding device may determine the quantization step size when quantizing spatial position or color information based on the density of VXL or VLM. For example, the encoding device may set the quantization step size to be smaller when the VXL or VLM is denser.

[0260] As described above, the encoding device or decoding device according to the present embodiment performs spatial encoding or decoding in spatial units having coordinate information.

[0261] Furthermore, the encoding device and the decoding device perform encoding or decoding in units of volume in space. The volume includes voxels, which are the minimum units corresponding to the position information.

[0262] Furthermore, the encoding device and decoding device perform encoding or decoding by creating a table that associates various elements of spatial information, including coordinates, objects, and time, with GOPs, or by creating a table that associates various elements, thereby establishing a correspondence between arbitrary elements. Furthermore, the decoding device uses the values of the selected elements to determine coordinates, identifies a volume, voxel, or space based on the coordinates, and decodes the space including the volume or voxel, or the identified space.

[0263] Furthermore, the encoding device determines a volume, voxel, or space that can be selected by an element through feature point extraction or object recognition, and encodes it as a volume, voxel, or space that can be randomly accessed.

[0264] Spaces are divided into three types: I-SPC, which can be encoded or decoded by the space itself; P-SPC, which can be encoded or decoded with reference to any one processed space; and B-SPC, which can be encoded or decoded with reference to any two processed spaces.

[0265] One or more volumes correspond to static objects or dynamic objects. The space containing static objects and the space containing dynamic objects are encoded or decoded as different GOSs. That is, the SPC containing static objects and the SPC containing dynamic objects are assigned to different GOSs.

[0266] Dynamic objects are encoded or decoded for each object and correspond to one or more spaces containing only static objects. In other words, multiple dynamic objects are encoded separately, and the resulting encoded data of multiple dynamic objects corresponds to an SPC containing only static objects.

[0267] The encoding and decoding devices increase the priority of the I-SPC within the GOS to perform encoding or decoding. For example, the encoding device performs encoding to minimize degradation of the I-SPC (enabling more faithful reproduction of the original 3D data after decoding). Furthermore, the decoding device, for example, only decodes the I-SPC.

[0268] The encoding device can change the frequency of using I-SPCs to perform encoding according to the density or number of objects in world space. Specifically, the encoding device changes the frequency of selecting I-SPCs according to the number or density of objects contained in the three-dimensional data. For example, the encoding device may increase the frequency of using I-space as the density of objects in world space increases.

[0269] Furthermore, the encoding device sets a random access point in units of GOS, and stores information indicating a spatial region corresponding to the GOS in the header information.

[0270] The encoding device, for example, uses a default value as the spatial size of the GOS. Alternatively, the encoding device may change the size of the GOS based on the number (quantity) or density of objects or dynamic objects. For example, the encoding device may set the spatial size of the GOS to be smaller when the objects or dynamic objects are denser or more numerous.

[0271] Furthermore, the space or volume includes a cluster of feature points derived from information obtained by sensors such as depth sensors, gyroscopes, or cameras. The coordinates of the feature points are set to the center of the voxel. Furthermore, by subdividing the voxels, it is possible to achieve higher accuracy in positional information.

[0272] The feature point cluster is derived using multiple pictures. The multiple pictures have at least two types of time information: actual time information and spatially corresponding time information common to multiple pictures (for example, encoding time used for rate control, etc.).

[0273] Furthermore, encoding or decoding is performed in units of GOS including one or more spaces.

[0274] The encoding device and the decoding device refer to the space in the processed GOS to predict the P space or B space in the GOS to be processed.

[0275] Alternatively, the encoding device and the decoding device predict the P space or B space in the GOS to be processed by using the processed space in the GOS to be processed without referring to different GOSs.

[0276] Furthermore, the encoding device and the decoding device transmit or receive the encoded stream in units of a world space including one or more GOSs.

[0277] Furthermore, the GOS has a layer structure in at least one direction within world space, and the encoding and decoding devices perform encoding or decoding starting from the lowest layer. For example, the randomly accessible GOS belongs to the lowest layer. The GOS belonging to a higher layer only references the GOS belonging to the layers below it. In other words, the GOS is spatially divided in a predetermined direction and includes multiple layers, each having one or more SPCs. The encoding and decoding devices perform encoding or decoding for each SPC by referencing SPCs contained in layers in the same layer or layers below it.

[0278] Furthermore, the encoding device and decoding device continuously encode or decode the GOS within a world space unit that includes multiple GOS. The encoding device and decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. In other words, the encoded data includes information indicating the order in which the multiple GOS were encoded.

[0279] Furthermore, the encoding device and the decoding device encode or decode two or more different spaces or GOSs in parallel.

[0280] Furthermore, the encoding device and the decoding device encode or decode the space information (coordinates, size, etc.) of the space or the GOS.

[0281] Furthermore, the encoding device and the decoding device encode or decode the space or GOS included in the specific space specified based on external information related to the own position and / or area size, such as GPS, route information, or magnification.

[0282] The encoding device or decoding device performs encoding or decoding by giving a lower priority to a space far from the own position than to a space close to the own position.

[0283] The encoding device sets a direction in the world space according to a magnification or application, and encodes a GOS having a layer structure in that direction. Furthermore, the decoding device decodes the GOS having a layer structure in the world space direction set according to the magnification or application, preferentially starting from the lower layer.

[0284] The encoding device varies the accuracy of feature point extraction and object recognition, the size of the spatial area, etc. in indoor and outdoor spaces. However, the encoding device and decoding device encode or decode indoor GOS and outdoor GOS with close coordinates adjacent to each other in world space, and also encode or decode these identifiers in correspondence.

[0285] (Implementation Method 2)

[0286] When using point cloud encoded data in actual devices or services, it is desirable to transmit and receive only the required information according to the application in order to reduce network bandwidth. However, existing 3D data encoding structures do not have this capability, and therefore no corresponding encoding method exists.

[0287] What will be described in this embodiment is a three-dimensional data encoding method and a three-dimensional data encoding device for providing the function of sending and receiving required information according to the purpose in the encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data.

[0288] A voxel (VXL) having a certain feature value or more is defined as a feature voxel (FVXL), and a world space (WLD) formed by the FVXL is defined as a sparse world space (SWLD). Figure 11This diagram shows an example of a sparse world space and its configuration. SWLD includes: FGOS, a GOS constructed from FVXL; FSPC, an SPC constructed from FVXL; and FVLM, a VLM constructed from FVXL. The data structures and prediction structures of FGOS, FSPC, and FVLM can be the same as those of GOS, SPC, and VLM.

[0289] Feature quantities represent the three-dimensional positional information of the VXL or visible light information at the VXL position. In particular, feature quantities are more likely to be detected at corners and edges of three-dimensional objects. Specifically, these feature quantities may be three-dimensional feature quantities or visible light feature quantities as described below, but any feature quantity may be used as long as it represents the position, brightness, or color information of the VXL.

[0290] As the three-dimensional feature, a SHOT feature (Signature of Histograms of Orientations), a PFH feature (Point Feature Histograms), or a PPF feature (Point Pair Feature) is used.

[0291] SHOT features are obtained by segmenting the area around the VXL, calculating the inner product between the reference point and the normal vector of the segmented area, and then histizing the result. These features are characterized by high dimensionality and high expressiveness.

[0292] PFH features are obtained by selecting multiple pairs of points near VXL, calculating normal vectors and other parameters from these two points, and then histogramming them. Since these PFH features are histogram features, they are robust to small amounts of noise and have high expressiveness.

[0293] The PPF feature is calculated based on the VXL of two points using the normal vector, etc. Since all VXLs are used in this PPF feature, it is insensitive to occlusion.

[0294] Furthermore, as feature quantities of visible light, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients) that utilize information such as brightness gradient information of an image can be used.

[0295] SWLD is generated by calculating the above-mentioned feature quantity from each VXL of WLD and extracting FVXL. Here, SWLD can be updated every time WLD is updated, or it can be updated regularly after a certain period of time regardless of the update timing of WLD.

[0296] SWLDs can be generated for each feature. For example, as shown in SWLD1 based on SHOT features and SWLD2 based on SIFT features, SWLDs can be generated for each feature and used according to the application. Furthermore, the calculated feature values for each FVXL can be stored as feature value information in each FVXL.

[0297] Next, the method of using the sparse world space (SWLD) will be described. Since the SWLD only contains feature voxels (FVXL), the data size is generally smaller than that of the WLD which includes all VXLs.

[0298] In applications that utilize feature quantities to achieve a specific purpose, using SWLD information instead of WLD can reduce the time required to read data from the hard disk and reduce the bandwidth and transmission time during network transmission. For example, by pre-storing WLD and SWLD as map information on a server, switching the transmitted map information to WLD or SWLD based on client requests can reduce network bandwidth and transmission time. A specific example is shown below.

[0299] Figure 12 as well as Figure 13 FIG is a diagram showing an example of using SWLD and WLD. Figure 12 As shown, when client 1, which is an in-vehicle device, needs map information for determining its own position, client 1 sends a request to the server to obtain map data for estimating its own position (S301). The server sends SWLD to client 1 in accordance with the acquisition request (S302). Client 1 uses the received SWLD to determine its own position (S303). At this time, client 1 obtains VXL information of the surrounding area of client 1 through various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras, and estimates its own position information based on the obtained VXL information and SWLD. Here, the own position information includes the three-dimensional position information and orientation of client 1.

[0300] like Figure 13As shown, when client 2, an in-vehicle device, requires map information for mapping purposes such as three-dimensional maps, client 2 sends a request to the server to obtain map data for mapping (S311). In response to this request, the server sends a WLD to client 2 (S312). Client 2 uses the received WLD to perform map rendering (S313). At this point, client 2 creates a conceptual image using, for example, images captured by a visible light camera and the WLD received from the server, and then renders the created image on a screen such as a car navigation system.

[0301] As described above, the server sends SWLD to the client for applications that primarily require the characteristic values of each VXL, such as estimating its own position, and sends WLD to the client when detailed VXL information is required, such as for map drawing. This enables efficient transmission and reception of map data.

[0302] In addition, the client can determine whether it needs SWLD or WLD and request the server to send SWLD or WLD. Moreover, the server can determine whether to send SWLD or WLD based on the status of the client or the network.

[0303] Next, a method for switching between transmission and reception of the sparse world space (SWLD) and the world space (WLD) will be described.

[0304] The reception of WLD or SWLD can be switched according to the network bandwidth. Figure 14 This figure shows an example of the operation in this case. For example, when a low-speed network capable of utilizing the network bandwidth, such as in an LTE (Long Term Evolution) environment, is used, the client accesses the server via the low-speed network (S321) and obtains the SWLD (map information) from the server (S322). Alternatively, when a high-speed network with sufficient network bandwidth, such as in a WiFi environment, is used, the client accesses the server via the high-speed network (S323) and obtains the WLD from the server (S324). This allows the client to obtain appropriate map information based on the client's network bandwidth.

[0305] Specifically, the client receives SWLD via LTE outdoors, and acquires WLD via WiFi when entering a facility or other indoor location. This allows the client to acquire more detailed map information indoors.

[0306] In this way, the client can request WLD or SWLD from the server according to the frequency band of the network it is using. Alternatively, the client can send information indicating the frequency band of the network it is using to the server, and the server can send appropriate data (WLD or SWLD) to the client based on this information. Alternatively, the server can determine the network bandwidth of the client and send appropriate data (WLD or SWLD) to the client.

[0307] Furthermore, the reception of WLD or SWLD can be switched according to the moving speed. Figure 15 is a diagram showing an example of actions in this case. For example, when the client is moving at high speed (S331), the client receives SWLD from the server (S332). In addition, when the client is moving at low speed (S333), the client receives WLD from the server (S334). Accordingly, the client can suppress the network bandwidth and obtain map information according to the speed. Specifically, when the client is traveling on a highway, by receiving SWLD with a small amount of data, the map information can be updated at a roughly appropriate speed. In addition, when the client is traveling on a general road, by receiving WLD, more detailed map information can be obtained.

[0308] In this way, the client can request a WLD or SWLD from the server based on its own moving speed. Alternatively, the client can send information indicating its own moving speed to the server, and the server can send appropriate data (WLD or SWLD) to the client based on this information. Alternatively, the server can determine the client's moving speed and send appropriate data (WLD or SWLD) to the client.

[0309] Alternatively, the client can first obtain the SWLD from the server and then obtain the WLD of important areas within it. For example, when acquiring map data, the client can first use the SWLD to obtain general map information, filter out areas with a high incidence of features such as buildings, signs, or people, and then obtain the WLD of these filtered areas. This allows the client to reduce the amount of data received from the server while still obtaining detailed information about the desired area.

[0310] Alternatively, the server can create a separate SWLD for each object based on the WLD, and the client can receive each SWLD based on its intended purpose. This can reduce network bandwidth. For example, the server can pre-identify a person or a car from the WLD and create a SWLD for the person and a SWLD for the car. If the client wants to obtain information about people around it, it receives the SWLD for the person; if it wants to obtain information about the car, it receives the SWLD for the car. Furthermore, the type of SWLD can be distinguished based on information (flag or type, etc.) attached to the header.

[0311] Next, the configuration and operation flow of the three-dimensional data encoding device (for example, a server) according to this embodiment will be described. Figure 16 This is a block diagram of the three-dimensional data encoding device 400 according to this embodiment. Figure 17 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device 400.

[0312] Figure 16 The illustrated three-dimensional data encoding device 400 encodes input three-dimensional data 411 to generate encoded three-dimensional data 413 and 414 as encoded streams. Encoded three-dimensional data 413 corresponds to the WLD, and encoded three-dimensional data 414 corresponds to the SWLD. The three-dimensional data encoding device 400 includes an acquisition unit 401, a coding region determination unit 402, an SWLD extraction unit 403, a WLD encoding unit 404, and a SWLD encoding unit 405.

[0313] like Figure 17 As shown, first, the acquisition unit 401 acquires input three-dimensional data 411 which is point cloud data in a three-dimensional space ( S401 ).

[0314] Next, the coding region determination unit 402 determines a spatial region to be coded based on the spatial region where the point cloud data exists ( S402 ).

[0315] Next, the SWLD extraction unit 403 defines the spatial region to be encoded as a WLD and calculates features from each VXL contained in the WLD. Furthermore, the SWLD extraction unit 403 extracts VXLs whose features exceed a predetermined threshold, defines these VXLs as FVXLs, and appends these FVXLs to the SWLD to generate extracted three-dimensional data 412 (S403). Specifically, extracted three-dimensional data 412 with features exceeding the threshold is extracted from the input three-dimensional data 411.

[0316] Next, the WLD encoder 404 encodes the input 3D data 411 corresponding to the WLD to generate encoded 3D data 413 corresponding to the WLD (S404). At this time, the WLD encoder 404 appends information to the header of the encoded 3D data 413 that identifies the encoded 3D data 413 as a stream containing the WLD.

[0317] Then, the SWLD encoding unit 405 encodes the extracted 3D data 412 corresponding to the SWLD to generate encoded 3D data 414 corresponding to the SWLD (S405). At this time, the SWLD encoding unit 405 adds information to the header of the encoded 3D data 414 to distinguish that the encoded 3D data 414 is a stream containing the SWLD.

[0318] Furthermore, the order of the process of generating the encoded three-dimensional data 413 and the process of generating the encoded three-dimensional data 414 may be reversed. Furthermore, part or all of the above processes may be executed in parallel.

[0319] The information assigned to the header of the encoded three-dimensional data 413 and 414 is defined as a parameter such as "world_type". In the case of world_type = 0, it indicates that the stream contains WLD, and in the case of world_type = 1, it indicates that the stream contains SWLD. When more categories are defined, the assigned value can be increased, such as world_type = 2. In addition, a specific flag can be included in one of the encoded three-dimensional data 413 and 414. For example, the encoded three-dimensional data 414 can be assigned a flag indicating that the stream contains SWLD. In this case, the decoding device can determine whether it is a stream containing WLD or a stream containing SWLD based on the presence or absence of the flag.

[0320] Furthermore, the encoding method used by the WLD encoding unit 404 when encoding the WLD may be different from the encoding method used by the SWLD encoding unit 405 when encoding the SWLD.

[0321] For example, since data is decimated in SWLD, the correlation with surrounding data may be lower than that in WLD. Therefore, in the encoding method for SWLD, inter-frame prediction is prioritized over intra-frame prediction and inter-frame prediction in comparison with the encoding method for WLD.

[0322] Furthermore, the encoding method used for SWLD and the encoding method used for WLD may differ in the representation of the three-dimensional position. For example, the three-dimensional position of FVXL may be represented by three-dimensional coordinates in FWLD, while the three-dimensional position may be represented by an octree (described later) in WLD, or vice versa.

[0323] Furthermore, the SWLD encoding unit 405 encodes the SWLD encoded three-dimensional data 414 so that the data size is smaller than the WLD encoded three-dimensional data 413. For example, as described above, the correlation between data in SWLD and WLD may be lower. This may reduce the encoding efficiency, and the data size of the encoded three-dimensional data 414 may be larger than the data size of the WLD encoded three-dimensional data 413. Therefore, if the data size of the obtained encoded three-dimensional data 414 is larger than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 re-encodes the data to generate the encoded three-dimensional data 414 with a reduced data size.

[0324] For example, the SWLD extraction unit 403 generates extracted three-dimensional data 412 again with a reduced number of extracted feature points, and the SWLD encoding unit 405 encodes the extracted three-dimensional data 412. Alternatively, the degree of quantization in the SWLD encoding unit 405 may be coarsened. For example, in the octree structure described later, the degree of quantization can be coarsened by rounding the data in the lowest layer.

[0325] Furthermore, if the data size of the SWLD encoded three-dimensional data 414 cannot be made smaller than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 may not generate the SWLD encoded three-dimensional data 414. Alternatively, the WLD encoded three-dimensional data 413 may be copied to the SWLD encoded three-dimensional data 414. In other words, the WLD encoded three-dimensional data 413 may be used as the SWLD encoded three-dimensional data 414.

[0326] Next, the configuration and operation flow of the three-dimensional data decoding device (eg, client) according to this embodiment will be described. Figure 18 This is a block diagram of a three-dimensional data decoding device 500 according to this embodiment. Figure 19 3D data decoding processing performed by the 3D data decoding apparatus 500 is shown in FIG.

[0327] Figure 18 The illustrated three-dimensional data decoding apparatus 500 generates decoded three-dimensional data 512 or 513 by decoding encoded three-dimensional data 511. Here, the encoded three-dimensional data 511 is, for example, the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding apparatus 400.

[0328] The three-dimensional data decoding device 500 includes an acquisition unit 501 , a header analysis unit 502 , a WLD decoding unit 503 , and a SWLD decoding unit 504 .

[0329] like Figure 19 As shown, first, the acquisition unit 501 obtains the encoded 3D data 511 (S501). Next, the header analysis unit 502 analyzes the header of the encoded 3D data 511 to determine whether the encoded 3D data 511 is a stream containing a WLD or a stream containing a SWLD (S502). For example, this determination is made by referring to the world_type parameter described above.

[0330] If the encoded 3D data 511 is a stream containing a WLD ("Yes" in S503), the WLD decoding unit 503 decodes the encoded 3D data 511 to generate decoded 3D WLD data 512 (S504). If the encoded 3D data 511 is a stream containing a SWLD ("No" in S503), the SWLD decoding unit 504 decodes the encoded 3D data 511 to generate decoded 3D SWLD data 513 (S505).

[0331] Furthermore, similar to the encoding device, the decoding method used by the WLD decoding unit 503 when decoding the WLD may be different from the decoding method used by the SWLD decoding unit 504 when decoding the SWLD. For example, in the decoding method for the SWLD, priority may be given to the inter-frame prediction of intra-frame prediction and inter-frame prediction over the decoding method for the WLD.

[0332] Furthermore, the three-dimensional position representation method can be different between the decoding method used for SWLD and the decoding method used for WLD. For example, the three-dimensional position of FVXL can be represented by three-dimensional coordinates in SWLD, while the three-dimensional position can be represented by an octree (described later) in WLD, or vice versa.

[0333] Next, octree representation as a method of representing three-dimensional positions will be described. VXL data included in three-dimensional data is converted into an octree structure and then encoded. Figure 20 This is a diagram showing an example of VXL of WLD. Figure 21 Yes Figure 20 The octree structure of WLD is shown in Figure 1. Figure 20 In the example shown, there are three VXLs 1 to 3 that are VXLs containing point groups (hereinafter referred to as valid VXLs). Figure 21 As shown, the octree structure consists of nodes and leaves. Each node has a maximum of 8 nodes or leaves. Each leaf has VXL information. Figure 21 Among the leaves shown, leaves 1, 2, and 3 represent Figure 20 VXL1, VXL2, and VXL3 shown.

[0334] Specifically, each node and leaf corresponds to a three-dimensional position. Figure 20 The block corresponding to node 1 is divided into 8 blocks. Among the 8 blocks, the blocks containing valid VXLs are set as nodes, and the remaining blocks are set as leaves. The block corresponding to the node is further divided into 8 nodes or leaves. This process is repeated as many times as the number of levels in the tree structure. Finally, all the blocks at the bottom level are set as leaves.

[0335] and, Figure 22 It means from Figure 20 The WLD diagram shows an example of SWLD generation. Figure 20 The feature extraction results of VXL1 and VXL2 are determined to be FVXL1 and FVXL2 and are included in SWLD. VXL3 is not determined to be FVXL and is therefore not included in SWLD. Figure 23 Yes Figure 22 The octree structure of SWLD is shown in Figure 1. Figure 23 In the octree structure shown, Figure 21 The leaf 3 shown, which corresponds to VXL3, is deleted. Figure 21 The node 3 shown does not have a valid VXL and is changed to a leaf. In this way, generally speaking, the number of leaves of SWLD is smaller than that of WLD, and the encoded three-dimensional data of SWLD is also smaller than that of WLD.

[0336] Modifications of this embodiment will be described below.

[0337] For example, when a client such as a vehicle-mounted device estimates its own position, it receives SWLD from the server, uses SWLD to estimate its own position, and performs obstacle detection. It then uses various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras to perform obstacle detection based on the three-dimensional information of the surrounding area obtained by itself.

[0338] Furthermore, it's generally difficult to include VXL data for flat areas in SWLD. Therefore, the server maintains a subsampled world space (SubWLD) that downsamples the WLD for stationary obstacle detection and can send both the SWLD and SubWLD to the client. This reduces network bandwidth while enabling client-side position estimation and obstacle detection.

[0339] Furthermore, when clients need to quickly render 3D map data, a grid-like structure is often more convenient. Therefore, the server can generate a grid based on the WLD and store it in advance as a grid world space (MWLD). For example, if the client requires a coarse 3D rendering, it receives the MWLD; if it requires a detailed 3D rendering, it receives the WLD. This helps reduce network bandwidth.

[0340] Furthermore, while the server sets the VXL with a feature value above a threshold value from each VXL as the FVXL, the FVXL can also be calculated using different methods. For example, if the server determines that the VXL, VLM, SPC, or GOS that constitute a signal or intersection are necessary for self-position estimation, driving assistance, or autonomous driving, they can be included in the SWLD as FVXL, FVLM, FSPC, or FGOS. Furthermore, this determination can be made manually. In addition, the FVXL, etc. obtained by the above method can be added to the FVXL, etc. set based on the feature value. That is, the SWLD extraction unit 403 can further extract data corresponding to objects with predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412.

[0341] Furthermore, labels different from feature values can be assigned to situations required for these purposes. The server can separately maintain FVXL, which is required for estimating its own position, such as signals or intersections, driving assistance, or autonomous driving, as a higher layer of SWLD (e.g., lane world space).

[0342] Furthermore, the server can also add attributes to the VXL within the WLD in random access units or specified units. Attributes include, for example, information indicating whether a location is required or not for estimating the location, or information indicating whether traffic information such as signals or intersections is important. Attributes can also include the correspondence between lane information (GDF: Geographic Data Files, etc.) and features (such as intersections or roads).

[0343] Furthermore, as a method of updating WLD or SWLD, the following method can be adopted.

[0344] Update information showing changes in people, construction, or street trees (trajectory orientation) is uploaded to the server as a point group or metadata. The server updates the WLD based on this upload, and then updates the SWLD using the updated WLD.

[0345] Furthermore, if the client detects a mismatch between the 3D information generated by itself when estimating its own position and the 3D information received from the server, it can send the generated 3D information to the server along with an update notification. In this case, the server uses the WLD to update the SWLD. If the SWLD is not updated, the server determines that the WLD itself is outdated.

[0346] Furthermore, while information distinguishing between WLD and SWLD is added to the coded stream header, if there are multiple world spaces, such as a grid world space or a lane world space, information distinguishing between them can be added to the header. Furthermore, if there are multiple SWLDs with different feature values, information distinguishing between them can also be added to the header.

[0347] Furthermore, while the SWLD consists of FVXLs, it can also include VXLs that are not identified as FVXLs. For example, the SWLD can include adjacent VXLs used to calculate the characteristics of the FVXLs. This allows the client to calculate the characteristics of the FVXLs upon receiving the SWLD, even if the FVXLs in the SWLD do not have feature information attached. Furthermore, the SWLD can include information that distinguishes each VXL as an FVXL or a VXL.

[0348] As described above, the three-dimensional data encoding device 400 extracts extracted three-dimensional data 412 (second three-dimensional data) whose feature value is above a threshold value from the input three-dimensional data 411 (first three-dimensional data), and generates encoded three-dimensional data 414 (first encoded three-dimensional data) by encoding the extracted three-dimensional data 412.

[0349] Based on this, the three-dimensional data encoding device 400 generates encoded three-dimensional data 414 by encoding data with a feature value greater than or equal to a threshold value. This reduces the amount of data compared to directly encoding the input three-dimensional data 411. Therefore, the three-dimensional data encoding device 400 can reduce the amount of data required for transmission.

[0350] Furthermore, the three-dimensional data encoding device 400 further encodes the input three-dimensional data 411 to generate encoded three-dimensional data 413 (second encoded three-dimensional data).

[0351] With this, the three-dimensional data encoding device 400 can selectively transmit the encoded three-dimensional data 413 and the encoded three-dimensional data 414 according to the intended use, for example.

[0352] Furthermore, the extracted three-dimensional data 412 is encoded using a first encoding method, and the input three-dimensional data 411 is encoded using a second encoding method that is different from the first encoding method.

[0353] Accordingly, the three-dimensional data encoding device 400 can adopt appropriate encoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0354] Furthermore, in the first encoding method, compared with the second encoding method, priority is given to inter-frame prediction among intra-frame prediction and inter-frame prediction.

[0355] According to this, the three-dimensional data encoding apparatus 400 can increase the priority of inter-frame prediction for the extracted three-dimensional data 412 where the correlation between adjacent data tends to be low.

[0356] Furthermore, the first encoding method and the second encoding method use different methods to represent three-dimensional positions. For example, the second encoding method represents three-dimensional positions using an octree, while the first encoding method represents three-dimensional positions using three-dimensional coordinates.

[0357] With this, the three-dimensional data encoding device 400 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data amounts (number of VXLs or FVXLs).

[0358] Furthermore, at least one of the coded 3D data 413 and 414 includes an identifier indicating whether the coded 3D data is obtained by encoding the input 3D data 411 or by encoding a portion of the input 3D data 411. In other words, the identifier indicates whether the coded 3D data is the coded 3D data 413 of the WLD or the coded 3D data 414 of the SWLD.

[0359] Based on this, the decoding device can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

[0360] Furthermore, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 so that the data amount of the encoded three-dimensional data 414 is smaller than the data amount of the encoded three-dimensional data 413 .

[0361] As a result, the three-dimensional data encoding device 400 can make the data amount of the encoded three-dimensional data 414 smaller than the data amount of the encoded three-dimensional data 413 .

[0362] Furthermore, the three-dimensional data encoding device 400 extracts data corresponding to objects having predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412. For example, the objects having predetermined attributes may be objects required for self-position estimation, driving assistance, or autonomous driving, such as signals or intersections.

[0363] In this way, the three-dimensional data encoding device 400 can generate the encoded three-dimensional data 414 including the data required by the decoding device.

[0364] Furthermore, the three-dimensional data encoding device 400 (server) further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the status of the client.

[0365] Thus, the three-dimensional data encoding device 400 can send appropriate data according to the status of the client.

[0366] Furthermore, the client status includes the client's communication status (eg, network bandwidth) or the client's moving speed.

[0367] Furthermore, the three-dimensional data encoding device 400 further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the client's request.

[0368] Thus, the three-dimensional data encoding device 400 can send appropriate data according to the client's request.

[0369] Furthermore, the three-dimensional data decoding device 500 according to this embodiment decodes the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400 .

[0370] Specifically, the three-dimensional data decoding apparatus 500 decodes the encoded three-dimensional data 414 obtained by encoding the extracted three-dimensional data 412 whose feature quantity extracted from the input three-dimensional data 411 is greater than or equal to a threshold value using a first decoding method. Furthermore, the three-dimensional data decoding apparatus 500 decodes the encoded three-dimensional data 413 obtained by encoding the input three-dimensional data 411 using a second decoding method that is different from the first decoding method.

[0371] Thus, the 3D data decoding device 500 can selectively receive the encoded 3D data 414 and 413, which are obtained by encoding data with feature quantities exceeding a threshold value, according to, for example, the intended use. This reduces the amount of data transmitted. Furthermore, the 3D data decoding device 500 can employ appropriate decoding methods for each of the input 3D data 411 and the extracted 3D data 412.

[0372] Furthermore, in the first decoding method, compared with the second decoding method, priority is given to inter-frame prediction among intra-frame prediction and inter-frame prediction.

[0373] According to this, the three-dimensional data decoding apparatus 500 can increase the priority of inter-frame prediction for extracting three-dimensional data in which the correlation between adjacent data tends to be low.

[0374] Furthermore, the first decoding method and the second decoding method use different methods to represent the three-dimensional position. For example, the second decoding method represents the three-dimensional position using an octree, while the first decoding method represents the three-dimensional position using three-dimensional coordinates.

[0375] Thus, the three-dimensional data decoding apparatus 500 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data amounts (number of VXLs or FVXLs).

[0376] Furthermore, at least one of the encoded 3D data 413 and 414 includes an identifier indicating whether the encoded 3D data is obtained by encoding the input 3D data 411 or by encoding a portion of the input 3D data 411. The 3D data decoding apparatus 500 identifies the encoded 3D data 413 and 414 by referring to the identifier.

[0377] Based on this, the three-dimensional data decoding apparatus 500 can easily determine whether the obtained encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414 .

[0378] Furthermore, the 3D data decoding device 500 notifies the server of the status of the client (the 3D data decoding device 500 ) and receives one of the encoded 3D data 413 and 414 transmitted from the server according to the status of the client.

[0379] Thus, the three-dimensional data decoding apparatus 500 can receive appropriate data according to the status of the client.

[0380] Furthermore, the client status includes the client's communication status (eg, network bandwidth) or the client's moving speed.

[0381] Furthermore, the three-dimensional data decoding apparatus 500 further requests one of the encoded three-dimensional data 413 and 414 from the server, and receives one of the encoded three-dimensional data 413 and 414 transmitted from the server in accordance with the request.

[0382] This allows the three-dimensional data decoding apparatus 500 to receive appropriate data corresponding to the intended use.

[0383] (Implementation 3)

[0384] In this embodiment, a method of transmitting and receiving three-dimensional data between vehicles will be described.

[0385] Figure 24 Schematic diagram showing the state of transmission and reception of three-dimensional data 607 between the own vehicle 600 and the surrounding vehicles 601 .

[0386] When acquiring three-dimensional data using sensors mounted on the vehicle 600 (e.g., distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras), obstacles such as surrounding vehicles 601 may result in areas within the sensor detection range 602 of the vehicle 600 where three-dimensional data cannot be generated (hereinafter referred to as occlusion areas 604). Furthermore, while the larger the area in which three-dimensional data is acquired, the higher the accuracy of autonomous operation, the sensor detection range of the vehicle 600 itself is limited.

[0387] The sensor detection range 602 of the host vehicle 600 includes an area 603 where three-dimensional data can be obtained and an obstruction area 604. The area where the host vehicle 600 intends to obtain three-dimensional data includes the sensor detection range 602 of the host vehicle 600 and other areas. Furthermore, the sensor detection range 605 of the neighboring vehicle 601 includes the obstruction area 604 and an area 606 not included in the sensor detection range 602 of the host vehicle 600.

[0388] Surrounding vehicles 601 transmit information detected by surrounding vehicles 601 to own vehicle 600. By acquiring information detected by surrounding vehicles 601, such as vehicles traveling ahead, own vehicle 600 can obtain three-dimensional data 607 for an obstructed area 604 and an area 606 outside the sensor detection range 602 of own vehicle 600. Using the information acquired from surrounding vehicles 601, own vehicle 600 supplements the three-dimensional data for obstructed area 604 and area 606 outside the sensor detection range.

[0389] The three-dimensional data used in autonomous vehicle or robot operations can be used to estimate the vehicle's position, detect surrounding conditions, or both. For example, to estimate the vehicle's position, three-dimensional data generated by the vehicle 600 based on sensor information from the vehicle 600 is used. To detect surrounding conditions, three-dimensional data obtained from surrounding vehicles 601 is used in addition to the three-dimensional data generated by the vehicle 600.

[0390] The surrounding vehicles 601 for transmitting three-dimensional data 607 to the vehicle 600 can be determined based on the state of the vehicle 600. For example, the surrounding vehicles 601 may be the vehicle in front of the vehicle 600 when the vehicle 600 is traveling straight, the oncoming vehicle when the vehicle 600 is turning right, or the vehicle behind the vehicle when the vehicle 600 is backing up. Alternatively, the driver of the vehicle 600 may directly designate the surrounding vehicles 601 for transmitting three-dimensional data 607 to the vehicle 600.

[0391] Furthermore, the vehicle 600 may search for surrounding vehicles 601 that hold 3D data for areas that the vehicle 600 cannot obtain, and that are within the space for which the 3D data 607 is to be obtained. Areas that the vehicle 600 cannot obtain include, for example, an occlusion area 604 and an area 606 outside the sensor detection range 602.

[0392] Furthermore, the vehicle 600 may determine the blocked area 604 based on sensor information of the vehicle 600. For example, the vehicle 600 may determine the blocked area 604 as an area within the sensor detection range 602 of the vehicle 600 where three-dimensional data cannot be generated.

[0393] The following describes an example of operation in the case where the vehicle transmitting the three-dimensional data 607 is the vehicle traveling ahead. Figure 25 FIG. 1 is a diagram showing an example of three-dimensional data to be transmitted in this case.

[0394] like Figure 25 As shown, the three-dimensional data 607 transmitted from the preceding vehicle is, for example, a sparse world space (SWLD) point cloud. Specifically, the preceding vehicle generates three-dimensional WLD data (point cloud) based on information detected by its sensors. The preceding vehicle generates the SWLD three-dimensional data (point cloud) by extracting data with feature quantities exceeding a threshold from the WLD three-dimensional data. The preceding vehicle then transmits the generated SWLD three-dimensional data to the vehicle 600.

[0395] The own vehicle 600 receives the SWLD and merges the received SWLD into the point cloud created by the own vehicle 600 .

[0396] The transmitted SWLD includes information on absolute coordinates (the position of the SWLD in the coordinate system of the three-dimensional map), and the vehicle 600 can perform a merging process by superimposing the point cloud generated by the vehicle 600 based on the absolute coordinates.

[0397] The SWLD transmitted from the surrounding vehicle 601 may be the SWLD of an area 606 outside the sensor detection range 602 of the own vehicle 600 and within the sensor detection range 605 of the surrounding vehicle 601, or the SWLD of an obstruction area 604 relative to the own vehicle 600, or both. Furthermore, the transmitted SWLD may be the SWLD of an area used by the surrounding vehicle 601 for detecting surrounding conditions, among the aforementioned SWLDs.

[0398] Furthermore, the neighboring vehicle 601 can vary the density of the transmitted point cloud according to the communication time based on the speed difference between the own vehicle 600 and the neighboring vehicle 601. For example, when the speed difference is large and the communication time is short, the neighboring vehicle 601 can reduce the point cloud density (data volume) by extracting three-dimensional points with large feature values from the SWLD.

[0399] Furthermore, the detection of surrounding conditions refers to determining the presence of people, vehicles, road construction equipment, etc., identifying their types, and detecting their positions, moving directions, and moving speeds.

[0400] Furthermore, the own vehicle 600 may obtain the braking information of the surrounding vehicles 601 instead of the three-dimensional data 607 generated by the surrounding vehicles 601, or in addition to the three-dimensional data 607. Here, the braking information of the surrounding vehicles 601 is, for example, information indicating whether the accelerator or brake of the surrounding vehicles 601 is depressed, or the degree to which the accelerator or brake is depressed.

[0401] Furthermore, the point clouds generated by each vehicle are segmented into random access units, taking into account low-latency communication between vehicles. Furthermore, map data downloaded from the server, such as 3D maps, is segmented into larger random access units than in the case of inter-vehicle communication.

[0402] Data in areas that are prone to being blocked, such as an area in front of a preceding vehicle or an area behind a following vehicle, is divided into small random access units as data for low latency.

[0403] When traveling at high speed, the importance of the front becomes higher, so each vehicle generates SWLD in a narrow viewing angle range in small random access units when traveling at high speed.

[0404] When the SWLD created for transmission by the preceding vehicle includes an area where a point cloud can be obtained by the own vehicle 600, the preceding vehicle can reduce the transmission amount by removing the point cloud in this area.

[0405] Next, the configuration and operation of the three-dimensional data creation device 620 as the three-dimensional data receiving device according to this embodiment will be described.

[0406] Figure 26 This is a block diagram of a three-dimensional data creation device 620 according to this embodiment. This three-dimensional data creation device 620 is included in, for example, the aforementioned vehicle 600 and creates denser third three-dimensional data 636 by combining received second three-dimensional data 635 with first three-dimensional data 632 created by the three-dimensional data creation device 620.

[0407] The three-dimensional data creation device 620 includes a three-dimensional data creation unit 621 , a request range determination unit 622 , a search unit 623 , a reception unit 624 , a decoding unit 625 , and a synthesis unit 626 . Figure 27 This is a flowchart showing the operation of the three-dimensional data creation device 620.

[0408] First, the three-dimensional data creation unit 621 creates first three-dimensional data 632 (S621) using sensor information 631 detected by sensors included in the vehicle 600. Next, the request range determination unit 622 determines a request range, which is the three-dimensional spatial range where the data in the created first three-dimensional data 632 is insufficient (S622).

[0409] Next, the search unit 623 searches for surrounding vehicles 601 that hold three-dimensional data for the requested range and sends request range information 633 indicating the requested range to the surrounding vehicles 601 identified through the search (S623). Next, the receiving unit 624 receives the encoded three-dimensional data 634 as a coded stream for the requested range from the surrounding vehicles 601 (S624). Alternatively, the search unit 623 can indiscriminately issue requests to all vehicles within the identified range and receive the encoded three-dimensional data 634 from any responding vehicles. Furthermore, the search unit 623 is not limited to vehicles and can also issue requests to objects such as traffic lights and signs and receive the encoded three-dimensional data 634 from those objects.

[0410] Next, the decoding unit 625 decodes the received encoded 3D data 634 to obtain second 3D data 635 (S625). Next, the synthesizing unit 626 synthesizes the first 3D data 632 and the second 3D data 635 to create denser third 3D data 636 (S626).

[0411] Next, the configuration and operation of the three-dimensional data transmitting device 640 according to this embodiment will be described. Figure 28 3D data transmitting apparatus 640 is a block diagram.

[0412] The three-dimensional data sending device 640 is included in the above-mentioned surrounding vehicle 601, for example, and processes the fifth three-dimensional data 652 produced by the surrounding vehicle 601 into the sixth three-dimensional data 654 requested by the own vehicle 600, and generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654, and sends the encoded three-dimensional data 634 to the own vehicle 600.

[0413] The three-dimensional data transmitting device 640 includes a three-dimensional data creating unit 641 , a receiving unit 642 , an extracting unit 643 , an encoding unit 644 , and a transmitting unit 645 . Figure 29 3D data transmission device 640 is an operation flowchart.

[0414] First, the three-dimensional data creation unit 641 creates fifth three-dimensional data 652 ( S641 ) using sensor information 651 detected by sensors included in the surrounding vehicles 601. Next, the receiving unit 642 receives the requested range information 633 transmitted from the own vehicle 600 ( S642 ).

[0415] Next, the extraction unit 643 extracts the three-dimensional data within the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652, processing the fifth three-dimensional data 652 into sixth three-dimensional data 654 (S643). The encoding unit 644 then encodes the sixth three-dimensional data 654 to generate encoded three-dimensional data 634 as an encoded stream (S644). The transmission unit 645 then transmits the encoded three-dimensional data 634 to the vehicle 600 (S645).

[0416] In addition, although the example in which the own vehicle 600 includes the 3D data creation device 620 and the surrounding vehicle 601 includes the 3D data transmission device 640 is described here, each vehicle may also have the functions of the 3D data creation device 620 and the 3D data transmission device 640.

[0417] Hereinafter, the configuration and operation of the three-dimensional data creation device 620 in the case of being a surrounding situation detection device that realizes a process of detecting the surrounding situation of the own vehicle 600 will be described. Figure 30 3D data creation device 620A is a block diagram showing the configuration of the 3D data creation device 620A in this case. Figure 30 The three-dimensional data production device 620A shown in FIG. Figure 26 In addition to the configuration of the three-dimensional data creation device 620 shown, the three-dimensional data creation device 620A further includes a detection area determination unit 627 , a surrounding situation detection unit 628 , and an autonomous operation control unit 629 .

[0418] Figure 31 This is a flowchart of the peripheral condition detection process of the vehicle 600 performed by the three-dimensional data creation device 620A.

[0419] First, the three-dimensional data creation unit 621 creates first three-dimensional data 632 as a point cloud using sensor information 631 of the detection range of the vehicle 600 detected by sensors included in the vehicle 600 (S661). Furthermore, the three-dimensional data creation device 620A can also use the sensor information 631 to estimate its own position.

[0420] Next, the detection area determination unit 627 determines a detection target range as a spatial area for detecting surrounding conditions (S662). For example, the detection area determination unit 627 calculates the area required for detecting surrounding conditions for safe autonomous operation based on the autonomous operation (autonomous driving) conditions such as the driving direction and speed of the vehicle 600, and determines this area as the detection target range.

[0421] Next, the request range determination unit 622 determines the blocked area 604 and a spatial area that is outside the detection range of the sensor of the own vehicle 600 but is necessary for peripheral condition detection as the request range ( S663 ).

[0422] If the request range determined in step S663 exists ("Yes" in S664), the search unit 623 searches for surrounding vehicles that have information related to the request range. For example, the search unit 623 can inquire about the surrounding vehicles whether they have information related to the request range, and determine whether the surrounding vehicles have information related to the request range based on the request range and the position of the surrounding vehicles. Next, the search unit 623 sends a request signal 637 for requesting the transmission of three-dimensional data to the surrounding vehicles 601 determined by the search. In addition, after receiving the permission signal sent from the surrounding vehicle 601 indicating acceptance of the request of the request signal 637, the search unit 623 sends the request range information 633 indicating the request range to the surrounding vehicles 601 (S665).

[0423] Next, the receiving unit 624 detects a transmission notification of the transmission data 638 as information on the request range, and receives the transmission data 638 ( S666 ).

[0424] Alternatively, the three-dimensional data creation device 620A may issue a request to all vehicles within a specified range without discriminating, rather than searching for a destination to send a request to, and receive data 638 from any vehicle that responds with information related to the requested range. Furthermore, the search unit 623 is not limited to vehicles; it may also issue a request to an object such as a traffic light or sign and receive data 638 from that object.

[0425] Furthermore, the transmitted data 638 includes at least one of the encoded three-dimensional data 634 generated by the surrounding vehicle 601 and obtained by encoding the three-dimensional data of the requested range, and the surrounding condition detection results 639 of the requested range. The surrounding condition detection results 639 indicate the position, movement direction, and movement speed of people and vehicles detected by the surrounding vehicle 601. Furthermore, the transmitted data 638 may also include information indicating the position and movement of the surrounding vehicle 601. For example, the transmitted data 638 may include braking information of the surrounding vehicle 601.

[0426] If the received transmission data 638 includes the encoded three-dimensional data 634 ("Yes" in S667), the decoder 625 decodes the encoded three-dimensional data 634 to obtain the second three-dimensional SWLD data 635 (S668). In other words, the second three-dimensional data 635 is three-dimensional data (SWLD) generated by extracting data having a feature value greater than a threshold value from the fourth three-dimensional data (WLD).

[0427] Next, the first three-dimensional data 632 and the second three-dimensional data 635 are synthesized by the synthesizing unit 626 to generate third three-dimensional data 636 ( S669 ).

[0428] Next, the surrounding condition detection unit 628 detects the surrounding conditions of the own vehicle 600 using the third three-dimensional data 636, which is the point cloud of the spatial region required for surrounding condition detection (S670). Furthermore, if the received transmission data 638 includes the surrounding condition detection results 639, the surrounding condition detection unit 628 uses the surrounding condition detection results 639 in addition to the third three-dimensional data 636 to detect the surrounding conditions of the own vehicle 600. Furthermore, if the received transmission data 638 includes braking information of the surrounding vehicle 601, the surrounding condition detection unit 628 uses this braking information in addition to the third three-dimensional data 636 to detect the surrounding conditions of the own vehicle 600.

[0429] Next, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the vehicle 600 (S671) based on the peripheral condition detection result by the peripheral condition detection unit 628. In addition, the peripheral condition detection result can be presented to the driver through a UI (user interface) or the like.

[0430] If the requested range does not exist in step S663 ("No" in S664), that is, if all the spatial area information required for peripheral condition detection has been generated based on the sensor information 631, the peripheral condition detection unit 628 detects the peripheral conditions of the host vehicle 600 using the first three-dimensional data 632, which is the point cloud of the spatial area required for peripheral condition detection (S672). Then, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the host vehicle 600 based on the peripheral condition detection results of the peripheral condition detection unit 628 (S671).

[0431] If the received transmission data 638 does not include the encoded three-dimensional data 634 (No in S667), that is, if the transmission data 638 only includes the surrounding condition detection result 639 or braking information of the surrounding vehicle 601, the surrounding condition detection unit 628 detects the surrounding conditions of the own vehicle 600 using the first three-dimensional data 632 and the surrounding condition detection result 639 or braking information (S673). The autonomous operation control unit 629 then controls the autonomous operation (autonomous driving) of the own vehicle 600 based on the surrounding condition detection results of the surrounding condition detection unit 628 (S671).

[0432] Next, the three-dimensional data transmitting device 640A that transmits the transmission data 638 to the three-dimensional data creating device 620A described above will be described. Figure 32 FIG. 6 is a block diagram of the three-dimensional data transmitting device 640A.

[0433] Figure 32 The three-dimensional data transmitting device 640A shown in FIG. Figure 28 In addition to the configuration of the three-dimensional data transmitting device 640 shown, the device further includes a transmission availability determination unit 646. The three-dimensional data transmitting device 640A is included in the surrounding vehicle 601.

[0434] Figure 33 3D data transmission device 640A is a flowchart showing an example of its operation. First, the 3D data creation unit 641 creates fifth 3D data 652 using sensor information 651 detected by sensors included in the surrounding vehicles 601 (S681).

[0435] Next, the receiving unit 642 receives a delegation signal 637 for requesting the transmission of three-dimensional data from the own vehicle 600 (S682). Next, the transmission feasibility determination unit 646 decides whether to respond to the delegation indicated by the delegation signal 637 (S683). For example, the transmission feasibility determination unit 646 decides whether to respond to the delegation based on the content pre-set by the user. Alternatively, the receiving unit 642 may first accept a request from the other party such as a request range, and the transmission feasibility determination unit 646 may decide whether to respond to the delegation based on the content. For example, the transmission feasibility determination unit 646 may decide to respond to the delegation when it holds three-dimensional data within the requested range, and may decide not to respond to the delegation when it does not have three-dimensional data within the requested range.

[0436] If the request is accepted ("Yes" in S683), the three-dimensional data transmitting device 640A transmits a permission signal to the vehicle 600, and the receiving unit 642 receives the requested range information 633 indicating the requested range (S684). Next, the extraction unit 643 extracts the point cloud of the requested range from the fifth three-dimensional data 652, which is a point cloud, and creates the transmission data 638 containing the SWLD of the extracted point cloud, namely, the sixth three-dimensional data 654 (S685).

[0437] Specifically, the three-dimensional data transmitting device 640A may generate seventh three-dimensional data (WLD) based on the sensor information 651 and extract data having a feature value greater than a threshold value from the seventh three-dimensional data (WLD) to generate fifth three-dimensional data 652 (SWLD). Furthermore, the three-dimensional data generating unit 641 may pre-generate three-dimensional data for the SWLD, and the extracting unit 643 may extract three-dimensional data for the SWLD within the requested range from the three-dimensional data for the SWLD. Alternatively, the extracting unit 643 may generate three-dimensional data for the SWLD within the requested range based on the three-dimensional data for the WLD within the requested range.

[0438] Furthermore, the transmitted data 638 may include the surrounding condition detection result 639 of the requested range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601. Furthermore, the transmitted data 638 may not include the sixth three-dimensional data 654, but may include only at least one of the surrounding condition detection result 639 of the requested range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601.

[0439] When the transmission data 638 includes the sixth three-dimensional data 654 (Yes in S686 ), the encoding unit 644 encodes the sixth three-dimensional data 654 to generate the encoded three-dimensional data 634 ( S687 ).

[0440] Then, the transmitting unit 645 transmits the transmission data 638 including the encoded three-dimensional data 634 to the own vehicle 600 ( S688 ).

[0441] Furthermore, in the case where the sending data 638 does not include the sixth three-dimensional data 654 ("No" in S686), the sending unit 645 will send the sending data 638 including at least one of the surrounding condition detection results 639 of the requested range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601 to the own vehicle 600 (S688).

[0442] Modifications of this embodiment will be described below.

[0443] For example, the information transmitted from neighboring vehicle 601 may not be the three-dimensional data or surrounding condition detection results generated by the neighboring vehicle, but rather accurate feature point information of neighboring vehicle 601 itself. Own vehicle 600 uses this feature point information of neighboring vehicle 601 to correct the feature point information of the preceding vehicle within the point cloud obtained by own vehicle 600. This improves the matching accuracy of own vehicle 600's position estimation.

[0444] Furthermore, the feature point information of the preceding vehicle is three-dimensional point information composed of, for example, color information and coordinate information. Therefore, even if the sensor of the own vehicle 600 is a laser sensor or a stereo camera, the feature point information of the preceding vehicle can be used regardless of its type.

[0445] Furthermore, vehicle 600 is not limited by transmission time and can use the SWLD point cloud to calculate the accuracy of its own position estimate. For example, if vehicle 600's sensor is a stereo camera or other imaging device, it detects two-dimensional points in the image captured by the camera and uses these two-dimensional points to estimate its own position. Furthermore, while estimating its own position, vehicle 600 creates a point cloud of surrounding objects. Vehicle 600 reprojects the SWLD three-dimensional points onto the two-dimensional image and evaluates the accuracy of its own position estimate based on the error between the detected points on the two-dimensional image and the reprojected points.

[0446] In addition, when the sensor of the own vehicle 600 is a laser sensor such as LIDAR, the own vehicle 600 evaluates the accuracy of its own position estimation based on the error calculated by the SWLD of the produced point cloud and the SWLD of the three-dimensional map through the iterative closest point algorithm (Iterative Closest Point).

[0447] Furthermore, when the communication status via a base station or server such as 5G is poor, the own vehicle 600 can obtain a three-dimensional map from the surrounding vehicles 601.

[0448] Furthermore, vehicle-to-vehicle communication can be used to obtain distant information that cannot be obtained from vehicles surrounding the vehicle 600. For example, information about a recent traffic accident several hundred meters or kilometers ahead can be obtained by communicating with oncoming vehicles or by receiving information from surrounding vehicles. The data transmitted in this manner is meta-information above the dynamic three-dimensional map.

[0449] Furthermore, the detection results of the surrounding conditions and the information detected by the vehicle 600 can be presented to the user through a user interface, for example, by overlaying the navigation screen or the windshield.

[0450] Furthermore, it is also possible that, without supporting autonomous driving, a vehicle with cruise control can track a surrounding vehicle when it detects the surrounding vehicle traveling in autonomous driving mode.

[0451] Furthermore, when a three-dimensional map cannot be obtained or the own position cannot be estimated due to too many blocked areas, the own vehicle 600 can switch the operation mode from the automatic driving mode to the surrounding vehicle tracking mode.

[0452] Furthermore, the tracked vehicle can warn the user of the tracked situation, and a user interface can be provided in which the user can specify whether to allow tracking. At this point, advertisements can be displayed on the tracked vehicle, and payment rewards can be provided on the tracked side.

[0453] Furthermore, while the transmitted information is basically SWLD as three-dimensional data, it may also be information corresponding to a request setting set for the vehicle 600 or a public setting set for the preceding vehicle. For example, the transmitted information may be a WLD of a dense point cloud, the detection results of the surrounding conditions by the preceding vehicle, or the braking information of the preceding vehicle.

[0454] The vehicle 600 then receives the WLD, visualizes the 3D data from the WLD, and displays the visualized 3D data to the driver using a GUI. In this case, the vehicle 600 can use different colors to display information so that the user can distinguish between the point cloud generated by the vehicle 600 and the received point cloud.

[0455] In addition, when the own vehicle 600 uses the GUI to prompt the driver with the information detected by the own vehicle 600 and the detection results of the surrounding vehicles 601, the information is prompted by color differentiation, etc. in a manner that the user can distinguish the information detected by the own vehicle 600 and the received detection results.

[0456] As described above, in the three-dimensional data creation device 620 according to this embodiment, the three-dimensional data creation unit 621 creates first three-dimensional data 632 based on sensor information 631 detected by the sensor. The receiving unit 624 receives encoded three-dimensional data 634 obtained by encoding the second three-dimensional data 635. The decoding unit 625 decodes the received encoded three-dimensional data 634 to obtain the second three-dimensional data 635. The synthesis unit 626 synthesizes the first three-dimensional data 632 and the second three-dimensional data 635 to create third three-dimensional data 636.

[0457] Thus, the three-dimensional data creation device 620 can create detailed third three-dimensional data 636 using the created first three-dimensional data 632 and the received second three-dimensional data 635 .

[0458] Furthermore, the synthesis unit 626 can synthesize the first three-dimensional data 632 and the second three-dimensional data 635 to create third three-dimensional data 636 having a higher density than the first three-dimensional data 632 and the second three-dimensional data 635 .

[0459] Furthermore, the second three-dimensional data 635 (for example, SWLD) is three-dimensional data generated by extracting data having a feature value equal to or greater than a threshold value from the fourth three-dimensional data (for example, WLD).

[0460] As a result, the three-dimensional data creation device 620 can reduce the amount of three-dimensional data to be transmitted.

[0461] Furthermore, the three-dimensional data creation device 620 further includes a search unit 623 that searches for a transmission device that is a transmission source of the encoded three-dimensional data 634. The reception unit 624 receives the encoded three-dimensional data 634 from the searched transmission device.

[0462] With this, the three-dimensional data creation device 620 can identify the transmission device holding the required three-dimensional data by searching, for example.

[0463] The three-dimensional data creation device further includes a request range determination unit 622 that determines a request range, which is the range of the three-dimensional space for which three-dimensional data is requested. The search unit 623 transmits request range information 633 indicating the request range to the transmission device. The second three-dimensional data 635 includes the three-dimensional data within the request range.

[0464] As a result, the three-dimensional data creation device 620 can not only receive the required three-dimensional data but also reduce the amount of three-dimensional data to be transmitted.

[0465] Furthermore, the request range determination unit 622 determines a spatial range including the blocked area 604 that cannot be detected by the sensor as the request range.

[0466] Furthermore, in the three-dimensional data transmitting device 640 according to this embodiment, a three-dimensional data generating unit 641 generates fifth three-dimensional data 652 based on sensor information 651 detected by a sensor. An extracting unit 643 extracts a portion of the fifth three-dimensional data 652 to generate sixth three-dimensional data 654. An encoding unit 644 encodes the sixth three-dimensional data 654 to generate encoded three-dimensional data 634. A transmitting unit 645 transmits the encoded three-dimensional data 634.

[0467] As a result, the three-dimensional data transmitting device 640 can not only transmit the three-dimensional data produced by itself to other devices, but also reduce the amount of the transmitted three-dimensional data.

[0468] Furthermore, the three-dimensional data generating unit 641 generates seventh three-dimensional data (e.g., WLD) based on sensor information 651 detected by the sensor, and generates fifth three-dimensional data 652 (e.g., SWLD) by extracting data having a feature value greater than a threshold value from the seventh three-dimensional data.

[0469] Accordingly, the three-dimensional data transmitting apparatus 640 can reduce the amount of three-dimensional data to be transmitted.

[0470] The three-dimensional data transmitting device 640 further includes a receiving unit 642 that receives, from the receiving device, request range information 633 indicating a requested range, the range of the three-dimensional space for which three-dimensional data is requested. An extraction unit 643 extracts the three-dimensional data within the requested range from the fifth three-dimensional data 652 to create sixth three-dimensional data 654. A transmission unit 645 transmits the encoded three-dimensional data 634 to the receiving device.

[0471] Accordingly, the three-dimensional data transmitting apparatus 640 can reduce the amount of three-dimensional data to be transmitted.

[0472] (Implementation 4)

[0473] In this embodiment, an operation related to an abnormal situation in self-position estimation based on a three-dimensional map will be described.

[0474] Applications such as autonomous driving of vehicles and autonomous movement of mobile objects such as robots and drones are expected to expand in the future. One example of a method for achieving such autonomous movement is a method in which a mobile object estimates its own position within a three-dimensional map (self-position estimation) and drives according to the map.

[0475] Self-position estimation is achieved by matching the three-dimensional map with the three-dimensional information around the own vehicle obtained by sensors such as the rangefinder (LIDAR, etc.) or stereo camera installed on the own vehicle (hereinafter referred to as the own vehicle detection three-dimensional data), and estimating the own vehicle position within the three-dimensional map.

[0476] 3D maps, such as those offered by HERE's HD maps, are not just 3D point clouds; they also include 2D map data such as road and intersection shape information, as well as real-time information such as traffic jams and accidents. 3D maps are constructed from multiple layers of data, including 3D and 2D data and metadata that changes over time. Devices can retrieve only the data they need, or they can reference only the data they need.

[0477] The point cloud data may be the aforementioned SWLD, or may include point group data that is not a feature point. Furthermore, the transmission and reception of the point cloud data is basically performed in one or more random access units.

[0478] The following methods can be used to match a 3D map with the 3D data detected by the own vehicle. For example, the device compares the shapes of the point clusters in the respective point clouds and determines areas with high similarity between feature points as being at the same location. Furthermore, if the 3D map is constructed using SWLD, the device compares and matches the feature points that make up the SWLD with the 3D feature points extracted from the 3D data detected by the own vehicle.

[0479] To accurately estimate the vehicle's position, the following conditions (A) and (B) must be met: (A) a 3D map and 3D vehicle detection data must be available, and (B) their accuracy must meet a predetermined standard. However, in the following exceptional circumstances, either (A) or (B) may not be met.

[0480] (1) A three-dimensional map cannot be obtained through the communication path.

[0481] (2) There is no three-dimensional map, or the obtained three-dimensional map is damaged.

[0482] (3) The sensor of the own vehicle fails, or due to bad weather, the accuracy of the three-dimensional data generated by the own vehicle is insufficient.

[0483] The following describes the actions to deal with these abnormal situations. Although the following uses a vehicle as an example to explain its actions, the following method can also be applied to all moving objects that move autonomously, such as robots and drones.

[0484] The following describes the configuration and operation of the three-dimensional information processing device according to this embodiment for detecting abnormalities in three-dimensional data corresponding to a three-dimensional map or a vehicle. Figure 34 This is a block diagram showing a configuration example of a three-dimensional information processing device 700 according to this embodiment. Figure 35 3D information processing method performed by the 3D information processing apparatus 700 is a flowchart.

[0485] The three-dimensional information processing device 700 is mounted on a mobile object such as a motor vehicle. Figure 34 As shown, the three-dimensional information processing device 700 includes a three-dimensional map acquisition unit 701 , a vehicle detection data acquisition unit 702 , an abnormality determination unit 703 , a countermeasure action determination unit 704 , and an action control unit 705 .

[0486] The 3D information processing device 700 may also include a camera that captures 2D images, or a sensor (not shown) that uses ultrasonic or laser-based 1D data to detect structures or moving objects around the vehicle. Furthermore, the 3D information processing device 700 may include a communication unit (not shown) that generates a 3D map via a mobile communication network such as 4G or 5G, or between vehicles or between roads and vehicles.

[0487] like Figure 35 As shown, the three-dimensional map obtaining unit 701 obtains a three-dimensional map 711 near the driving route (S701). For example, the three-dimensional map obtaining unit 701 obtains the three-dimensional map 711 through a mobile communication network, vehicle-to-vehicle communication, or road-to-vehicle communication.

[0488] Next, the vehicle detection data acquisition unit 702 acquires vehicle detection three-dimensional data 712 based on the sensor information ( S702 ). For example, the vehicle detection data acquisition unit 702 generates vehicle detection three-dimensional data 712 based on sensor information acquired by sensors included in the vehicle.

[0489] Next, the abnormality determination unit 703 detects an abnormality by performing a predetermined check on at least one of the obtained three-dimensional map 711 and the vehicle's own three-dimensional detection data 712 (S703). In other words, the abnormality determination unit 703 determines whether at least one of the obtained three-dimensional map 711 and the vehicle's own three-dimensional detection data 712 is abnormal.

[0490] In step S703, if an abnormality is detected ("Yes" in S704), the response action determination unit 704 determines the response action for the abnormality (S705). Next, the action control unit 705 controls the operations of the various processing units required to implement the response action, such as the three-dimensional map acquisition unit 701 (S706).

[0491] If no abnormality is detected in step S703 (No in step S704 ), the three-dimensional information processing apparatus 700 ends the processing.

[0492] The 3D information processing device 700 estimates the vehicle's own position using the 3D map 711 and the own vehicle detection 3D data 712. The 3D information processing device 700 then uses the result of the own position estimation to enable the vehicle to autonomously drive.

[0493] Based on this, the three-dimensional information processing device 700 obtains map data (three-dimensional map 711) including the first three-dimensional position information via the channel. For example, the first three-dimensional position information is encoded in units of subspaces having three-dimensional coordinate information. The first three-dimensional position information includes multiple random access units, each of which is a collection of one or more subspaces and can be independently decoded. For example, the first three-dimensional position information is data (SWLD) encoded at feature points where the three-dimensional feature quantity exceeds a predetermined threshold.

[0494] The three-dimensional information processing device 700 generates second three-dimensional position information (vehicle detection three-dimensional data 712) based on the information detected by the sensor. The three-dimensional information processing device 700 then performs abnormality determination processing on the first three-dimensional position information or the second three-dimensional position information to determine whether the first three-dimensional position information or the second three-dimensional position information is abnormal.

[0495] When the three-dimensional information processing device 700 determines that the first three-dimensional position information or the second three-dimensional position information is abnormal, it determines a response action to the abnormality and then executes control required for executing the response action.

[0496] With this, the three-dimensional information processing device 700 can detect abnormality in the first three-dimensional position information or the second three-dimensional position information and perform a corresponding action.

[0497] The following describes the response to abnormal situation 1, that is, the case where the three-dimensional map 711 cannot be obtained through communication.

[0498] Three-dimensional map 711 is required for estimating the vehicle's own position. However, if the vehicle does not already have three-dimensional map 711 corresponding to the route to the destination, it must obtain three-dimensional map 711 through communication. However, due to channel congestion or poor radio reception, the vehicle may not be able to obtain three-dimensional map 711 along the route.

[0499] Abnormality determination unit 703 checks whether three-dimensional map 711 has been obtained for all sections on the route to the destination, or for sections within a predetermined range from the current location. If it cannot be obtained, it determines that abnormality 1 has occurred. Specifically, abnormality determination unit 703 determines whether three-dimensional map 711 (first three-dimensional position information) can be obtained via the channel. If it cannot be obtained via the channel, it determines that three-dimensional map 711 is abnormal.

[0500] When it is determined that the abnormal situation is 1, the response action determination unit 704 selects one of the following two response actions: (1) continuing the self-position estimation, and (2) stopping the self-position estimation.

[0501] First, (1) a specific example of the response action when the self-position estimation is continued will be described. When the self-position estimation is continued, a three-dimensional map 711 on the route to the destination is required.

[0502] For example, the vehicle determines a location within the range of the acquired three-dimensional map 711 where a channel can be used, moves to that location, and acquires the three-dimensional map 711. In this case, the vehicle can acquire all three-dimensional maps 711 up to the destination, or it can acquire three-dimensional maps 711 per random access unit within the maximum size that can be stored in the vehicle's memory, HDD, or other storage device.

[0503] Alternatively, the vehicle may obtain the communication status of the route. If the communication status is predicted to deteriorate along the route, the vehicle may obtain a three-dimensional map 711 for that section before reaching the section with poor communication status, or obtain a three-dimensional map 711 for the maximum range that can be obtained. In other words, the three-dimensional information processing device 700 predicts whether the vehicle will enter an area with poor communication status. If the three-dimensional information processing device 700 predicts that the vehicle will enter an area with poor communication status, the three-dimensional map 711 is obtained before the vehicle enters the area.

[0504] Alternatively, the vehicle may identify a random access unit of the three-dimensional map 711 that is narrower than normal and constitutes the minimum required for estimating its own position on the route, and receive the identified random access unit. In other words, if the three-dimensional information processing device 700 cannot obtain the three-dimensional map 711 (first three-dimensional position information) via the channel, it can obtain third three-dimensional position information with a narrower range than the first three-dimensional position information via the channel.

[0505] Furthermore, when a vehicle cannot access the distribution server of the three-dimensional map 711, it can obtain the three-dimensional map 711 from other vehicles or other mobile bodies traveling around its own vehicle. At this time, the other vehicles or other mobile bodies are mobile bodies that have already obtained the three-dimensional map 711 on the path to the destination and can communicate with its own vehicle.

[0506] Next, a specific example of the response action in the case of (2) stopping the own position estimation will be described. In this case, the three-dimensional map 711 on the route to the destination is not necessary.

[0507] For example, the vehicle notifies the driver that functions such as autonomous driving based on its own position estimation cannot continue to be executed, and the operation mode is shifted to a manual operation mode performed by the driver.

[0508] While the level of self-position estimation may differ depending on the presence of a human, autonomous driving is typically performed. Furthermore, the results of self-position estimation may also be used for navigation purposes, such as when a human is driving. Therefore, the results of self-position estimation are not necessarily used for autonomous driving.

[0509] Furthermore, when the vehicle cannot utilize commonly used channels such as mobile communication networks such as 4G or 5G, it is possible to confirm whether the three-dimensional map 711 can be obtained via other channel paths such as Wi-Fi (registered trademark) or millimeter wave communication between the road and the vehicle, or communication between vehicles, and the channel used can be switched to a channel that can obtain the three-dimensional map 711.

[0510] Furthermore, even if the vehicle cannot obtain three-dimensional map 711, it can obtain a two-dimensional map and continue autonomous driving using the two-dimensional map and the vehicle's own three-dimensional detection data 712. Specifically, even if the vehicle cannot obtain three-dimensional map 711 via a channel, the three-dimensional information processing device 700 can obtain map data (a two-dimensional map) including two-dimensional position information via the channel and estimate the vehicle's own position using the two-dimensional position information and the vehicle's own three-dimensional detection data 712.

[0511] Specifically, the vehicle uses the two-dimensional map and the vehicle's own three-dimensional detection data 712 to estimate its own position, and uses the vehicle's own three-dimensional detection data 712 to detect surrounding vehicles, pedestrians, and obstacles.

[0512] Here, map data such as HD maps can include not only 3D maps 711 composed of 3D point clouds, but also 2D map data (2D maps), simplified map data that extracts characteristic information such as road shapes and intersections from 2D map data, and metadata indicating real-time information such as traffic jams, accidents, and construction. For example, map data has a layered structure with 3D data (3D maps 711), 2D data (2D maps), and metadata arranged in order from the lower layer.

[0513] Here, two-dimensional data is smaller than three-dimensional data. Therefore, even in poor communication conditions, the vehicle can obtain a two-dimensional map. Furthermore, the vehicle can uniformly obtain a large-scale two-dimensional map in areas with good communication conditions. Therefore, if the vehicle has difficulty obtaining three-dimensional map 711 due to poor channel conditions, it can receive the layer containing the two-dimensional map instead of the three-dimensional map 711. Furthermore, because metadata is small in size, the vehicle can, for example, receive metadata constantly regardless of communication conditions.

[0514] Among the methods of estimating the own position using a two-dimensional map and the own vehicle detection three-dimensional data 712 , there are, for example, the following two methods.

[0515] The first method is a method of matching two-dimensional feature quantities. Specifically, the vehicle extracts two-dimensional feature quantities from the vehicle detection three-dimensional data 712 and matches the extracted two-dimensional feature quantities with the two-dimensional map.

[0516] For example, the vehicle projects its own vehicle detection three-dimensional data 712 onto the same plane as the two-dimensional map and matches the obtained two-dimensional data with the two-dimensional map. The matching is performed using two-dimensional image features extracted from both.

[0517] If 3D map 711 includes SWLD, it can simultaneously store 3D feature quantities for feature points within 3D space, as well as 2D feature quantities within the same plane as the 2D map. For example, identification information can be assigned to the 2D feature quantities. Alternatively, the 2D feature quantities can be stored in a separate layer from the 3D data and the 2D map, allowing the vehicle to obtain the 2D feature quantity data simultaneously with the 2D map.

[0518] When a two-dimensional map represents information on positions of white lines, guardrails, and buildings on the road at different heights from the ground (not in the same plane) on the same map, the vehicle extracts feature values from multiple height data of the three-dimensional data 712 detected by its own vehicle.

[0519] Furthermore, information indicating the correspondence between the feature points in the two-dimensional map and the feature points in the three-dimensional map 711 may be stored as meta-information of the map data.

[0520] The second method is to match three-dimensional feature quantities. Specifically, the vehicle obtains three-dimensional feature quantities corresponding to feature points in the two-dimensional map and matches the obtained three-dimensional feature quantities with the three-dimensional feature quantities of the vehicle's detected three-dimensional data 712.

[0521] Specifically, the three-dimensional feature values corresponding to the feature points of the two-dimensional map are stored in the map data. When the vehicle obtains the two-dimensional map, it also obtains the three-dimensional feature values. In addition, when the three-dimensional map 711 includes SWLD, by assigning information for identifying the feature points of the SWLD that correspond to the feature points of the two-dimensional map, the vehicle can determine the location corresponding to the two-dimensional map based on the identification information. Figure 1 In addition, in this case, since it is sufficient to express the two-dimensional position, the amount of data can be reduced compared to the case of expressing the three-dimensional position.

[0522] Furthermore, when estimating its own position using a two-dimensional map, the accuracy of the estimated own position is lower than that using a three-dimensional map 711. Therefore, the vehicle determines whether to continue autonomous driving even if the estimation accuracy decreases, and only continues autonomous driving if it determines that it can continue.

[0523] Whether autonomous driving can continue is influenced by factors such as whether the vehicle is traveling on an urban area or a highway or other road with few vehicles or pedestrians, as well as the driving environment, such as road width and road clutter (vehicle or pedestrian density). Furthermore, markers for sensors such as cameras to identify can be placed on business premises, on streets, or inside buildings. In these specific areas, the markers can be identified with high precision by two-dimensional sensors, for example, by including the marker's location information in a two-dimensional map, enabling highly accurate self-position estimation.

[0524] Furthermore, by including identification information indicating whether each area is a specific area within the map, the vehicle can determine whether it is within a specific area. If the vehicle is within the specific area, the vehicle determines to continue autonomous driving. In this way, the vehicle can determine whether to continue autonomous driving based on the accuracy of its own position estimation when using a two-dimensional map or the vehicle's driving environment.

[0525] In this way, the three-dimensional information processing device 700 can determine whether to perform automatic driving of the vehicle based on the vehicle's driving environment (the moving environment of the moving body), which is the result of estimating the vehicle's own position using a two-dimensional map and the vehicle's own detection three-dimensional data 712.

[0526] Furthermore, the vehicle may not determine whether to continue autonomous driving, but may instead switch the autonomous driving level (mode) based on the accuracy of its own position estimate or the vehicle's driving environment. Switching the autonomous driving level (mode) here may include, for example, limiting speed, increasing the amount of driver input (lowering the autonomous driving level), obtaining driving information from a preceding vehicle and switching the mode based on that information, or obtaining driving information from a vehicle set to the same destination and using that information to switch the autonomous driving mode.

[0527] Furthermore, the map can include information, associated with location information, indicating the recommended level of autonomous driving when estimating the vehicle's position using a two-dimensional map. The recommended level can be metadata that changes dynamically based on factors such as traffic volume. This eliminates the need for vehicles to determine the level based on the surrounding environment and can instead determine the level solely based on information within the map. Furthermore, by having multiple vehicles referencing the same map, the level of autonomous driving for each vehicle can be maintained consistently. Furthermore, the recommended level can be a mandatory level rather than a recommendation.

[0528] Furthermore, the vehicle can switch the level of autonomous driving depending on whether there is a driver (manned or unmanned). For example, the vehicle can reduce the level of autonomous driving when there is a driver and stop when there is no one. The vehicle determines where it can safely stop by identifying pedestrians, vehicles, and traffic signs in the surrounding area. Alternatively, the map may include location information showing where the vehicle can safely stop, and the vehicle can refer to this location information to determine where it can safely stop.

[0529] Next, the response to abnormal situation 2, that is, when the three-dimensional map 711 does not exist or the obtained three-dimensional map 711 is damaged, will be described.

[0530] Abnormality determination unit 703 determines which of the following situations (1) and (2) it falls under. If it falls under either of these situations, it determines that abnormality 2 is the situation where (1) three-dimensional map 711 for part or all of the sections on the route to the destination is not available on the distribution server that is the access destination and cannot be obtained, or (2) part or all of the obtained three-dimensional map 711 is damaged. In other words, abnormality determination unit 703 determines whether the data of three-dimensional map 711 is complete. If the data of three-dimensional map 711 is incomplete, it determines that three-dimensional map 711 is abnormal.

[0531] When it is determined to be abnormal situation 2, the following countermeasures are performed. First, an example of a countermeasure in the case where (1) the three-dimensional map 711 cannot be obtained is described.

[0532] For example, the vehicle sets a route that does not pass through a section without the three-dimensional map 711 .

[0533] If the vehicle cannot set an alternative route due to reasons such as the absence of an alternative route or the significant increase in distance required even if an alternative route exists, the vehicle sets a route that includes a section where the three-dimensional map 711 is not available. Furthermore, the vehicle notifies the driver of the need to switch driving modes in this section, thereby switching the driving mode to manual operation mode.

[0534] (2) When part or all of the acquired three-dimensional map 711 is destroyed, the following countermeasures are performed.

[0535] The vehicle identifies the damaged area in three-dimensional map 711, requests data about the damaged area through communication, obtains the data, and updates three-dimensional map 711 using the obtained data. The vehicle can specify the damaged area using positional information such as absolute or relative coordinates in three-dimensional map 711, or by index number of the random access unit that constitutes the damaged area. In this case, the vehicle replaces the random access unit containing the damaged area with the obtained random access unit.

[0536] Next, the response action to abnormal situation 3, that is, when the sensors of the own vehicle cannot generate the own vehicle detection three-dimensional data 712 due to a malfunction or bad weather, will be described.

[0537] The abnormality determination unit 703 checks whether the generation error of the self-vehicle detection three-dimensional data 712 is within the allowable range. If not, it determines that abnormality 3 has occurred. Specifically, the abnormality determination unit 703 determines whether the generation accuracy of the self-vehicle detection three-dimensional data 712 is above a reference value. If the generation accuracy of the self-vehicle detection three-dimensional data 712 is not above the reference value, the self-vehicle detection three-dimensional data 712 is determined to be abnormal.

[0538] As a method for confirming whether the generation error of the own vehicle detection three-dimensional data 712 is within the allowable range, the following method can be adopted.

[0539] The spatial resolution of the vehicle's detected three-dimensional data 712 during normal operation is predetermined based on the resolution in the depth and scanning directions of the vehicle's three-dimensional sensors, such as range finders and stereo cameras, or the density of the point cloud that can be generated. Furthermore, the vehicle obtains the spatial resolution of the three-dimensional map 711 based on metadata included in the three-dimensional map 711.

[0540] The vehicle uses the spatial resolution of both to estimate a baseline value for matching error when matching the vehicle's detected 3D data 712 with the 3D map 711 based on 3D feature quantities. Matching error can be achieved using statistics such as the error in the 3D feature quantity for each feature point, the average of the errors between multiple feature points, or the error in the spatial distance between multiple feature points. The allowable range for deviation from the baseline value is pre-set.

[0541] If the matching error between the vehicle detection three-dimensional data 712 generated before or during the vehicle starts traveling and the three-dimensional map 711 is not within the allowable range, it is determined to be abnormal situation 3.

[0542] Alternatively, the vehicle may utilize a test pattern having a known three-dimensional shape for accuracy checking to obtain its own vehicle detection three-dimensional data 712 for a test pattern such as before driving begins, and determine whether it is an abnormal situation 3 based on whether the shape error is within an allowable range.

[0543] For example, the vehicle performs the above determination each time before starting a journey. Alternatively, the vehicle performs the above determination at regular intervals while driving, thereby obtaining a time-series variation in the matching error. If the matching error shows a tendency to increase, the vehicle may determine that an abnormality is occurring, even if the error is within the allowable range. Furthermore, if the vehicle can predict an abnormality based on the time-series variation, the user may be notified of the predicted abnormality by displaying a message urging an inspection or repair. Furthermore, by distinguishing between abnormalities caused by temporary factors such as bad weather and abnormalities due to sensor failures based on the time-series variation, the user may be notified of only abnormalities due to sensor failures.

[0544] Furthermore, when the vehicle is judged to be in abnormal situation 3, any one of the following three response actions is selected or selectively executed: (1) activating an alternative sensor for emergency use (rescue mode), (2) switching the operating mode, and (3) performing action correction of the three-dimensional sensor.

[0545] First, (1) the case of operating an emergency substitute sensor will be described. The vehicle operates an emergency substitute sensor that is different from the three-dimensional sensor used during normal operation. Specifically, when the generation accuracy of the own vehicle detection three-dimensional data 712 is not above a reference value, the three-dimensional information processing device 700 generates own vehicle detection three-dimensional data 712 (fourth three-dimensional position information) based on information detected by the substitute sensor that is different from the normal sensor.

[0546] Specifically, when a vehicle uses multiple cameras or LIDARs to obtain 3D vehicle detection data 712, the vehicle identifies a malfunctioning sensor based on, for example, the direction in which the matching error in 3D vehicle detection data 712 exceeds the allowable range. The vehicle then activates a replacement sensor corresponding to the malfunctioning sensor.

[0547] The replacement sensor can be a three-dimensional sensor, a camera that obtains two-dimensional images, or a one-dimensional sensor such as ultrasound. If the replacement sensor is a sensor other than a three-dimensional sensor, the accuracy of the vehicle's position estimation may be reduced or even impossible. Therefore, the vehicle can switch the autonomous driving mode based on the type of replacement sensor.

[0548] For example, if the replacement sensor is a three-dimensional sensor, the vehicle continues in autonomous driving mode. Furthermore, if the replacement sensor is a two-dimensional sensor, the vehicle switches from fully autonomous driving to semi-autonomous driving mode, which requires human control. Furthermore, if the replacement sensor is a one-dimensional sensor, the vehicle switches to manual braking mode, which disables automatic braking control.

[0549] Furthermore, the vehicle can switch between autonomous driving modes depending on the driving environment. For example, if the vehicle uses a two-dimensional sensor instead of a 2D sensor, it can continue in fully autonomous driving mode on highways and switch to semi-autonomous driving mode in urban areas.

[0550] Furthermore, if the vehicle can continue to estimate its position without a replacement sensor and can obtain a sufficient number of feature points using only the sensors operating normally, it will be able to do so. However, since it cannot detect specific directions, the vehicle will switch to semi-autonomous driving or manual operation mode.

[0551] Next, (2) the response action for switching the operating mode is described. The vehicle switches the operating mode from the automatic driving mode to the manual operation mode. Alternatively, the vehicle may continue automatic driving until it reaches a place such as a roadside where it can stop safely, and then stop. Furthermore, the vehicle may switch the operating mode to the manual operation mode after stopping. In this way, the three-dimensional information processing device 700 switches the automatic driving mode when the generation accuracy of the vehicle detection three-dimensional data 712 is not above the reference value.

[0552] Next, (3) will describe the corresponding action for correcting the motion of the three-dimensional sensor. The vehicle determines the three-dimensional sensor with malfunction based on the direction in which the matching error occurs, and calibrates the determined sensor. Specifically, when multiple LIDARs or cameras are used as sensors, a portion of the three-dimensional space reconstructed by each sensor overlaps. That is, the data of the overlapping portion is obtained by multiple sensors. The three-dimensional point group data obtained for the overlapping portion is different between the normal sensor and the sensor with malfunction. Therefore, the vehicle performs origin correction of the LIDAR, or exposure and focus of the camera, etc., in such a way that the sensor with malfunction can obtain the same three-dimensional point group data as the normal sensor, and adjusts the motion of the predetermined part.

[0553] After adjustment, if the matching error is within the allowable range, the vehicle continues the previous operating mode. Otherwise, if the matching accuracy is not within the allowable range after adjustment, the vehicle performs the above-mentioned response actions (1) of using the emergency replacement sensor or (2) of switching the operating mode.

[0554] In this manner, the three-dimensional information processing device 700 performs sensor operation correction when the generation accuracy of the own vehicle detection three-dimensional data 712 is not equal to or greater than the reference value.

[0555] The following describes how to select a response action. The response action can be selected by a user such as the driver, or it can be selected automatically by the vehicle without the user's intervention.

[0556] Furthermore, the vehicle can also switch control depending on whether a driver is aboard. For example, if a driver is aboard, the vehicle prioritizes switching to manual operation mode. Otherwise, if a driver is not aboard, the vehicle prioritizes stopping at a safe location.

[0557] The information indicating the stopping place may be included as meta-information in the three-dimensional map 711. Alternatively, the vehicle may send a response request for the stopping place to a service that manages the operation information of the autonomous driving, thereby obtaining the information indicating the stopping place.

[0558] Furthermore, when the vehicle is operating on a prescribed route, for example, the vehicle's operating mode can be shifted to a mode where an operator manages the vehicle's operations via a communication channel. In particular, in vehicles operating in fully autonomous mode, the risk of a malfunction in the vehicle's position estimation function is high. Therefore, when a vehicle detects an abnormality, or if the detected abnormality cannot be corrected, it notifies the service managing operational information via a communication channel. This service can notify nearby vehicles of the presence of an abnormal vehicle or issue instructions to vacate nearby parking spaces.

[0559] Furthermore, when an abnormal situation is detected, the vehicle can travel slower than usual.

[0560] If a self-driving vehicle used in a taxi-like ride-sharing service experiences an abnormality, it will notify the operation management center and stop at a safe location. Alternatively, the ride-sharing service can dispatch a replacement vehicle. Alternatively, the user of the ride-sharing service can drive the vehicle. In these situations, discounts on fares or special points can be offered.

[0561] Furthermore, in the method for dealing with abnormal situation 1, a method of estimating the own position based on a two-dimensional map has been described. However, the own position can also be estimated using a two-dimensional map in normal situations. Figure 36 This is a flowchart of the self-position estimation process in this case.

[0562] First, the vehicle obtains a three-dimensional map 711 of the vicinity of the driving route (S711). Next, the vehicle obtains its own vehicle detection three-dimensional data 712 based on sensor information (S712).

[0563] Next, the vehicle determines whether a three-dimensional map 711 is necessary for estimating its own position (S713). Specifically, the vehicle determines whether a three-dimensional map 711 is necessary based on the accuracy of its own position estimation using a two-dimensional map and the driving environment. For example, the same method as described above for abnormal situation 1 is used.

[0564] If the vehicle determines that 3D map 711 is not necessary ("No" in S714), it obtains a 2D map (S715). At this point, the vehicle can also obtain the additional information described in the handling method for abnormal situation 1. Furthermore, the vehicle can generate a 2D map based on 3D map 711. For example, the vehicle can extract an arbitrary plane from 3D map 711 to generate a 2D map.

[0565] Next, the vehicle estimates its own position using the vehicle detection three-dimensional data 712 and the two-dimensional map (S716). The method for estimating the own position using the two-dimensional map is the same as that described in the method for dealing with abnormal situation 1 above.

[0566] If the vehicle determines that the three-dimensional map 711 is necessary (Yes in S714 ), the vehicle obtains the three-dimensional map 711 ( S717 ) and estimates its own position using the vehicle detection three-dimensional data 712 and the three-dimensional map 711 ( S718 ).

[0567] Furthermore, the vehicle can switch between primarily using a two-dimensional map and a three-dimensional map 711 based on the corresponding speed of its own communication device or the channel conditions. For example, while receiving three-dimensional map 711, the vehicle can pre-set the communication speed required for driving. If the communication speed is below the set value, the vehicle primarily uses the two-dimensional map. If the communication speed is greater than the set value, the vehicle primarily uses the three-dimensional map 711. Alternatively, the vehicle may not determine whether to switch between using a two-dimensional map and a three-dimensional map and primarily use the two-dimensional map.

[0568] (Implementation 5)

[0569] In this embodiment, a method of transmitting three-dimensional data to a following vehicle will be described. Figure 37 This figure shows an example of an object space of three-dimensional data transmitted to a following vehicle or the like.

[0570] Vehicle 801 sends three-dimensional data such as point cloud (point group) included in the rectangular space 802 with width W, height H and depth D in front of vehicle 801 at a distance L from vehicle 801 to traffic cloud monitoring or the rear vehicle that monitors road conditions at a time interval of Δt.

[0571] When the three-dimensional data included in the previously transmitted space 802 changes due to a vehicle or a person entering the space 802 from outside, the vehicle 801 also transmits the three-dimensional data of the changed space.

[0572] In addition, Figure 37 Although an example in which the space 802 is in the shape of a rectangular parallelepiped is shown, the space 802 does not necessarily need to be a rectangular parallelepiped as long as it includes a space on the road ahead that becomes a blind spot when viewed from the rear vehicle.

[0573] Distance L is preferably set to a distance at which the following vehicle, having received the three-dimensional data, can safely stop. For example, distance L is set as the sum of the following distances: the distance the following vehicle travels while receiving the three-dimensional data; the distance the following vehicle travels until it begins decelerating according to the received data; and the distance the following vehicle typically requires to safely stop. Since these distances vary with speed, distance L can be varied according to the vehicle's speed V, as shown by L = a × V + b (a and b are constants).

[0574] The width W is set to a value greater than at least the width of the lane in which the vehicle 801 travels. More preferably, the width W is set to a size that includes adjacent spaces such as left and right lanes or curbs.

[0575] While depth D can be a fixed value, it can also vary according to vehicle speed V, as shown by D = c × V + d (c and d are constants). Furthermore, by setting D such that D > V × Δt, the transmission space can overlap with the space that was previously transmitted. This allows vehicle 801 to more reliably transmit the space on the road to vehicles behind it, etc. without omission.

[0576] In this way, by limiting the three-dimensional data transmitted by the vehicle 801 to a space useful to the vehicle behind, the capacity of the transmitted three-dimensional data can be effectively reduced, thereby achieving low communication delay and low cost.

[0577] Next, the configuration of the three-dimensional data creation device 810 according to this embodiment will be described. Figure 38This is a block diagram illustrating an example configuration of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is mounted on, for example, a vehicle 801. The three-dimensional data creation device 810 transmits and receives three-dimensional data with external traffic cloud monitoring, preceding vehicles, and following vehicles, while also creating and storing the three-dimensional data.

[0578] The three-dimensional data production device 810 includes: a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data production unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a sending control unit 820, a format conversion unit 821, and a data sending unit 822.

[0579] The data receiving unit 811 receives 3D data 831 from traffic cloud monitoring or the preceding vehicle. The 3D data 831 includes information about areas that cannot be detected by the vehicle's sensors 815, such as point clouds, visible light images, depth information, sensor position information, or speed information.

[0580] The communication unit 812 communicates with the traffic cloud monitoring or the vehicle ahead, and sends a data transmission request or the like to the traffic cloud monitoring or the vehicle ahead.

[0581] The reception control unit 813 exchanges information such as a corresponding format with the communication partner via the communication unit 812 to establish communication with the communication partner.

[0582] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion on the three-dimensional data 831 received by the data reception unit 811. Furthermore, if the three-dimensional data 831 is compressed or coded, the format conversion unit 814 performs decompression or decoding processing.

[0583] Multiple sensors 815 are a group of sensors such as LIDAR, visible light cameras, or infrared cameras that acquire information outside of vehicle 801 and generate sensor information 833. For example, if sensor 815 is a laser sensor such as LIDAR, sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Furthermore, there may not be multiple sensors 815.

[0584] The three-dimensional data generating unit 816 generates three-dimensional data 834 based on the sensor information 833. The three-dimensional data 834 includes, for example, point clouds, visible light images, depth information, sensor position information, or speed information.

[0585] The three-dimensional data synthesis unit 817 synthesizes the three-dimensional data 832 produced by traffic cloud monitoring or the vehicle in front, etc., into the three-dimensional data 834 produced based on the sensor information 833 of the own vehicle, thereby constructing three-dimensional data 835 that also includes the space in front of the vehicle in front that cannot be detected by the sensor 815 of the own vehicle.

[0586] The three-dimensional data accumulation unit 818 accumulates the generated three-dimensional data 835 and the like.

[0587] The communication unit 819 communicates with the traffic cloud monitoring system or the vehicle behind, and sends a data transmission request or the like to the traffic cloud monitoring system or the vehicle behind.

[0588] The transmission control unit 820 exchanges information such as the corresponding format with the communication partner via the communication unit 819 to establish communication with the communication partner. Furthermore, the transmission control unit 820 determines the spatial transmission area of the three-dimensional data to be transmitted based on the three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and the data transmission request from the communication partner.

[0589] Specifically, based on data transmission requests from traffic cloud monitoring or following vehicles, the transmission control unit 820 determines a transmission area, including the space in front of the vehicle itself, which is not detectable by the following vehicle's sensors. Furthermore, the transmission control unit 820 determines the transmission area by determining, based on the three-dimensional data construction information, whether updates to the available transmission space or the already transmitted space have occurred. For example, the transmission control unit 820 determines as the transmission area an area that is both specified in the data transmission request and contains the corresponding three-dimensional data 835. Furthermore, the transmission control unit 820 notifies the format conversion unit 821 of the communication partner's supported format and the transmission area.

[0590] Format conversion unit 821 converts three-dimensional data 836 in the transmission area of three-dimensional data 835 stored in three-dimensional data storage unit 818 into a format compatible with the receiving end, thereby generating three-dimensional data 837. Format conversion unit 821 may also compress or encode three-dimensional data 837 to reduce the data volume.

[0591] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic cloud monitoring system or the vehicle behind. The three-dimensional data 837 includes, for example, information about areas that are blind spots for the vehicle behind, such as a point cloud, visible light image, depth information, or sensor position information in front of the vehicle.

[0592] In addition, although the format conversion units 814 and 821 perform format conversion and the like, the format conversion may not be performed.

[0593] With this configuration, three-dimensional data creation device 810 obtains three-dimensional data 831 from an external source for an area that is undetectable by the vehicle's sensors 815, and generates three-dimensional data 835 by combining three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the vehicle's sensors 815. This allows three-dimensional data creation device 810 to generate three-dimensional data for an area that is undetectable by the vehicle's sensors 815.

[0594] In addition, the three-dimensional data production device 810 can send three-dimensional data of the space in front of its own vehicle that cannot be detected by the sensors of the rear vehicle to the traffic cloud monitoring or the rear vehicle, etc. according to the data sending request from the traffic cloud monitoring or the rear vehicle.

[0595] Next, the procedure for transmitting three-dimensional data to the following vehicle in the three-dimensional data creation device 810 will be described. Figure 39 This is a flowchart showing an example of the procedure for transmitting three-dimensional data to a traffic cloud monitor or a following vehicle by the three-dimensional data creation device 810.

[0596] First, a three-dimensional data creation device 810 generates and updates three-dimensional data 835 of a space including space 802 on the road ahead of the vehicle 801 (S801). Specifically, the three-dimensional data creation device 810 combines three-dimensional data 831 generated by traffic cloud monitoring or a preceding vehicle with three-dimensional data 834 generated based on sensor information 833 of the vehicle 801. This combination creates three-dimensional data 835 that also includes the space ahead of the preceding vehicle, which cannot be detected by the vehicle's sensors 815.

[0597] Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 included in the transmitted space has changed ( S802 ).

[0598] When the three-dimensional data 835 contained in the space changes due to a vehicle or a person entering the sent space from the outside ("Yes" in S802), the three-dimensional data production device 810 sends the three-dimensional data including the three-dimensional data 835 of the changed space to the traffic cloud monitoring or the rear vehicle (S803).

[0599] Furthermore, while the 3D data creation device 810 can transmit the 3D data of the changed space at the same timing as the 3D data transmission at the predetermined interval, it can also transmit the data immediately after detecting the change. In other words, the 3D data creation device 810 can prioritize the transmission of the 3D data of the changed space over the 3D data transmitted at the predetermined interval.

[0600] Furthermore, the three-dimensional data production device 810 may send all three-dimensional data of the changed space as three-dimensional data of the changed space, or may send only the differences in the three-dimensional data (such as information on the appearance or disappearance of three-dimensional points, or information on the displacement of three-dimensional points, etc.).

[0601] Furthermore, the 3D data creation device 810 may also transmit metadata related to the vehicle's own evasive maneuver, such as an emergency braking warning, to the vehicle behind it before transmitting the 3D data of the changed space. This allows the vehicle behind it to detect the emergency braking of the vehicle ahead and initiate evasive maneuvers such as deceleration more quickly.

[0602] When the three-dimensional data 835 contained in the sent space does not change ("No" in S802) or after step S803, the three-dimensional data production device 810 sends the three-dimensional data contained in the space of a specified shape at a distance L in front of its own vehicle 801 to the traffic cloud monitoring or the rear vehicle (S804).

[0603] Furthermore, for example, the processing of steps S801 to S804 is repeatedly executed at predetermined time intervals.

[0604] Furthermore, when there is no difference between the three-dimensional data 835 of the space 802 currently being transmitted and the three-dimensional map, the three-dimensional data creation device 810 may not transmit the three-dimensional data 837 of the space 802 .

[0605] Figure 40 3D data creation apparatus 810 is a flowchart showing the operation of the 3D data creation apparatus 810 in this case.

[0606] First, the three-dimensional data creation device 810 generates and updates the three-dimensional data 835 of the space including the space 802 on the road ahead of the own vehicle 801 ( S811 ).

[0607] Next, the three-dimensional data creation device 810 determines whether the generated three-dimensional data 835 of the space 802 has been updated with any differences from the three-dimensional map (S812). Specifically, the three-dimensional data creation device 810 determines whether there are any differences between the generated three-dimensional data 835 of the space 802 and the three-dimensional map. Here, the three-dimensional map is three-dimensional map information managed by an infrastructure device such as a traffic cloud monitoring system. For example, this three-dimensional map can be obtained as three-dimensional data 831.

[0608] If there is an update ("Yes" in S812), the three-dimensional data creation device 810 transmits the three-dimensional data included in the space 802 to the traffic cloud monitoring or the following vehicle (S813) in the same manner as described above.

[0609] If no update has been made ("No" in S812), the three-dimensional data creation device 810 does not transmit the three-dimensional data contained in space 802 to the traffic cloud monitoring system or the vehicle behind (S814). Alternatively, the three-dimensional data creation device 810 may control the transmission of the three-dimensional data for space 802 by setting the volume of space 802 to zero. Furthermore, the three-dimensional data creation device 810 may transmit information indicating that no update has been made to space 802 to the traffic cloud monitoring system or the vehicle behind.

[0610] As described above, for example, if there are no obstacles on the road, there will be no difference between the generated three-dimensional data 835 and the three-dimensional map on the infrastructure side, so the data will not be transmitted. This can suppress the transmission of unnecessary data.

[0611] In the above description, although the three-dimensional data creation device 810 is mounted on a vehicle as an example, the three-dimensional data creation device 810 is not limited to being mounted on a vehicle and can be mounted on any mobile object.

[0612] As described above, the three-dimensional data creation device 810 according to this embodiment is mounted on a mobile object equipped with a sensor 815 and a communication unit (such as a data receiving unit 811 or a data transmitting unit 822) for transmitting and receiving three-dimensional data to and from the outside. The three-dimensional data creation device 810 creates three-dimensional data 835 (second three-dimensional data) based on sensor information 833 detected by the sensor 815 and three-dimensional data 831 (first three-dimensional data) received by the data receiving unit 811. The three-dimensional data creation device 810 then transmits three-dimensional data 837, which is a portion of the three-dimensional data 835, to the outside.

[0613] Thus, the three-dimensional data creation device 810 can generate three-dimensional data of the range that the own vehicle cannot detect. In addition, the three-dimensional data creation device 810 can send the three-dimensional data of the range that other vehicles cannot detect to the other vehicles.

[0614] Furthermore, the three-dimensional data creation device 810 repeatedly creates three-dimensional data 835 and transmits three-dimensional data 837 at predetermined intervals. The three-dimensional data 837 is three-dimensional data of a small space 802 of a predetermined size located a predetermined distance L ahead of the current position of the vehicle 801 in the direction of movement of the vehicle 801.

[0615] According to this, since the range of the three-dimensional data 837 to be transmitted is limited, the data amount of the three-dimensional data 837 to be transmitted can be reduced.

[0616] Furthermore, the predetermined distance L varies according to the moving speed V of the vehicle 801. For example, the greater the moving speed V, the longer the predetermined distance L. Thus, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801 and transmit the three-dimensional data 837 of the small space 802 to the following vehicle, etc.

[0617] Furthermore, the predetermined size varies according to the moving speed V of vehicle 801. For example, the greater the moving speed V, the larger the predetermined size. For example, the greater the moving speed V, the greater the depth D, which is the length of small space 802 in the vehicle's moving direction. Thus, vehicle 801 can set an appropriate small space 802 according to its moving speed V and transmit three-dimensional data 837 of that small space 802 to, for example, a following vehicle.

[0618] Then, three-dimensional data creation device 810 determines whether there has been a change in three-dimensional data 835 of small space 802 corresponding to the transmitted three-dimensional data 837. If three-dimensional data creation device 810 determines that there has been a change, three-dimensional data 837 (fourth three-dimensional data) representing at least a portion of the changed three-dimensional data 835 is transmitted to an external vehicle behind, for example.

[0619] As a result, the vehicle 801 can transmit the three-dimensional data 837 of the changed space to the following vehicle and the like.

[0620] Furthermore, the three-dimensional data creation device 810 prioritizes the transmission of the modified three-dimensional data 837 (fourth three-dimensional data) over the regularly transmitted normal three-dimensional data 837 (third three-dimensional data). Specifically, the three-dimensional data creation device 810 transmits the modified three-dimensional data 837 (fourth three-dimensional data) before the regularly transmitted normal three-dimensional data 837 (third three-dimensional data). In other words, the three-dimensional data creation device 810 transmits the modified three-dimensional data 837 (fourth three-dimensional data) non-periodically without waiting for the regularly transmitted normal three-dimensional data 837.

[0621] According to this, since the vehicle 801 can preferentially transmit the three-dimensional data 837 of the changed space to the following vehicle, etc., the following vehicle, etc. can quickly make a judgment based on the three-dimensional data.

[0622] Furthermore, the changed three-dimensional data 837 (fourth three-dimensional data) shows the difference between the three-dimensional data 835 of the small space 802 corresponding to the transmitted three-dimensional data 837 and the changed three-dimensional data 835. This can reduce the amount of the transmitted three-dimensional data 837.

[0623] Furthermore, if there is no difference between the three-dimensional data 837 of the small space 802 and the three-dimensional data 831 of the small space 802, the three-dimensional data creation device 810 does not transmit the three-dimensional data 837 of the small space 802. Alternatively, the three-dimensional data creation device 810 may transmit information indicating that there is no difference between the three-dimensional data 837 of the small space 802 and the three-dimensional data 831 of the small space 802 to the outside.

[0624] According to this, since the transmission of unnecessary three-dimensional data 837 can be suppressed, the data amount of the transmitted three-dimensional data 837 can be reduced.

[0625] (Implementation 6)

[0626] In this embodiment, a display device and a display method for displaying information obtained from a three-dimensional map or the like, and a storage device and a storage method for the three-dimensional map or the like are described.

[0627] Mobile objects such as vehicles and robots utilize three-dimensional maps obtained through communication with servers or other vehicles, as well as two-dimensional images and three-dimensional vehicle detection data obtained from sensors mounted on the vehicle itself, for autonomous driving or robot movement. The data that users wish to view or save varies depending on the situation. The following describes a display device that switches its display according to different situations.

[0628] Figure 41 This is a flowchart showing an overview of a display method using a display device. The display device is mounted on a mobile object such as a vehicle or a robot. In the following description, an example in which the mobile object is a vehicle (automobile) is used.

[0629] First, the display device determines whether to display two-dimensional surrounding information or three-dimensional surrounding information based on the vehicle's driving conditions (S901). The two-dimensional surrounding information corresponds to the first surrounding information in the technical solution, and the three-dimensional surrounding information corresponds to the second surrounding information in the technical solution. Here, surrounding information is information showing the surroundings of a moving object, such as an image viewed from the vehicle in a predetermined direction or a map of the vehicle's surroundings.

[0630] Two-dimensional surrounding information is information generated using two-dimensional data. Here, two-dimensional data refers to two-dimensional map information or images. For example, two-dimensional surrounding information is a map of the vehicle's surroundings obtained from a two-dimensional map, or an image captured by a camera mounted on the vehicle. Furthermore, two-dimensional surrounding information, for example, does not include three-dimensional information. Specifically, if the two-dimensional surrounding information is a map of the vehicle's surroundings, the map does not include altitude information. Furthermore, if the two-dimensional surrounding information is an image captured by a camera, the image does not include depth information.

[0631] Furthermore, three-dimensional surrounding information is information generated using three-dimensional data. Here, the three-dimensional data is, for example, a three-dimensional map. Furthermore, the three-dimensional data may include information showing the three-dimensional position or shape of objects around the vehicle, obtained from other vehicles or a server, or detected by the vehicle itself. For example, the three-dimensional surrounding information is a two-dimensional or three-dimensional image or map of the vehicle's surroundings generated using a three-dimensional map. Furthermore, the three-dimensional surrounding information may include three-dimensional information. For example, if the three-dimensional surrounding information is an image in front of the vehicle, the image may include information showing the distance to objects within the image. Furthermore, the image may include, for example, pedestrians blocked by the vehicle ahead. Furthermore, the three-dimensional surrounding information may include information showing these distances or pedestrians, etc., superimposed on an image obtained by sensors mounted on the vehicle. Furthermore, the three-dimensional surrounding information may also include information on the height direction superimposed on the two-dimensional map.

[0632] Furthermore, three-dimensional data can be displayed three-dimensionally, and a two-dimensional image or a two-dimensional map obtained from the three-dimensional data can also be displayed on a two-dimensional display or the like.

[0633] If, in step S901, the decision is made to display three-dimensional surrounding information ("Yes" in S902), the display device displays the three-dimensional surrounding information (S903). Alternatively, if, in step S901, the decision is made to display two-dimensional surrounding information ("No" in S902), the display device displays two-dimensional surrounding information (S904). In this manner, the display device displays the three-dimensional surrounding information or the two-dimensional surrounding information determined to be displayed in step S901.

[0634] The following describes specific examples. In the first example, the display device switches the surrounding information to be displayed depending on whether the vehicle is being driven automatically or manually. Specifically, during automatic driving, the display device displays two-dimensional surrounding information (e.g., a two-dimensional map) because the driver does not need to know detailed surrounding road information. Furthermore, during manual driving, three-dimensional surrounding information (e.g., a three-dimensional map) is displayed for safe driving, allowing the driver to understand detailed surrounding road information.

[0635] Furthermore, during autonomous driving, the display device can display information that influences driving operations (e.g., SWLD, lanes, road signs, and surrounding condition detection results used to estimate the vehicle's position) to show the user what information the vehicle is using to drive. For example, the display device can add this information to a two-dimensional map.

[0636] The surrounding information displayed during automatic and manual driving described above is merely an example. The display device may also display three-dimensional surrounding information during automatic driving and two-dimensional surrounding information during manual driving. Furthermore, in at least one of automatic and manual driving, the display device may display metadata or surrounding condition detection results in addition to or in place of a two-dimensional or three-dimensional map or image. Metadata refers to information indicating the three-dimensional position or shape of an object obtained from a server or another vehicle. Furthermore, surrounding condition detection results refer to information indicating the three-dimensional position or shape of an object detected by the vehicle itself.

[0637] In the second example, the display device switches the surrounding information to be displayed according to the driving environment. For example, the display device switches the surrounding information to be displayed according to the ambient brightness. Specifically, when the surroundings of the vehicle are bright, the display device displays a two-dimensional image captured by a camera mounted on the vehicle, or displays three-dimensional surrounding information generated using this two-dimensional image. Alternatively, when the surroundings of the vehicle are dark, the display device displays three-dimensional surrounding information generated using a lidar or millimeter-wave radar, as the two-dimensional image captured by the camera mounted on the vehicle is dim and difficult to see.

[0638] Furthermore, the display device can also switch the surrounding information displayed based on the vehicle's current location, i.e., the driving area. For example, the display device can display three-dimensional surrounding information in a manner that provides the user with information about buildings near tourist attractions, city centers, or the destination. Furthermore, considering that detailed information about the surrounding area is often not required in mountainous or suburban areas, the display device can also display two-dimensional surrounding information.

[0639] Furthermore, the display device can also switch the surrounding information displayed based on weather conditions. For example, on sunny days, the display device displays 3D surrounding information generated by a camera or lidar. On rainy days or in dense fog, the display device displays 3D surrounding information generated by a millimeter-wave radar, as 3D surrounding information generated by a camera or lidar is prone to noise.

[0640] Furthermore, the switching of these displays may be performed automatically by the system or manually by the user.

[0641] In addition, the three-dimensional surrounding information is generated based on one or more of the following data, namely: dense point group data generated based on WLD, grid data generated based on MWLD, sparse data generated based on SWLD, lane data generated based on lane world space, two-dimensional map data including three-dimensional shape information of roads and intersections, and metadata including three-dimensional position or three-dimensional shape information that changes according to actual time or the vehicle detection results itself.

[0642] The aforementioned WLD is three-dimensional point cloud data. SWLD is data obtained by extracting point clouds with features exceeding a threshold from the WLD. Furthermore, MWLD is data with a mesh structure generated from the WLD. Lane world space is data obtained by extracting point clouds with features exceeding a threshold from the WLD and required for self-position estimation, driving assistance, or autonomous driving.

[0643] MWLD and SWLD have smaller data volumes than WLD. Therefore, WLD can be used when more detailed data is required, while MWLD or SWLD can be used in other situations to appropriately reduce the amount of communication data and processing. Furthermore, the lane world space has smaller data volumes than SWLD. Therefore, using the lane world space can further reduce the amount of communication data and processing.

[0644] Furthermore, while the above description uses the example of switching between two-dimensional surrounding information and three-dimensional surrounding information, the display device may also switch the type of data (WLD, SWLD, etc.) used to generate the three-dimensional surrounding information based on the above conditions. Specifically, in the above description, in the example where the display device displays three-dimensional surrounding information, the display device may display three-dimensional surrounding information generated based on a larger amount of first data (e.g., WLD or SWLD), while in the example where the display device displays two-dimensional surrounding information, the display device may display three-dimensional surrounding information generated not based on the two-dimensional surrounding information but based on second data (e.g., SWLD or lane world space), where the second data is smaller than the first data.

[0645] Furthermore, the display device displays the two-dimensional surrounding information or the three-dimensional surrounding information on, for example, a two-dimensional display, a head-up display, or a head-mounted display mounted on the vehicle itself. Alternatively, the display device can transmit the two-dimensional surrounding information or the three-dimensional surrounding information to a mobile terminal such as a smartphone via wireless communication and display it. In other words, the display device is not limited to being mounted on a moving object, but can operate in conjunction with the moving object. For example, when a user holding a display device such as a smartphone is riding on a moving object or driving a moving object, information about the moving object, such as the position of the moving object estimated based on the moving object's own position, can be displayed on the display device, or this information can be displayed on the display device together with surrounding information.

[0646] Furthermore, when displaying a three-dimensional map, the display device may render the three-dimensional map and display it as two-dimensional data, or may utilize a three-dimensional display or a three-dimensional hologram to display it as three-dimensional data.

[0647] Next, we'll explain how to save a three-dimensional map. Mobile objects like vehicles and robots utilize automated driving or autonomous movement, utilizing three-dimensional maps obtained through communication with servers and other vehicles, as well as two-dimensional images from sensors mounted on the vehicle itself, or three-dimensional data detected by the vehicle itself. It's important to consider that the data a user wishes to view or save may vary depending on the situation. The following describes how to save data based on the situation.

[0648] The storage device is mounted on a mobile object such as a vehicle or a robot. The following description uses an example where the mobile object is a vehicle (automobile). Furthermore, the storage device may also be included in the display device described above.

[0649] In the first example, the storage device determines whether to store the three-dimensional map based on the region. By storing the three-dimensional map on the vehicle's storage medium, autonomous driving can be performed within the storage space without communicating with the server. However, due to storage capacity limitations, only a limited amount of data can be stored. For this reason, the storage device is limited to the storage areas shown below.

[0650] For example, the storage device prioritizes saving three-dimensional maps of frequently visited areas, such as the route to work or the area around one's home. This eliminates the need to acquire data for frequently used areas, effectively reducing the amount of communication data. Furthermore, priority storage means that high-priority data is stored within a predetermined storage capacity. For example, if the storage capacity cannot hold new data, data with lower priority than the new data is deleted.

[0651] Furthermore, the storage device prioritizes storing three-dimensional maps in areas with poor communication environments. This eliminates the need to obtain data through communication in areas with poor communication environments, thereby preventing the three-dimensional map from being unavailable due to poor communication.

[0652] Alternatively, the storage device prioritizes the storage of three-dimensional maps for areas with heavy traffic. This allows for the prioritization of three-dimensional maps for accident-prone areas. This can prevent a reduction in the accuracy of autonomous driving or driving assistance in such areas, which could be caused by poor communication and inability to obtain three-dimensional maps.

[0653] Alternatively, the storage device prioritizes saving three-dimensional maps of areas with low traffic. In areas with low traffic, the likelihood of not being able to use the autonomous driving mode that automatically follows the vehicle ahead is increased. Consequently, more detailed surrounding information may be required. Therefore, by prioritizing the saving of three-dimensional maps of areas with low traffic, the accuracy of autonomous driving or driving assistance in such areas can be improved.

[0654] Furthermore, the plurality of storage methods described above may be combined, and the regions where these three-dimensional maps are preferentially stored may be automatically determined by the system or specified by the user.

[0655] Furthermore, the storage device can delete three-dimensional maps that have been stored for a specified period of time, or update them with the latest data. This prevents old map data from being used. Furthermore, when updating map data, the storage device compares the old map with the new map to detect spatially distinct regions, known as difference regions. Data from the difference regions of the new map can be added to the old map, or removed from the old map, allowing data to be updated only in the regions that have changed.

[0656] Furthermore, in this example, the stored three-dimensional map is used for autonomous driving. Therefore, by using SWLD as this three-dimensional map, the amount of communication data can be reduced. Furthermore, the three-dimensional map is not limited to SWLD and can also be other types of data such as WLD.

[0657] In the second example, the storage device stores the three-dimensional map according to the event.

[0658] For example, the storage device can store special events encountered while driving as a three-dimensional map. This allows the user to view and listen to the details of the event afterward. The following is an example of an event stored as a three-dimensional map. Alternatively, the storage device can also store three-dimensional surrounding information generated from the three-dimensional map.

[0659] For example, the storage device stores the three-dimensional map before and after a collision or when danger is sensed.

[0660] Alternatively, the storage device holds a three-dimensional map of a characteristic scene such as a beautiful scenery, a place where people gather, or a tourist attraction.

[0661] These events to be saved can be determined automatically by the system or specified in advance by the user. For example, machine learning can be used as a method to determine these events.

[0662] Furthermore, in this example, the stored three-dimensional map is used for viewing. Therefore, by using WLD as the three-dimensional map, high-quality images can be provided. Furthermore, the three-dimensional map is not limited to WLD and can also be other types of data such as SWLD.

[0663] The following describes a method for controlling the display on a display device in accordance with user control. When the display device overlays the surrounding condition detection results obtained through inter-vehicle communication onto a map and displays them, it can display surrounding vehicles as wireframes or by adding transparency to the surrounding vehicles, thereby allowing for the visibility of detected objects obscured by surrounding vehicles. Alternatively, the display device can display an image from an overhead viewpoint, or it can display the vehicle itself, surrounding vehicles, and surrounding condition detection results from a bird's-eye view.

[0664] When using a head-up display to overlay the surrounding condition detection results or point group data onto the Figure 42 In the case of the surrounding environment viewed through the windshield as shown, the position of the superimposed information may deviate due to differences in the user's posture, body shape, or eye position. Figure 43 It is a diagram showing a display example of the head-up display when a positional deviation occurs.

[0665] To correct this misalignment, the display device uses information from in-vehicle cameras or seat-mounted sensors to detect the user's posture, body shape, or eye position. The display device adjusts the position of the overlaid information based on the detected user's posture, body shape, or eye position. Figure 44 FIG. 1 is a diagram showing an example of a display on a head-up display after adjustment.

[0666] In addition, such adjustment of the overlapping position can be performed manually by the user using a control device mounted on the vehicle.

[0667] Furthermore, the display device can display safe places in the event of a disaster on a map and prompt the user. Alternatively, the vehicle can inform the user of the disaster details and a message to move to a safe place, and then perform autonomous driving until reaching a safe place.

[0668] For example, if a vehicle sets its destination at a high altitude area that is safe from tsunamis in the event of an earthquake, the vehicle can communicate with the server to obtain information about roads that are difficult to navigate due to the earthquake and take appropriate action to avoid these roads.

[0669] In addition, autonomous driving can include multiple modes such as mobile mode and long-distance driving mode.

[0670] In mobile mode, the vehicle decides the route to the destination based on factors such as early arrival time, low cost, short driving distance, and low fuel consumption, and drives autonomously along the determined route.

[0671] In driving tour mode, the vehicle automatically determines a route to reach the destination at the user-specified time. For example, if the user sets a destination and arrival time, the vehicle will automatically drive along the route, allowing for sightseeing and reaching the destination within the specified time.

[0672] (Implementation 7)

[0673] In the fifth embodiment, a client device such as a vehicle transmits three-dimensional data to another vehicle or a server such as a traffic cloud monitoring system. In this embodiment, the client device transmits sensor information obtained by a sensor to a server or another client device.

[0674] First, the configuration of the system according to this embodiment will be described. Figure 45 This diagram shows the configuration of a system for transmitting and receiving three-dimensional maps and sensor information according to this embodiment. The system includes a server 901 and client devices 902A and 902B. In the absence of a distinction between client devices 902A and 902B, they are referred to as client device 902.

[0675] The client device 902 is, for example, an in-vehicle device mounted on a mobile object such as a vehicle. The server 901 is, for example, a traffic cloud monitoring system, and is capable of communicating with a plurality of client devices 902 .

[0676] The server 901 transmits a three-dimensional map composed of point clouds to the client device 902. In addition, the composition of the three-dimensional map is not limited to point clouds, and can also be expressed by other three-dimensional data such as a mesh structure.

[0677] The client device 902 sends sensor information obtained by the client device 902 to the server 901. The sensor information includes, for example, at least one of LIDAR information, visible light images, infrared images, depth images, sensor position information, and speed information.

[0678] The data sent and received between server 901 and client device 902 can be compressed to reduce data size, or not compressed to maintain data accuracy. When data compression is used, for example, a three-dimensional compression method based on an octree can be used for point clouds. Furthermore, two-dimensional image compression methods can be used for visible light images, infrared images, and depth images. Examples of two-dimensional image compression methods include MPEG-4 AVC and HEVC standardized by MPEG.

[0679] Furthermore, in response to a request from client device 902 to transmit the three-dimensional map, server 901 transmits the three-dimensional map managed by server 901 to client device 902. Alternatively, server 901 may transmit the three-dimensional map without waiting for a request from client device 902 to transmit the three-dimensional map. For example, server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, server 901 may transmit a three-dimensional map appropriate for the location of client device 902 at regular intervals to a client device 902 that has received a transmission request. Furthermore, server 901 may transmit the three-dimensional map to client device 902 whenever the three-dimensional map managed by server 901 is updated.

[0680] The client device 902 sends a three-dimensional map transmission request to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 sends the three-dimensional map transmission request to the server 901.

[0681] In addition, the client device 902 may also issue a request to transmit a three-dimensional map to the server 901 in the following situations. If the three-dimensional map held by the client device 902 is relatively old, the client device 902 may also issue a request to transmit the three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may also issue a request to transmit the three-dimensional map to the server 901.

[0682] Alternatively, client device 902 may issue a three-dimensional map transmission request to server 901 a certain time before client device 902 is about to exit the space shown on the three-dimensional map held by client device 902. For example, client device 902 may issue a three-dimensional map transmission request to server 901 when client device 902 is within a predetermined distance from the boundary of the space shown on the three-dimensional map held by client device 902. Furthermore, if the movement path and speed of client device 902 are known, the time at which client device 902 will exit the space shown on the three-dimensional map held by client device 902 can be predicted based on the known movement path and speed.

[0683] When the error between the three-dimensional data generated by the client device 902 based on the sensor information and the position of the three-dimensional map is greater than a certain range, the client device 902 may send a request to the server 901 to send the three-dimensional map.

[0684] Client device 902 transmits sensor information to server 901 in response to a sensor information transmission request received from server 901. Alternatively, client device 902 may transmit sensor information to server 901 without waiting for a sensor information transmission request from server 901. For example, after receiving a sensor information transmission request from server 901, client device 902 may periodically transmit sensor information to server 901 over a predetermined period. Furthermore, if the position error between three-dimensional data generated by client device 902 based on sensor information and a three-dimensional map received from server 901 is within a predetermined range, client device 902 may determine that the three-dimensional map surrounding client device 902 may have changed, and transmit this determination result along with the sensor information to server 901.

[0685] Server 901 issues a sensor information transmission request to client device 902. For example, server 901 receives location information of client device 902, such as GPS, from client device 902. Based on the location information of client device 902, if server 901 determines that client device 902 is approaching a space with less information on the three-dimensional map managed by server 901, server 901 issues a sensor information transmission request to client device 902 in order to regenerate the three-dimensional map. Alternatively, server 901 may issue a sensor information transmission request to update the three-dimensional map, to check road conditions such as during snow accumulation or disasters, or to check traffic conditions or incidents.

[0686] Furthermore, the client device 902 may set the data volume of the sensor information to be transmitted to the server 901 according to the communication state or bandwidth at the time of receiving the sensor information transmission request received from the server 901. Setting the data volume of the sensor information to be transmitted to the server 901 may include, for example, increasing or decreasing the data itself or selecting an appropriate compression method.

[0687] Figure 46 This is a block diagram illustrating an example configuration of client device 902. Client device 902 receives a three-dimensional map composed of, for example, a point cloud from server 901 and estimates its own position based on the three-dimensional data generated based on sensor information from client device 902. Client device 902 then transmits the acquired sensor information to server 901.

[0688] The client device 902 includes: a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data production unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a sending control unit 1021, and a data sending unit 1022.

[0689] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0690] The communication unit 1012 communicates with the server 901 and transmits a data transmission request (for example, a transmission request for a three-dimensional map) and the like to the server 901 .

[0691] The reception control unit 1013 exchanges information such as the corresponding format with the communication partner via the communication unit 1012 and establishes communication with the communication partner.

[0692] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding. If the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding.

[0693] Multiple sensors 1015 are a group of sensors mounted on client device 902, such as LIDAR, visible light cameras, infrared cameras, or depth sensors, for obtaining information outside the vehicle and generating sensor information 1033. For example, if sensor 1015 is a laser sensor such as LIDAR, sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Furthermore, there may not be multiple sensors 1015.

[0694] The three-dimensional data generation unit 1016 generates three-dimensional data 1034 of the surrounding area of the own vehicle based on the sensor information 1033. For example, the three-dimensional data generation unit 1016 uses information obtained by LIDAR and visible light images obtained by a visible light camera to generate point cloud data with color information of the surrounding area of the own vehicle.

[0695] The three-dimensional image processing unit 1017 uses the received three-dimensional map 1032, such as a point cloud, and three-dimensional data 1034 of the vehicle's surroundings generated based on the sensor information 1033 to perform processing such as estimating the vehicle's own position. Alternatively, the three-dimensional image processing unit 1017 may synthesize the three-dimensional map 1032 and the three-dimensional data 1034 to create three-dimensional data 1035 of the vehicle's surroundings, and use the created three-dimensional data 1035 to perform processing for estimating the vehicle's own position.

[0696] The three-dimensional data storage unit 1018 stores the three-dimensional map 1032 , the three-dimensional data 1034 , and the three-dimensional data 1035 .

[0697] The format conversion unit 1019 converts the sensor information 1033 into a format compatible with the receiving end, thereby generating sensor information 1037. Furthermore, the format conversion unit 1019 can reduce the data volume by compressing or encoding the sensor information 1037. Furthermore, if format conversion is not necessary, the format conversion unit 1019 can omit the processing. Furthermore, the format conversion unit 1019 can control the amount of data transmitted according to the specified transmission range.

[0698] The communication unit 1020 communicates with the server 901 and receives a data transmission request (a sensor information transmission request) and the like from the server 901 .

[0699] The transmission control unit 1021 exchanges information such as the corresponding format with the communication partner via the communication unit 1020 to establish communication.

[0700] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information obtained by the plurality of sensors 1015, such as information obtained by LIDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0701] Next, the configuration of the server 901 will be described. Figure 47 This is a block diagram illustrating an example configuration of server 901. Server 901 receives sensor information transmitted from client device 902 and generates three-dimensional data based on the received sensor information. Server 901 uses the generated three-dimensional data to update the three-dimensional map managed by server 901. Furthermore, server 901 transmits the updated three-dimensional map to client device 902 in response to a request for transmission of the three-dimensional map from client device 902.

[0702] The server 901 includes: a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data production unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a sending control unit 1121, and a data sending unit 1122.

[0703] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information obtained by LIDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0704] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a sensor information transmission request) and the like to the client device 902 .

[0705] The reception control unit 1113 exchanges information such as the corresponding format with the communication partner via the communication unit 1112 to establish communication.

[0706] If the received sensor information 1037 is compressed or coded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. If the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0707] The three-dimensional data generation unit 1116 generates three-dimensional data 1134 of the surroundings of the client device 902 based on the sensor information 1132. For example, the three-dimensional data generation unit 1116 uses information obtained by LIDAR and visible light images obtained by a visible light camera to generate point cloud data with color information of the surroundings of the client device 902.

[0708] The three-dimensional data synthesis unit 1117 synthesizes the three-dimensional data 1134 generated based on the sensor information 1132 with the three-dimensional map 1135 managed by the server 901 , thereby updating the three-dimensional map 1135 .

[0709] The three-dimensional data storage unit 1118 stores the three-dimensional map 1135 and the like.

[0710] Format conversion unit 1119 generates three-dimensional map 1031 by converting three-dimensional map 1135 into a format compatible with the receiving end. Format conversion unit 1119 may also compress or encode three-dimensional map 1135 to reduce the data volume. Furthermore, if format conversion is not necessary, format conversion unit 1119 may omit the processing. Furthermore, format conversion unit 1119 may control the amount of data transmitted according to the specified transmission range.

[0711] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) and the like from the client device 902 .

[0712] The transmission control unit 1121 exchanges information such as the corresponding format with the communication partner via the communication unit 1120 to establish communication.

[0713] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0714] Next, the operation flow of the client device 902 will be described. Figure 48 This is a flowchart showing the operation of the client device 902 when obtaining a three-dimensional map.

[0715] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 also transmits its location information obtained by GPS, etc., and thus requests the server 901 to transmit a three-dimensional map related to the location information.

[0716] Next, the client device 902 receives a 3D map from the server 901 ( S1002 ). If the received 3D map is compressed data, the client device 902 decodes the received 3D map to generate an uncompressed 3D map ( S1003 ).

[0717] Next, the client device 902 creates three-dimensional data 1034 of the client device 902's surroundings based on the sensor information 1033 obtained from the plurality of sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created based on the sensor information 1033 (S1005).

[0718] Figure 49 1037 is a flowchart illustrating the operations performed by client device 902 when transmitting sensor information. First, client device 902 receives a request to transmit sensor information from server 901 (S1011). Upon receiving the request, client device 902 transmits sensor information 1037 to server 901 (S1012). If sensor information 1033 includes multiple pieces of information obtained by multiple sensors 1015, client device 902 compresses each piece of information using a compression method appropriate for that piece of information, thereby generating sensor information 1037.

[0719] Next, the operation flow of the server 901 will be described. Figure 50 This is a flowchart illustrating the operations performed by server 901 when acquiring sensor information. First, server 901 requests client device 902 to transmit sensor information (S1021). Next, server 901 receives sensor information 1037 transmitted from client device 902 in response to the request (S1022). Next, server 901 creates three-dimensional data 1134 using received sensor information 1037 (S1023). Next, server 901 reflects created three-dimensional data 1134 on three-dimensional map 1135 (S1024).

[0720] Figure 511 is a flowchart showing the actions of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). The server 901, having received the request to transmit the three-dimensional map, transmits the three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 can extract a three-dimensional map in the vicinity of the client device 902 corresponding to the location information of the client device 902 and transmit the extracted three-dimensional map. Furthermore, the server 901 can compress the three-dimensional map composed of the point cloud, for example, using an octree compression method, and transmit the compressed three-dimensional map.

[0721] Hereinafter, modifications of this embodiment will be described.

[0722] Server 901 uses sensor information 1037 received from client device 902 to create three-dimensional data 1134 near the location of client device 902. Next, server 901 matches the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by server 901, calculating the difference between the three-dimensional data 1134 and the three-dimensional map 1135. If the difference exceeds a predetermined threshold, server 901 determines that some abnormality has occurred near client device 902. For example, if ground subsidence occurs due to a natural disaster such as an earthquake, a significant difference may occur between the three-dimensional map 1135 managed by server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0723] Sensor information 1037 may also include at least one of the sensor type, sensor performance, and sensor model. Furthermore, a category ID corresponding to the sensor's performance may be added to sensor information 1037. For example, if sensor information 1037 is obtained by LIDAR, an identifier may be assigned based on sensor performance. For example, a sensor capable of obtaining information with an accuracy of several millimeters may be assigned category 1, a sensor capable of obtaining information with an accuracy of several centimeters may be assigned category 2, and a sensor capable of obtaining information with an accuracy of several meters may be assigned category 3. Furthermore, server 901 may estimate sensor performance information based on the model of client device 902. For example, if client device 902 is mounted in a vehicle, server 901 may determine sensor specifications based on the vehicle's model. In this case, server 901 may obtain the vehicle's model information in advance or include this information in the sensor information. Furthermore, server 901 may use the obtained sensor information 1037 to switch the degree of correction applied to three-dimensional data 1134 generated using sensor information 1037. For example, when the sensor performance is high accuracy (category 1), the server 901 does not perform correction on the three-dimensional data 1134. When the sensor performance is low accuracy (category 3), the server 901 applies correction appropriate to the sensor accuracy to the three-dimensional data 1134. For example, the lower the sensor accuracy, the greater the degree (intensity) of correction.

[0724] The server 901 can also simultaneously request sensor information to multiple client devices 902 located in a given space. When the server 901 receives multiple sensor information items from multiple client devices 902, it is not necessary to utilize all of the sensor information in creating the three-dimensional data 1134. For example, the server 901 can select sensor information to be used based on sensor performance. For example, when updating the three-dimensional map 1135, the server 901 can select highly accurate sensor information (category 1) from the multiple sensor information items received and use this information to create the three-dimensional data 1134.

[0725] The server 901 is not limited to servers such as traffic cloud monitoring, but can also be other client devices (vehicle-mounted). Figure 52 This is a diagram showing the system configuration in this case.

[0726] For example, client device 902C requests sensor information from nearby client device 902A and receives the sensor information from client device 902A. Client device 902C then uses the sensor information received from client device 902A to create three-dimensional data and update the three-dimensional map on client device 902C. In this way, client device 902C can leverage the performance of client device 902C to generate a three-dimensional map of the space available from client device 902A. This is possible, for example, when client device 902C has high performance.

[0727] In this case, client device 902A, which has provided the sensor information, is granted the right to obtain the highly accurate three-dimensional map generated by client device 902C. Client device 902A receives the highly accurate three-dimensional map from client device 902C in accordance with this right.

[0728] Alternatively, client device 902C may request the transmission of sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B is high-performance, client device 902C can create three-dimensional data using the sensor information obtained by the high-performance sensor.

[0729] Figure 53 This is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes a 3D map, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0730] The client device 902 includes a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives the compressed encoded data of the three-dimensional map, decodes the encoded data, and obtains the three-dimensional map. The sensor information compression processing unit 1212 does not compress the three-dimensional data generated by the obtained sensor information, but instead compresses the sensor information itself and transmits the compressed encoded data of the sensor information to the server 901. With this configuration, the client device 902 can internally retain the processing unit (device or LSI) for decoding the three-dimensional map (point cloud, etc.), without having to internally retain the processing unit for compressing the three-dimensional data of the three-dimensional map (point cloud, etc.). This can reduce the cost and power consumption of the client device 902.

[0731] As described above, the client device 902 according to this embodiment is mounted on a mobile object and generates three-dimensional data 1034 of the mobile object's surroundings based on sensor information 1033 indicating the surrounding conditions of the mobile object obtained by the sensor 1015 mounted on the mobile object. The client device 902 estimates the mobile object's own position using the generated three-dimensional data 1034. The client device 902 transmits the obtained sensor information 1033 to the server 901 or another mobile object 902.

[0732] Based on this, the client device 902 transmits the sensor information 1033 to the server 901, etc. This can potentially reduce the amount of data transmitted compared to when transmitting three-dimensional data. Furthermore, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing required by the client device 902 can be reduced. Consequently, the client device 902 can reduce the amount of data transmitted and simplify the device configuration.

[0733] Furthermore, the client device 902 sends a request to send a three-dimensional map to the server 901, and receives the three-dimensional map 1031 from the server 901. The client device 902 estimates its own position using the three-dimensional data 1034 and the three-dimensional map 1032.

[0734] Furthermore, the sensor information 1033 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0735] Also, the sensor information 1033 includes information showing the performance of the sensor.

[0736] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile object 902. Thus, the client device 902 can reduce the amount of transmitted data.

[0737] For example, the client device 902 includes a processor and a memory, and the processor performs the above-described processing using the memory.

[0738] Furthermore, the server 901 according to this embodiment can communicate with a client device 902 mounted on a mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object obtained by a sensor 1015 mounted on the mobile object from the client device 902. Based on the received sensor information 1037, the server 901 creates three-dimensional data 1134 of the surroundings of the mobile object.

[0739] Based on this, server 901 uses sensor information 1037 transmitted from client device 902 to create three-dimensional data 1134. This potentially reduces the amount of data transmitted compared to when client device 902 transmits the three-dimensional data. Furthermore, since client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the processing load on client device 902 can be reduced. This allows server 901 to reduce the amount of data transmitted and simplify the device configuration.

[0740] Furthermore, the server 901 further transmits a request to transmit sensor information to the client device 902 .

[0741] Furthermore, the server 901 further updates the three-dimensional map 1135 using the generated three-dimensional data 1134 , and transmits the three-dimensional map 1135 to the client device 902 in response to a transmission request for the three-dimensional map 1135 from the client device 902 .

[0742] Furthermore, the sensor information 1037 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0743] Also, the sensor information 1037 includes information showing the performance of the sensor.

[0744] Furthermore, the server 901 further calibrates the three-dimensional data according to the performance of the sensor.

[0745] Furthermore, when receiving sensor information, server 901 receives multiple sensor information 1037 from multiple client devices 902 and selects sensor information 1037 for use in creating three-dimensional data 1134 based on multiple pieces of information indicating sensor performance included in the multiple sensor information 1037. This allows server 901 to improve the quality of three-dimensional data 1134.

[0746] Furthermore, the server 901 decodes or decompresses the received sensor information 1037 and creates three-dimensional data 1134 based on the decoded or decompressed sensor information 1132. In this way, the server 901 can reduce the amount of data to be transmitted.

[0747] For example, the server 901 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0748] (Implementation 8)

[0749] In this embodiment, a method for encoding and a method for decoding three-dimensional data using inter-frame prediction processing will be described.

[0750] Figure 54 3D data encoding device 1300 according to this embodiment is a block diagram. The 3D data encoding device 1300 encodes 3D data to generate an encoded bit stream (hereinafter also referred to as a bit stream) as an encoded signal. Figure 54 As shown, the three-dimensional data encoding device 1300 includes: a segmentation unit 1301, a subtraction unit 1302, a transformation unit 1303, a quantization unit 1304, an inverse quantization unit 1305, an inverse transformation unit 1306, an addition unit 1307, a reference volume memory 1308, an intra-frame prediction unit 1309, a reference space memory 1310, an inter-frame prediction unit 1311, a prediction control unit 1312, and an entropy coding unit 1313.

[0751] The segmentation unit 1301 segments each space (SPC) contained in the three-dimensional data into a plurality of volumes (VLMs) serving as coding units. Furthermore, the segmentation unit 1301 performs an octree representation (octree conversion) on the voxels within each volume. Alternatively, the segmentation unit 1301 may make the space and the volume the same size and perform an octree representation on the space. Furthermore, the segmentation unit 1301 may append information required for octree conversion (such as depth information) to the header of the bitstream.

[0752] The subtraction unit 1302 calculates the difference between the volume (coding target volume) output from the division unit 1301 and the prediction volume generated by intra prediction or inter prediction described later, and outputs the calculated difference as a prediction residual to the transformation unit 1303 . Figure 55 : is a diagram showing an example of calculating a prediction residual. Note that the bit strings of the encoding target volume and the predicted volume shown here are, for example, position information showing the positions of three-dimensional points (eg, point clouds) included in the volume.

[0753] The following describes octree representation and the voxel scanning order. The volume is converted to an octree structure (octreeing) and then encoded. The octree structure consists of nodes and leaf nodes. Each node has eight nodes or leaves, and each leaf node contains voxel (VXL) information. Figure 56 This is a diagram showing an example of the configuration of a volume including a plurality of voxels. Figure 57 It means Figure 56 The volume shown is transformed into an example of an octree structure. Figure 57 The leaf nodes 1, 2, and 3 shown in the figure represent Figure 56 The voxels VXL1, VXL2, and VXL3 shown express the VXL (hereinafter referred to as effective VXL) including the point group.

[0754] For example, an octree is represented by a binary sequence of 0 and 1. For example, when a node or a valid VXL is set to 1 and the others are set to 0, each node and leaf node is assigned Figure 57 The binary sequence shown in FIG. Then, the binary sequence is scanned in a width-first or depth-first scanning order. For example, in the case of a width-first scan, Figure 58 The binary sequence shown in A. When scanning in depth first, we get Figure 58 The binary sequence obtained by this scanning is encoded by entropy coding, thereby reducing the amount of information.

[0755] Next, we will explain the depth information in the octree representation. The depth in the octree representation is used to control the granularity of the point cloud information contained in the volume. If the depth is set to a large value, the point cloud information can be reproduced at a finer level, but the amount of data used to represent the nodes and leaf nodes will increase. Conversely, if the depth is set to a small value, the amount of data can be reduced, but multiple point cloud information with different positions and colors will be treated as the same position and the same color, thus losing the original information of the point cloud information.

[0756] For example, Figure 59 It means Figure 57 The diagram shows an example of an octree with depth = 2 being represented by an octree with depth = 1. Figure 59 The octree shown is Figure 57 The octree shown has a small amount of data. Figure 59 The octree shown is the same as Figure 59 Compared to the octree shown, the number of bits after binary serialization is less. Figure 57 The leaf nodes 1 and 2 shown in the figure become Figure 58 The leaf node 1 shown is shown. That is, Figure 57 The leaf node 1 and the leaf node 2 shown are information on different locations.

[0757] Figure 60 Is to express Figure 59 A diagram of the volume corresponding to the octree shown. Figure 56 VXL1 and VXL2 shown with Figure 60 In this case, the three-dimensional data encoding device 1300 corresponds to the VXL12 shown. Figure 56 The color information of VXL1 and VXL2 shown here generates Figure 60For example, the three-dimensional data encoding device 1300 calculates the color information of VXL12 using the average, median, or weighted average of the color information of VXL1 and VXL2 as the color information of VXL12. In this way, the three-dimensional data encoding device 1300 can control the reduction of data volume by changing the depth of the octree.

[0758] The three-dimensional data encoding device 1300 may also set the depth information of the octree using any of world space units, space units, and volume units. Furthermore, in this case, the three-dimensional data encoding device 1300 may also attach the depth information to the world space header information, the space header information, or the volume header information. Furthermore, the same value may be used as depth information in all world spaces, spaces, and volumes at different times. In this case, the three-dimensional data encoding device 1300 may also attach the depth information to the header information that manages the world space at all times.

[0759] When the voxels contain color information, the transformation unit 1303 applies a frequency transform, such as an orthogonal transform, to the prediction residuals of the color information of the voxels within the volume. For example, the transformation unit 1303 scans the prediction residuals in a certain scanning order to create a one-dimensional array. The transformation unit 1303 then applies a one-dimensional orthogonal transform to the created one-dimensional array, thereby transforming the one-dimensional array into the frequency domain. Consequently, when the values of the prediction residuals within the volume are close, the values of the frequency components in the low-frequency band become larger, while the values of the frequency components in the high-frequency band become smaller. Consequently, the quantization unit 1304 can more effectively reduce the amount of code.

[0760] Furthermore, the transform unit 1303 may utilize an orthogonal transform of two or more dimensions instead of a one-dimensional orthogonal transform. For example, the transform unit 1303 maps the prediction residual into a two-dimensional arrangement in a certain scanning order, and applies a two-dimensional orthogonal transform to the obtained two-dimensional arrangement. Furthermore, the transform unit 1303 may select the orthogonal transform method to be used from a plurality of orthogonal transform methods. In this case, the three-dimensional data encoding device 1300 appends information indicating which orthogonal transform method is used to the bitstream. Furthermore, the transform unit 1303 may select the orthogonal transform method to be used from a plurality of orthogonal transform methods with different dimensions. In this case, the three-dimensional data encoding device 1300 appends information indicating which dimensional orthogonal transform method is used to the bitstream.

[0761] For example, the transformation unit 1303 matches the scanning order of the prediction residuals with the scanning order (width-first or depth-first, etc.) in the octree within the volume. Accordingly, since it is not necessary to append information indicating the scanning order of the prediction residuals to the bitstream, it is possible to reduce additional overhead. Furthermore, the transformation unit 1303 may also apply a scanning order different from the scanning order of the octree. In this case, the three-dimensional data encoding device 1300 appends information indicating the scanning order of the prediction residuals to the bitstream. Accordingly, the three-dimensional data encoding device 1300 can efficiently encode the prediction residuals. Furthermore, the three-dimensional data encoding device 1300 may append information indicating whether the scanning order of the octree is applied (a flag, etc.) to the bitstream, and in the case where the scanning order of the octree is not applied, append information indicating the scanning order of the prediction residuals to the bitstream.

[0762] The conversion unit 1303 may convert not only the prediction residual of color information but also other attribute information of the voxel. For example, the conversion unit 1303 may convert and encode information such as reflectivity obtained when a point cloud is acquired by LiDAR or the like.

[0763] The conversion unit 1303 may skip processing when the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding device 1300 may add information (flag) indicating whether to skip processing by the conversion unit 1303 to the bitstream.

[0764] The quantization unit 1304 quantizes the frequency components of the prediction residual generated by the transformation unit 1303 using the quantization control parameters to generate quantization coefficients. This reduces the amount of information. The generated quantization coefficients are output to the entropy coding unit 1313. The quantization unit 1304 can control the quantization control parameters according to world space units, space units, or volume units. In this case, the three-dimensional data encoding device 1300 attaches the quantization control parameters to the respective header information, etc. In addition, the quantization unit 1304 can also change the weight according to the frequency components of each prediction residual to perform quantization control. For example, the quantization unit 1304 can perform fine quantization on the low-frequency components and coarse quantization on the high-frequency components. In this case, the three-dimensional data encoding device 1300 can attach parameters representing the weights of each frequency component to the header.

[0765] The quantization unit 1304 may skip the process if the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding apparatus 1300 may add information (flag) indicating whether the process of the quantization unit 1304 is skipped to the bitstream.

[0766] The inverse quantization unit 1305 inversely quantizes the quantization coefficients generated by the quantization unit 1304 using the quantization control parameters, thereby generating inverse quantization coefficients of the prediction residual, and outputs the generated inverse quantization coefficients to the inverse transformation unit 1306 .

[0767] The inverse transform unit 1306 applies inverse transform to the inverse quantized coefficients generated by the inverse quantization unit 1305 to generate a post-inverse transform prediction residual. Since this post-inverse transform prediction residual is a prediction residual generated after quantization, it may not completely match the prediction residual output by the transform unit 1303.

[0768] The adder 1307 adds the inverse-transformed prediction residual generated by the inverse-transformer 1306 to the prediction volume generated by intra-frame prediction or inter-frame prediction, which is used to generate the pre-quantized prediction residual, to generate a reconstructed volume. This reconstructed volume is stored in the reference volume memory 1308 or the reference space memory 1310.

[0769] The intra prediction unit 1309 generates a predicted volume of the encoding target volume using the attribute information of the adjacent volumes stored in the reference volume memory 1308. The attribute information includes the color information or reflectivity of the voxels. The intra prediction unit 1309 generates a predicted value of the color information or reflectivity of the encoding target volume.

[0770] Figure 61 1309 is a diagram for explaining the operation of the intra prediction unit 1309. For example, Figure 61 As shown, the intra-frame prediction unit 1309 generates a predicted volume of the encoding target volume (volume idx = 3) based on the adjacent volume (volume idx = 0). Here, volume idx is the identifier information added to the volume in the space, and different values are assigned to each volume. The order of assigning volume idx can be the same as the encoding order or different from the encoding order. For example, as Figure 61 The intra-frame prediction unit 1309 uses the average value of the color information of the voxels contained in volume idx=0, which is the adjacent volume, as the predicted value of the color information of the encoding target volume. In this case, the prediction residual is generated by subtracting the predicted value of the color information from the color information of each voxel contained in the encoding target volume. The processing subsequent to the transformation unit 1303 is performed on this prediction residual. In this case, the three-dimensional data encoding device 1300 also adds the adjacent volume information and prediction mode information to the bitstream. Here, the adjacent volume information is information indicating the adjacent volume used in the prediction, such as the volume idx indicating the adjacent volume used in the prediction. The prediction mode information indicates the mode used to generate the prediction volume. The mode may be, for example, an average value mode that generates a prediction value based on the average value of the voxels in the adjacent volume, or an average value mode that generates a prediction value based on the average value of the voxels in the adjacent volume.

[0771] The intra prediction unit 1309 can also generate a prediction volume based on multiple adjacent volumes. Figure 61 In the illustrated configuration, the intra-frame prediction unit 1309 generates prediction volume 0 based on volume idx = 0, and generates prediction volume 1 based on volume idx = 1. The intra-frame prediction unit 1309 then generates the final prediction volume by averaging the prediction volumes 0 and 1. In this case, the three-dimensional data encoding apparatus 1300 may also append multiple volume idx values used in generating the prediction volumes to the bitstream.

[0772] Figure 62 This diagram schematically illustrates the inter-frame prediction process involved in this embodiment. The inter-frame prediction unit 1311 performs encoding (inter-frame prediction) on the space (SPC) at a certain time T_Cur using the coded space at a different time T_LX. In this case, the inter-frame prediction unit 1311 applies rotation and translation processing to the coded space at different times T_LX to perform encoding.

[0773] Furthermore, the 3D data encoding apparatus 1300 appends RT information related to the spatial rotation and translation processing applied at a different time T_LX to the bitstream. For example, the different time T_LX is a time T_L0 prior to the certain time T_Cur. In this case, the 3D data encoding apparatus 1300 may also append RT information RT_L0 related to the spatial rotation and translation processing applied at time T_L0 to the bitstream.

[0774] Alternatively, the different time T_LX is, for example, time T_L1 after the certain time T_Cur. In this case, the three-dimensional data encoding apparatus 1300 may add RT information RT_L1 regarding the spatial rotation and translation processing applied at time T_L1 to the bitstream.

[0775] Alternatively, the inter-frame prediction unit 1311 performs encoding by referring to spaces at different times T_L0 and T_L1 (bi-prediction). In this case, the 3D data encoding apparatus 1300 may add RT information RT_L0 and RT_L1 regarding spatial rotation and translation, respectively, to the bitstream.

[0776] In addition, although T_L0 is set as the time before T_Cur and T_L1 is set as the time after T_Cur, the present invention is not limited to this. For example, T_L0 and T_L1 can both be the time before T_Cur. Alternatively, T_L0 and T_L1 can both be the time after T_Cur.

[0777] Furthermore, when encoding with reference to multiple spaces at different times, the three-dimensional data encoding device 1300 may append RT information related to the rotation and translation applied to each space to the bitstream. For example, the three-dimensional data encoding device 1300 manages the multiple referenced encoded spaces using two reference lists (L0 list and L1 list). When the first reference space in the L0 list is set to L0R0, the second reference space in the L0 list is set to L0R1, the first reference space in the L1 list is set to L1R0, and the second reference space in the L1 list is set to L1R1, the three-dimensional data encoding device 1300 appends RT information RT_L0R0 for L0R0, RT information RT_L0R1 for L0R1, RT information RT_L1R0 for L1R0, and RT information RT_L1R1 for L1R1 to the bitstream. For example, the three-dimensional data encoding device 1300 appends this RT information to the header of the bitstream, etc.

[0778] Alternatively, when encoding with reference to multiple reference spaces at different times, the three-dimensional data encoding device 1300 may determine whether rotation and translation are applicable for each reference space. In this case, the three-dimensional data encoding device 1300 may append information indicating whether rotation and translation are applicable for each reference space (such as an RT applicable flag) to header information of the bitstream. For example, the three-dimensional data encoding device 1300 calculates RT information and an ICP error value using an ICP (Interactive Closest Point) algorithm for each reference space to be referenced based on the encoding target space. If the ICP error value is below a predetermined value, the three-dimensional data encoding device 1300 determines that rotation and translation are not necessary and sets the RT applicable flag to OFF (invalid). Furthermore, if the ICP error value is greater than the predetermined value, the three-dimensional data encoding device 1300 sets the RT applicable flag to ON (valid) and appends the RT information to the bitstream.

[0779] Figure 63 This figure shows an example of a syntax for appending RT information and an RT applicable flag to a header. Furthermore, the number of bits allocated to each syntax can be determined based on the range of values that the syntax can assume. For example, if the number of reference spaces included in reference list L0 is 8, 3 bits can be allocated to MaxRefSpc_10. The number of allocated bits can be varied based on the values that each syntax can assume, or can be fixed regardless of the values that can be assumed. When the number of allocated bits is fixed, the three-dimensional data encoding device 1300 can append this fixed number of bits to other header information.

[0780] Here, Figure 63MaxRefSpc_10 indicates the number of reference spaces included in reference list L0. RT_flag_10[i] is the RT application flag for reference space i in reference list L0. When RT_flag_10[i] is 1, rotation and translation are applied to reference space i. When RT_flag_10[i] is 0, rotation and translation are not applied to reference space i.

[0781] R_l0[i] and T_l0[i] are the RT information for reference space i within reference list L0. R_l0[i] is the rotation information for reference space i within reference list L0. The rotation information indicates the content of the applied rotation processing, such as a rotation matrix or quaternion. T_l0[i] is the translation information for reference space i within reference list L0. The translation information indicates the content of the applied translation processing, such as a translation vector.

[0782] MaxRefSpc_l1 indicates the number of reference spaces included in reference list L1. RT_flag_l1[i] is the RT application flag for reference space i in reference list L1. When RT_flag_l1[i] is 1, rotation and translation are applied to reference space i. When RT_flag_l1[i] is 0, rotation and translation are not applied to reference space i.

[0783] R_l1[i] and T_l1[i] are the RT information for reference space i within reference list L1. R_l1[i] is the rotation information for reference space i within reference list L1. The rotation information indicates the content of the applied rotation processing, such as a rotation matrix or quaternion. T_l1[i] is the translation information for reference space i within reference list L1. The translation information indicates the content of the applied translation processing, such as a translation vector.

[0784] The inter-frame prediction unit 1311 generates a predicted volume for the target volume using information about the already-encoded reference space stored in the reference space memory 1310. As described above, before generating the predicted volume for the target volume, the inter-frame prediction unit 1311 uses the ICP (Interactive Closest Point) algorithm to determine RT information in the target space and the reference space to approximate the overall positional relationship between the target space and the reference space. The inter-frame prediction unit 1311 then uses the determined RT information to rotate and translate the reference space, thereby obtaining reference space B. Subsequently, the inter-frame prediction unit 1311 generates a predicted volume for the target volume within the target space using information within reference space B. The three-dimensional data encoding device 1300 appends the RT information used to obtain reference space B to, for example, header information of the target space.

[0785] In this way, the inter-frame prediction unit 1311 applies rotation and translation processing to the reference space, thereby making the overall positional relationship between the encoding target space and the reference space close, and then uses the information in the reference space to generate the prediction volume. This can improve the accuracy of the prediction volume. In addition, since the prediction residual can be suppressed, the amount of coding can be reduced. In addition, although the example here shows the use of the encoding target space and the reference space for ICP, it is not limited to this. For example, in order to reduce the processing amount, the inter-frame prediction unit 1311 can also use at least one of the encoding target space from which the number of voxels or point clouds is extracted and the reference space from which the number of voxels or point clouds is extracted to perform ICP to obtain RT information.

[0786] Furthermore, if the ICP error value obtained from the ICP result is less than a predetermined first threshold, that is, for example, if the positional relationship between the encoding target space and the reference space is close, the inter-frame prediction unit 1311 may determine that rotation and translation are unnecessary and not perform rotation and translation. In this case, the three-dimensional data encoding device 1300 does not need to add RT information to the bitstream, thereby reducing overhead.

[0787] Furthermore, if the ICP error value exceeds a predetermined second threshold, the inter-frame prediction unit 1311 determines that the spatial shape change is significant and may apply intra-frame prediction to all volumes in the encoding target space. Hereinafter, the space to which intra-frame prediction is applied is referred to as the intra-frame space. Furthermore, the second threshold is a value greater than the first threshold described above. Furthermore, the method is not limited to ICP; any method that can determine RT information from two voxel sets or two point cloud sets may be applied.

[0788] Furthermore, when the three-dimensional data contains attribute information such as shape or color, the inter-frame prediction unit 1311 searches, for example, for a volume in the reference space that is closest to the shape or color attribute information of the target volume as a prediction volume for the target volume within the target space. Furthermore, this reference space is, for example, a reference space that has been subjected to the aforementioned rotation and translation processing. The inter-frame prediction unit 1311 generates a prediction volume based on the volume (reference volume) obtained through the search. Figure 64 is a diagram for explaining the generation operation of the prediction volume. Figure 64When encoding the target volume (volume idx = 0) shown in the figure using inter-frame prediction, the reference volumes within the reference space are sequentially scanned while searching for the volume with the smallest prediction residual, the difference between the target volume and the reference volume. The inter-frame prediction unit 1311 selects the volume with the smallest prediction residual as the prediction volume. The prediction residual between the target volume and the prediction volume is encoded by the transformation unit 1303 and subsequent processing. Here, the prediction residual refers to the difference between the attribute information of the target volume and the attribute information of the prediction volume. Furthermore, the 3D data encoding device 1300 appends the volume idx of the reference volume within the reference space used as the prediction volume to the header of the bitstream, etc.

[0789] exist Figure 64 In the example shown, the reference volume idx=4 in the reference space L0R0 is selected as the prediction volume of the encoding target volume. Then, the prediction residual between the encoding target volume and the reference volume and the reference volume idx=4 are encoded and added to the bitstream.

[0790] In addition, although the description here is given of an example of generating a predicted volume of attribute information, the same processing can be performed on a predicted volume of position information.

[0791] The prediction control unit 1312 controls whether intra-frame prediction or inter-frame prediction is used to encode the target volume. A mode that includes both intra-frame prediction and inter-frame prediction is referred to as a prediction mode. For example, the prediction control unit 1312 calculates an evaluation value using the prediction residual when the target volume is predicted using intra-frame prediction and the prediction residual when the target volume is predicted using inter-frame prediction, and selects the prediction mode with the lower evaluation value. Alternatively, the prediction control unit 1312 may apply orthogonal transform, quantization, and entropy coding to the prediction residual for intra-frame prediction and the prediction residual for inter-frame prediction, respectively, to calculate the actual code amount and use the calculated code amount as the evaluation value to select the prediction mode. Furthermore, overhead information other than the prediction residual (such as reference volume IDX information) may be added to the evaluation value. Furthermore, if the target space is predetermined to be encoded in intra-frame space, the prediction control unit 1312 may normally select intra-frame prediction.

[0792] The entropy coding unit 1313 generates a coded signal (coded bit stream) by performing variable-length coding on the quantized coefficients input from the quantization unit 1304. Specifically, the entropy coding unit 1313 binarizes the quantized coefficients and performs arithmetic coding on the resulting binary signal.

[0793] Next, a three-dimensional data decoding device that decodes the encoded signal generated by the three-dimensional data encoding device 1300 will be described. Figure 65This is a block diagram of a three-dimensional data decoding device 1400 according to this embodiment. The three-dimensional data decoding device 1400 includes an entropy decoding unit 1401, an inverse quantization unit 1402, an inverse transform unit 1403, an addition unit 1404, a reference volume memory 1405, an intra-frame prediction unit 1406, a reference space memory 1407, an inter-frame prediction unit 1408, and a prediction control unit 1409.

[0794] The entropy decoding unit 1401 performs variable-length decoding on the coded signal (coded bit stream). For example, the entropy decoding unit 1401 performs arithmetic decoding on the coded signal to generate a binary signal, and generates quantization coefficients based on the generated binary signal.

[0795] The inverse quantization unit 1402 performs inverse quantization on the quantized coefficients input from the entropy decoding unit 1401 using a quantization parameter added to a bit stream or the like, thereby generating inverse quantized coefficients.

[0796] The inverse transform unit 1403 generates a prediction residual by performing an inverse transform on the inverse quantized coefficients input from the inverse quantization unit 1402. For example, the inverse transform unit 1403 generates a prediction residual by performing an inverse orthogonal transform on the inverse quantized coefficients based on information added to the bitstream.

[0797] The adder 1404 adds the prediction residual generated by the inverse transform unit 1403 to the prediction volume generated by intra-frame prediction or inter-frame prediction to generate a reconstructed volume. This reconstructed volume is output as decoded 3D data and stored in the reference volume memory 1405 or the reference space memory 1407.

[0798] The intra prediction unit 1406 generates a prediction volume through intra prediction using the reference volume in the reference volume memory 1405 and information appended to the bitstream. Specifically, the intra prediction unit 1406 obtains prediction mode information and neighboring volume information (e.g., volume idx) appended to the bitstream, and generates a prediction volume using the neighboring volumes indicated by the neighboring volume information using the mode indicated by the prediction mode information. The details of this processing are the same as those of the intra prediction unit 1309 described above, except that the information appended to the bitstream is used.

[0799] The inter-frame prediction unit 1408 generates a prediction volume through inter-frame prediction using the reference space in the reference space memory 1407 and the information attached to the bitstream. Specifically, the inter-frame prediction unit 1408 uses the RT information of each reference space attached to the bitstream to apply rotation and translation processing to the reference space, and generates a prediction volume using the applied reference space. In addition, when the RT application flag of each reference space is present in the bitstream, the inter-frame prediction unit 1408 applies rotation and translation processing to the reference space according to the RT application flag. In addition, the details of the above-mentioned processing are the same as those of the above-mentioned inter-frame prediction unit 1311, except that the information attached to the bitstream is used.

[0800] Whether the decoding target volume is decoded using intra-frame prediction or inter-frame prediction is controlled by the prediction control unit 1409. For example, the prediction control unit 1409 selects intra-frame prediction or inter-frame prediction based on information appended to the bitstream indicating the prediction mode to be used. Alternatively, if the decoding target space is predetermined to be decoded as an intra-frame space, the prediction control unit 1409 may normally select intra-frame prediction.

[0801] The following describes variations of this embodiment. While this embodiment describes the application of rotation and translation using spatial units as an example, rotation and translation can also be applied using finer units. For example, the three-dimensional data encoding device 1300 can divide the space into subspaces and apply rotation and translation in subspace units. In this case, the three-dimensional data encoding device 1300 generates RT information for each subspace and appends the generated RT information to the header of the bitstream. Alternatively, the three-dimensional data encoding device 1300 can apply rotation and translation using volume units as encoding units. In this case, the three-dimensional data encoding device 1300 generates RT information in encoding volume units and appends the generated RT information to the header of the bitstream. Furthermore, combinations of these methods are possible. That is, the three-dimensional data encoding device 1300 can apply rotation and translation in large units and then apply rotation and translation in finer units. For example, the three-dimensional data encoding device 1300 can apply rotation and translation in spatial units and then apply different rotations and translations to each of the multiple volumes contained in the resulting space.

[0802] Furthermore, while this embodiment illustrates the application of rotation and translation to the reference space as an example, this is not limiting. For example, the three-dimensional data encoding device 1300 may apply scaling to change the size of the three-dimensional data. Furthermore, the three-dimensional data encoding device 1300 may also apply any one or both of rotation, translation, and scaling. Furthermore, as described above, when processing is applied in multiple stages using different units, the type of processing applied to each unit may be different. For example, rotation and translation may be applied to spatial units, while translation may be applied to volume units.

[0803] Note that these modifications are also applicable to the three-dimensional data decoding apparatus 1400 .

[0804] As described above, the three-dimensional data encoding device 1300 according to this embodiment performs the following processing. Figure 65 This is a flowchart of the inter-frame prediction process performed by the three-dimensional data encoding device 1300.

[0805] First, the 3D data encoding device 1300 generates predicted position information (e.g., predicted volume) using position information of 3D points included in target 3D data (e.g., encoding target space) and reference 3D data at different times (e.g., reference space) (S1301). Specifically, the 3D data encoding device 1300 generates predicted position information by applying rotation and translation processing to the position information of 3D points included in the reference 3D data.

[0806] Furthermore, the 3D data encoding device 1300 performs rotation and translation processing using a first unit (e.g., space) and generates predicted position information using a second unit (e.g., volume) that is smaller than the first unit. For example, the 3D data encoding device 1300 may search for a volume in the reference space after rotation and translation that minimizes the difference between the encoding target volume and the position information, and use the resulting volume as the predicted volume. Alternatively, the 3D data encoding device 1300 may perform rotation and translation processing and generation of predicted position information using the same unit.

[0807] Furthermore, the three-dimensional data encoding device 1300 may apply a first rotation and translation process to the position information of the three-dimensional points contained in the reference three-dimensional data using a first unit (e.g., space), and apply a second rotation and translation process to the position information of the three-dimensional points obtained by the first rotation and translation process using a second unit (e.g., volume) that is smaller than the first unit, thereby generating predicted position information.

[0808] Here, the position information of the three-dimensional point and the predicted position information are as follows: Figure 58As shown, the octree structure is used for representation. For example, the 3D point position information and predicted position information are represented in a depth-first scan order of the octree structure. Alternatively, the 3D point position information and predicted position information are represented in a depth-first scan order of the octree structure.

[0809] And, as Figure 63 As shown, the three-dimensional data encoding device 1300 encodes an RT applicable flag indicating whether rotation and translation processing is applied to the position information of the three-dimensional point included in the reference three-dimensional data. That is, the three-dimensional data encoding device 1300 generates a coded signal (coded bit stream) including the RT applicable flag. Furthermore, the three-dimensional data encoding device 1300 encodes RT information indicating the content of the rotation and translation processing. That is, the three-dimensional data encoding device 1300 generates a coded signal (coded bit stream) including RT information. Alternatively, the three-dimensional data encoding device 1300 may encode the RT information when the RT applicable flag indicates that rotation and translation processing is applied, and may not encode the RT information when the RT applicable flag indicates that rotation and translation processing is not applied.

[0810] The 3D data includes, for example, position information of 3D points and attribute information (color information, etc.) of each 3D point. The 3D data encoding device 1300 generates predicted attribute information by using the attribute information of the 3D points included in the reference 3D data (S1302).

[0811] Next, the three-dimensional data encoding device 1300 uses the predicted position information to encode the position information of the three-dimensional points included in the object three-dimensional data. Figure 55 As shown, differential position information, which is a difference between the position information of the three-dimensional point included in the target three-dimensional data and the predicted position information, is calculated (S1303).

[0812] The 3D data encoding device 1300 then uses the predicted attribute information to encode the attribute information of the 3D points included in the target 3D data. For example, the 3D data encoding device 1300 calculates differential attribute information, which is the difference between the attribute information of the 3D points included in the target 3D data and the predicted attribute information (S1304). The 3D data encoding device 1300 then transforms and quantizes the calculated differential attribute information (S1305).

[0813] Finally, the 3D data encoding device 1300 encodes (eg, entropy encodes) the differential position information and the quantized differential attribute information ( S1306 ). That is, the 3D data encoding device 1300 generates an encoded signal (encoded bitstream) including the differential position information and the differential attribute information.

[0814] If the 3D data does not contain attribute information, the 3D data encoding device 1300 may not perform steps S1302, S1304, and S1305. Furthermore, the 3D data encoding device 1300 may only encode the 3D point position information or the 3D point attribute information.

[0815] and, Figure 66 The order of processing shown is merely an example and is not intended to be limiting. For example, since the processing of location information (S1301, S1303) and the processing of attribute information (S1302, S1304, S1305) are independent of each other, they can be executed in any order, or some of them can be processed in parallel.

[0816] As described above, in this embodiment, the 3D data encoding device 1300 generates predicted position information using the position information of 3D points contained in the target 3D data and reference 3D data at different times. The device then encodes the difference between the position information of the 3D points contained in the target 3D data and the predicted position information, i.e., the differential position information. This reduces the amount of data in the encoded signal, thereby improving encoding efficiency.

[0817] Furthermore, in this embodiment, the 3D data encoding device 1300 generates predicted attribute information by using attribute information of 3D points contained in the reference 3D data. The device then encodes the difference between the attribute information of the 3D points contained in the target 3D data and the predicted attribute information, i.e., the difference attribute information. This reduces the amount of data in the encoded signal, thereby improving encoding efficiency.

[0818] For example, the three-dimensional data encoding device 1300 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0819] Figure 65 This is a flowchart of the inter-frame prediction process performed by the three-dimensional data decoding apparatus 1400 .

[0820] First, the three-dimensional data decoding apparatus 1400 decodes (eg, performs entropy decoding) the differential position information and the differential attribute information based on the encoded signal (encoded bit stream) ( S1401 ).

[0821] Furthermore, the 3D data decoding apparatus 1400 decodes, based on the coded signal, an RT application flag indicating whether rotation and translation processing is applied to the position information of the 3D points included in the reference 3D data. Furthermore, the 3D data decoding apparatus 1400 decodes RT information indicating the content of the rotation and translation processing. Furthermore, the 3D data decoding apparatus 1400 decodes the RT information if the RT application flag indicates that rotation and translation processing is applied, and does not decode the RT information if the RT application flag indicates that rotation and translation processing is not applied.

[0822] Next, the three-dimensional data decoding apparatus 1400 performs inverse quantization and inverse transformation on the decoded differential attribute information ( S1402 ).

[0823] Next, the 3D data decoding device 1400 generates predicted position information (e.g., predicted volume) using the position information of 3D points included in the target 3D data (e.g., the decoding target space) and the reference 3D data at different times (e.g., the reference space) (S1403). Specifically, the 3D data decoding device 1400 generates the predicted position information by applying rotation and translation processing to the position information of 3D points included in the reference 3D data.

[0824] More specifically, when the RT application flag indicates that rotation and translation processing is applicable, the three-dimensional data decoding apparatus 1400 applies the rotation and translation processing to the position information of the three-dimensional points included in the reference three-dimensional data indicated by the RT information. Furthermore, when the RT application flag indicates that rotation and translation processing is not applicable, the three-dimensional data decoding apparatus 1400 does not apply the rotation and translation processing to the position information of the three-dimensional points included in the reference three-dimensional data.

[0825] Furthermore, the three-dimensional data decoding apparatus 1400 may perform rotation and translation processing in a first unit (e.g., space), and may generate predicted position information in a second unit (e.g., volume) that is smaller than the first unit. Furthermore, the three-dimensional data decoding apparatus 1400 may perform rotation and translation processing and generation of predicted position information in the same unit.

[0826] Furthermore, the three-dimensional data decoding device 1400 may apply a first rotation and translation process using a first unit (e.g., space) to the position information of the three-dimensional points contained in the reference three-dimensional data, and may apply a second rotation and translation process using a second unit (e.g., volume) that is smaller than the first unit to the position information of the three-dimensional points obtained by the first rotation and translation process, thereby generating predicted position information.

[0827] Here, the position information of the three-dimensional point and the predicted position information are as follows: Figure 58As shown, the octree structure is used for representation. For example, the 3D point position information and predicted position information are represented in a depth-first scan order between depth and width in the octree structure. Alternatively, the 3D point position information and predicted position information are represented in a depth-first scan order between depth and width in the octree structure.

[0828] The three-dimensional data decoding apparatus 1400 generates predicted attribute information using attribute information of three-dimensional points included in the reference three-dimensional data ( S1404 ).

[0829] Next, the 3D data decoding device 1400 decodes the encoded position information contained in the encoded signal using the predicted position information, thereby restoring the position information of the 3D points contained in the target 3D data. Here, the encoded position information is, for example, differential position information, and the 3D data decoding device 1400 restores the position information of the 3D points contained in the target 3D data by adding the differential position information to the predicted position information (S1405).

[0830] The 3D data decoding device 1400 then decodes the coded attribute information contained in the coded signal using the predicted attribute information, thereby restoring the attribute information of t...

Claims

1. A three-dimensional data processing method, wherein: include: Generate an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and Information is generated, the information indicating whether 8 bits corresponding to the nodes included in the N-ary tree structure include 4 or more bits respectively corresponding to fixed values.

2. The three-dimensional data processing method according to claim 1, wherein: When N is 4, the number of the four or more bits is 4.

3. The three-dimensional data processing method according to claim 1, wherein: The number of the above 4 bits is (8-N).

4. The three-dimensional data processing method according to claim 1, wherein: The fixed value is 0.

5. The three-dimensional data processing method according to claim 1, wherein: The fixed value is set as an invalid bit.

6. The three-dimensional data processing method according to claim 1, wherein: Also includes: Generate parameters, Here, when the parameter represents the first information, the information indicates that 8 bits corresponding to the nodes included in the N-ary tree structure include 4 or more bits corresponding to fixed values.

7. The three-dimensional data processing method according to claim 6, wherein: When the parameter represents the first information, N is 4.

8. The three-dimensional data processing method according to claim 6, wherein: When the parameter indicates the second information, N is 8.

9. The three-dimensional data processing method according to claim 6, wherein: When the parameter indicates the second information, the information indicates that the 8 bits corresponding to the nodes included in the N-ary tree structure do not include a bit corresponding to a fixed value.

10. The three-dimensional data processing method according to claim 6, wherein: Also includes: The parameters are used to encode information of a current node contained in the N-ary tree structure of the plurality of three-dimensional points.

11. The three-dimensional data processing method according to claim 1, wherein: Also includes: Using the information, information of a current node contained in the N-ary tree structure of the plurality of three-dimensional points is encoded.

12. A three-dimensional data processing method, wherein: include: Obtain an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and Information indicating whether 8 bits corresponding to a node included in the N-ary tree structure include 4 or more bits each corresponding to a fixed value is obtained.

13. The three-dimensional data processing method according to claim 12, wherein: When N is 4, the number of the four or more bits is 4.

14. The three-dimensional data processing method according to claim 12, wherein: The number of the above 4 bits is (8-N).

15. The three-dimensional data processing method according to claim 12, wherein: The fixed value is 0.

16. The three-dimensional data processing method according to claim 12, wherein: The fixed value is set as an invalid bit.

17. The three-dimensional data processing method according to claim 12, wherein: Also includes: Get the parameters, In the case where the parameter represents the first information, the information indicates that 8 bits corresponding to the nodes included in the N-ary tree structure include 4 or more bits each corresponding to a fixed value.

18. The three-dimensional data processing method according to claim 17, wherein: When the parameter indicates the first information, N is 4.

19. The three-dimensional data processing method according to claim 17, wherein: When the parameter indicates the second information, N is 8.

20. The three-dimensional data processing method according to claim 17, wherein: When the parameter indicates the second information, the information indicates that 8 bits corresponding to the nodes included in the N-ary tree structure do not include a bit corresponding to a fixed value.

21. The three-dimensional data processing method according to claim 17, wherein: Also includes: The parameters are used to decode information of a current node contained in the N-ary tree structure of the plurality of three-dimensional points.

22. The three-dimensional data processing method according to claim 12, wherein: Also includes: Using the information, information of a current node contained in the N-ary tree structure of the plurality of three-dimensional points is decoded.

23. A three-dimensional data processing device, wherein: include: processor; as well as Memory, The processor uses the memory, Generate an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and Information is generated, the information indicating whether 8 bits corresponding to the nodes included in the N-ary tree structure include 4 or more bits respectively corresponding to fixed values.

24. A three-dimensional data processing device, wherein: include: processor; as well as Memory, The processor uses the memory, Obtain an N-ary tree structure of multiple three-dimensional points, where N is an integer greater than 2, and Information indicating whether 8 bits corresponding to a node included in the N-ary tree structure include 4 or more bits each corresponding to a fixed value is obtained.

Images