Three-dimensional data production method and three-dimensional data production device

Abstract

The present invention provides a three-dimensional data production method and a three-dimensional data production apparatus. The three-dimensional data production method is a three-dimensional data production method in a mobile body having a sensor and a communication unit that transmits and receives three-dimensional data to and from the outside. Based on the three-dimensional data detected by the sensor and the first three-dimensional data received by the communication unit, the second three-dimensional data is produced, and the third three-dimensional data, which is a part of the second three-dimensional data, is transmitted to the outside. It is determined whether the second three-dimensional data corresponding to the transmitted third three-dimensional data has changed. In the case of a change, the fourth three-dimensional data, which is at least a part of the changed second three-dimensional data, is transmitted to the outside.

Description

[0001] This application is a divisional of a patent application for an invention titled "Three-Dimensional Data Production Method and Three-Dimensional Data Production Device" with an application date of August 23, 2017, an application number of 201780054551.8. Technical Field

[0002] This application relates to a three-dimensional data production method and a three-dimensional data production device. Background Art

[0003] In large fields such as computer vision, map information, monitoring, infrastructure inspection, or video distribution for autonomous operation of automobiles or robots, devices or services that make flexible use of three-dimensional data will become widespread in the future. Three-dimensional data is obtained by various methods such as distance sensors like rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0004] As a method of representing three-dimensional data, there is a method called point cloud data, which represents the shape of a three-dimensional structure by a point group in a three-dimensional space (for example, refer to Non-Patent Document 1). The position and color of the point group are stored in the point cloud data. Although it is expected that the point cloud data will become the mainstream as a method of representing three-dimensional data, the amount of data of the point group is very large. Therefore, in the storage or transmission of three-dimensional data, like two-dimensional moving images (as an example, there are MPEG-4 AVC or HEVC standardized by MPEG), data volume compression is required through encoding.

[0005] Furthermore, for the compression of point cloud data, some are supported by publicly available program libraries (Point Cloud Library) that perform processing related to point cloud data.

[0006] (Prior Art Documents)

[0007] (Non-Patent Documents)

[0008] Non-Patent Document 1 "Octree-Based Progressive Geometry Coding of PointClouds", Eurographics Symposium on Point-Based Graphics (2006) Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] In such a three-dimensional data production device for producing three-dimensional data, it is desired to be able to produce more detailed three-dimensional data.

[0011] An object of the present application is to provide a three-dimensional data production method or a three-dimensional data production apparatus capable of generating detailed three-dimensional data.

[0012] Means for Solving the Problems

[0013] A three-dimensional data production method according to one aspect of the present application is a three-dimensional data production method in a mobile body including a sensor and a communication unit that transmits and receives three-dimensional data to and from the outside. Based on the three-dimensional data detected by the sensor and the first three-dimensional data received by the communication unit, second three-dimensional data is produced, and third three-dimensional data, which is a part of the second three-dimensional data, is transmitted to the outside. It is determined whether the second three-dimensional data corresponding to the transmitted third three-dimensional data has changed. If it has changed, fourth three-dimensional data, which is at least a part of the changed second three-dimensional data, is transmitted to the outside.

[0014] In addition, all or specific aspects of these can be implemented as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and can be implemented by combining the system, the method, the integrated circuit, the computer program, and the recording medium.

[0015] Advantageous Effects of the Invention

[0016] The present application can provide a three-dimensional data production method or a three-dimensional data production apparatus capable of generating detailed three-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shows the configuration of the encoded three-dimensional data according to Embodiment 1.

[0018] Figure 2 Shows an example of the prediction structure between SPCs belonging to the lowest layer of GOS according to Embodiment 1.

[0019] Figure 3 Shows an example of the interlayer prediction structure according to Embodiment 1.

[0020] Figure 4 Shows an example of the encoding order of GOS according to Embodiment 1.

[0021] Figure 5 Shows an example of the encoding order of GOS according to Embodiment 1.

[0022] Figure 6 Is a block diagram of the three-dimensional data encoding apparatus according to Embodiment 1.

[0023] Figure 7 Is a flowchart of the encoding process according to Embodiment 1.

[0024] Figure 8 It is a block diagram of the three-dimensional data decoding device according to Embodiment 1.

[0025] Figure 9 It is a flowchart of the decoding process according to Embodiment 1.

[0026] Figure 10 It shows an example of the meta information according to Embodiment 1.

[0027] Figure 11 It shows a configuration example of the SWLD according to Embodiment 2.

[0028] Figure 12 It shows an operation example of the server and the client according to Embodiment 2.

[0029] Figure 13 It shows an operation example of the server and the client according to Embodiment 2.

[0030] Figure 14 It shows an operation example of the server and the client according to Embodiment 2.

[0031] Figure 15 It shows an operation example of the server and the client according to Embodiment 2.

[0032] Figure 16 It is a block diagram of the three-dimensional data encoding device according to Embodiment 2.

[0033] Figure 17 It is a flowchart of the encoding process according to Embodiment 2.

[0034] Figure 18 It is a block diagram of the three-dimensional data decoding device according to Embodiment 2.

[0035] Figure 19 It is a flowchart of the decoding process according to Embodiment 2.

[0036] Figure 20 It shows a configuration example of the WLD according to Embodiment 2.

[0037] Figure 21 It shows an example of the octree structure of the WLD according to Embodiment 2.

[0038] Figure 22 It shows a configuration example of the SWLD according to Embodiment 2.

[0039] Figure 23 It shows an example of the octree structure of the SWLD according to Embodiment 2.

[0040] Figure 24 It is a schematic diagram showing the state of the transmission and reception of three-dimensional data between vehicles according to Embodiment 3.

[0041] Figure 25 It shows an example of the three-dimensional data transmitted between vehicles according to Embodiment 3.

[0042] Figure 26 It is a block diagram of the three-dimensional data production device according to Embodiment 3.

[0043] Figure 27 It is a flowchart of the three-dimensional data production process according to Embodiment 3.

[0044] Figure 28 It is a block diagram of the three-dimensional data transmission device according to Embodiment 3.

[0045] Figure 29 It is a flowchart of the three-dimensional data transmission process according to Embodiment 3.

[0046] Figure 30 It is a block diagram of the three-dimensional data production device according to Embodiment 3.

[0047] Figure 31 It is a flowchart of the three-dimensional data production process according to Embodiment 3.

[0048] Figure 32 It is a block diagram of the three-dimensional data transmission device according to Embodiment 3.

[0049] Figure 33 It is a flowchart of the three-dimensional data transmission process according to Embodiment 3.

[0050] Figure 34 It is a block diagram of the three-dimensional information processing device according to Embodiment 4.

[0051] Figure 35 It is a flowchart of the three-dimensional information processing method according to Embodiment 4.

[0052] Figure 36 It is a flowchart of the three-dimensional information processing method according to Embodiment 4.

[0053] Figure 37 It is a diagram for explaining the transmission process of three-dimensional data according to Embodiment 5.

[0054] Figure 38 It is a block diagram of the three-dimensional data production device according to Embodiment 5.

[0055] Figure 39 It is a flowchart of the three-dimensional data production method according to Embodiment 5.

[0056] Figure 40 It is a flowchart of the three-dimensional data production method related to Embodiment 5. Specific Embodiment

[0057] When using encoded data such as point cloud data in an actual device or service, random access is required for a desired spatial position or target object, etc. However, so far, random access in three-dimensional encoded data does not exist as a function, and thus, an encoding method therefor also does not exist.

[0058] In the present application, a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can provide a random access function in encoding three-dimensional data can be provided.

[0059] The three-dimensional data encoding method according to one aspect of the present application encodes three-dimensional data. The three-dimensional data encoding method includes: a dividing step of dividing the three-dimensional data into first processing units respectively corresponding to three-dimensional coordinates, and the first processing unit is a random access unit; and an encoding step of generating encoded data by encoding each of the plurality of first processing units.

[0060] Accordingly, random access to each first processing unit becomes possible. In this way, the three-dimensional data encoding method can provide a random access function in encoding three-dimensional data.

[0061] For example, it may also be that the three-dimensional data encoding method includes a generating step in which first information is generated, and the first information shows the plurality of first processing units and the three-dimensional coordinates corresponding to each of the plurality of first processing units, and the encoded data includes the first information.

[0062] For example, it may also be that the first information further shows at least one of an object, a time, and a data storage destination corresponding to each of the plurality of first processing units.

[0063] For example, it may also be that in the dividing step, the first processing unit is further divided into a plurality of second processing units, and in the encoding step, each of the plurality of second processing units is encoded.

[0064] For example, it may also be that in the encoding step, for a second processing unit of a processing object included in a first processing unit of a processing object, encoding is performed with reference to other second processing units included in the first processing unit of the processing object.

[0065] Accordingly, by referring to other second processing units, the encoding efficiency can be improved.

[0066] For example, it may also be that in the encoding step, the type of the second processing unit that is the processing object is selected from a first type that does not refer to other second processing units, a second type that refers to one other second processing unit, and a third type that refers to two other second processing units, and the second processing unit that is the processing object is encoded according to the selected type.

[0067] For example, it may also be that in the encoding step, the frequency of selecting the first type is changed according to the number or density of objects included in the three-dimensional data.

[0068] Accordingly, it is possible to appropriately set the random access performance and encoding efficiency in the trade-off relationship.

[0069] For example, it may also be that in the encoding step, the size of the first processing unit is determined according to the number or density of objects included in the three-dimensional data, or the number or density of dynamic objects.

[0070] Accordingly, it is possible to appropriately set the random access performance and encoding efficiency in the trade-off relationship.

[0071] For example, it may also be that the first processing unit includes a plurality of layers spatially divided in a predetermined direction, and each of the plurality of layers includes one or more of the second processing units. In the encoding step, the second processing unit is encoded by referring to the second processing units included in the same layer as the second processing unit or the layers lower than the second processing unit.

[0072] Accordingly, for example, it is possible to improve the random access performance of important layers in the system and suppress a decrease in encoding efficiency.

[0073] For example, it may also be that in the partitioning step, the second processing units including only static objects and the second processing units including only dynamic objects are assigned to different first processing units.

[0074] Accordingly, it is possible to easily control dynamic objects and static objects.

[0075] For example, it may also be that in the encoding step, a plurality of dynamic objects are encoded separately, and the encoded data of the plurality of dynamic objects corresponds to the second processing units including only static objects.

[0076] Accordingly, it is possible to easily control dynamic objects and static objects.

[0077] For example, it may also be that, in the dividing step, the second processing unit is further divided into a plurality of third processing units, and in the encoding step, each of the plurality of third processing units is encoded.

[0078] For example, it may also be that the third processing unit includes one or more voxels, which are the smallest units corresponding to position information.

[0079] For example, it may also be that the second processing unit includes a group of feature points derived from information obtained by a sensor.

[0080] For example, it may also be that the encoded data includes information indicating the encoding order of the plurality of first processing units.

[0081] For example, it may also be that the encoded data includes information indicating the sizes of the plurality of first processing units.

[0082] For example, it may also be that, in the encoding step, the plurality of first processing units are encoded in parallel.

[0083] Moreover, a three-dimensional data decoding method according to one aspect of the present application includes a decoding step, in which three-dimensional data of the first processing unit is generated by decoding each of the encoded data of the first processing units respectively corresponding to three-dimensional coordinates, and the first processing unit is a random access unit.

[0084] Accordingly, random access to each first processing unit becomes possible. In this way, the three-dimensional data decoding method can provide a random access function in encoded three-dimensional data.

[0085] Moreover, a three-dimensional data encoding apparatus according to one aspect of the present application may include: a dividing unit that divides the three-dimensional data into first processing units respectively corresponding to three-dimensional coordinates, and the first processing unit is a random access unit; and an encoding unit that generates encoded data by encoding each of the plurality of first processing units.

[0086] Accordingly, random access to each first processing unit becomes possible. In this way, the three-dimensional data encoding apparatus can provide a random access function in encoded three-dimensional data.

[0087] Moreover, a three-dimensional data decoding apparatus according to one aspect of the present application decodes three-dimensional data, and the three-dimensional data decoding apparatus includes a decoding unit that generates three-dimensional data of the first processing unit by decoding each of the encoded data of the first processing units respectively corresponding to three-dimensional coordinates, and the first processing unit is a random access unit.

[0088] Accordingly, random access for each first processing unit becomes possible. In this way, the three-dimensional data decoding device can provide a random access function for the encoded three-dimensional data.

[0089] In addition, with the configuration of partitioning and encoding the space in the present application, quantization, prediction, etc. of the space can be made possible, which is effective even without random access.

[0090] Moreover, the three-dimensional data encoding method according to one aspect of the present application includes: an extraction step of extracting second three-dimensional data having a feature amount equal to or greater than a threshold from first three-dimensional data; and a first encoding step of generating first encoded three-dimensional data by encoding the second three-dimensional data.

[0091] Accordingly, the three-dimensional data encoding method generates first encoded three-dimensional data obtained by encoding data having a feature amount equal to or greater than the threshold. In this way, compared with the case of directly encoding the first three-dimensional data, the data amount of the encoded three-dimensional data can be reduced. Therefore, the three-dimensional data encoding method can reduce the data amount during transmission.

[0092] For example, it may also be that the three-dimensional data encoding method further includes a second encoding step, in which second encoded three-dimensional data is generated by encoding the first three-dimensional data.

[0093] Accordingly, the three-dimensional data encoding method can selectively transmit the first encoded three-dimensional data and the second encoded three-dimensional data according to the usage purpose, etc.

[0094] For example, it may also be that the second three-dimensional data is encoded by a first encoding method, and the first three-dimensional data is encoded by a second encoding method different from the first encoding method.

[0095] Accordingly, the three-dimensional data encoding method can adopt appropriate encoding methods for the first three-dimensional data and the second three-dimensional data respectively.

[0096] For example, it may also be that in the first encoding method, inter-frame prediction in intra-frame prediction and inter-frame prediction is prioritized compared with the second encoding method.

[0097] Accordingly, the three-dimensional data encoding method can increase the priority of inter-frame prediction for the second three-dimensional data where the correlation between adjacent data is likely to be low.

[0098] For example, it may also be that the representation methods of three-dimensional positions are different in the first encoding method and the second encoding method.

[0099] Accordingly, the three-dimensional data encoding method can adopt a more appropriate representation method of three-dimensional positions for three-dimensional data with different numbers of data.

[0100] For example, it may also be that at least one of the first encoded three-dimensional data and the second encoded three-dimensional data includes an identifier that indicates whether the encoded three-dimensional data is obtained by encoding the first three-dimensional data or is obtained by encoding a part of the first three-dimensional data.

[0101] Accordingly, the decoding device can easily determine whether the obtained encoded three-dimensional data is the first encoded three-dimensional data or the second encoded three-dimensional data.

[0102] For example, it may also be that in the first encoding step, the second three-dimensional data is encoded in such a way that the data amount of the first encoded three-dimensional data becomes smaller than the data amount of the second encoded three-dimensional data.

[0103] Accordingly, this three-dimensional data encoding method can make the data amount of the first encoded three-dimensional data smaller than the data amount of the second encoded three-dimensional data.

[0104] For example, it may also be that in the extraction step, data corresponding to an object having a predetermined attribute is further extracted from the first three-dimensional data as the second three-dimensional data.

[0105] Accordingly, this three-dimensional data encoding method can generate the first encoded three-dimensional data including the data required by the decoding device.

[0106] For example, it may also be that the three-dimensional data encoding method further includes a sending step in which, according to the state of the client, one of the first encoded three-dimensional data and the second encoded three-dimensional data is sent to the client.

[0107] Accordingly, this three-dimensional data encoding method can send appropriate data according to the state of the client.

[0108] For example, it may also be that the state of the client includes the communication status of the client or the moving speed of the client.

[0109] For example, it may also be that the three-dimensional data encoding method further includes a sending step in which, according to the request of the client, one of the first encoded three-dimensional data and the second encoded three-dimensional data is sent to the client.

[0110] Accordingly, this three-dimensional data encoding method can send appropriate data according to the request of the client.

[0111] Moreover, a three-dimensional data decoding method according to one aspect of the present application includes: a first decoding step of decoding first encoded three-dimensional data by a first decoding method, the first encoded three-dimensional data being obtained by encoding second three-dimensional data whose feature amount extracted from first three-dimensional data is equal to or greater than a threshold value; and a second decoding step of decoding second encoded three-dimensional data obtained by encoding the first three-dimensional data by a second decoding method different from the first decoding method.

[0112] Accordingly, the three-dimensional data decoding method can selectively receive, for example, according to the usage purpose, etc., the first encoded three-dimensional data obtained by encoding data whose feature amount is equal to or greater than the threshold value, and the second encoded three-dimensional data. Accordingly, the three-dimensional data decoding method can reduce the amount of data during transmission. Moreover, the three-dimensional data decoding method can adopt appropriate decoding methods for the first three-dimensional data and the second three-dimensional data respectively.

[0113] For example, it may be that in the first decoding method, inter-frame prediction among intra-frame prediction and inter-frame prediction is prioritized over the second decoding method.

[0114] Accordingly, the three-dimensional data decoding method can increase the priority of inter-frame prediction for the second three-dimensional data where the correlation between adjacent data is likely to be low.

[0115] For example, it may be that the representation methods of three-dimensional positions are different in the first decoding method and the second decoding method.

[0116] Accordingly, the three-dimensional data decoding method can adopt a more appropriate representation method of three-dimensional positions for three-dimensional data with different numbers of data.

[0117] For example, it may be that at least one of the first encoded three-dimensional data and the second encoded three-dimensional data includes an identifier indicating whether the encoded three-dimensional data is obtained by encoding the first three-dimensional data or by encoding a part of the first three-dimensional data, and the first encoded three-dimensional data and the second encoded three-dimensional data are identified with reference to the identifier.

[0118] Accordingly, the three-dimensional data decoding method can easily determine whether the obtained encoded three-dimensional data is the first encoded three-dimensional data or the second encoded three-dimensional data.

[0119] For example, it may be that the three-dimensional data decoding method further includes: a notification step of notifying the state of the client to the server; and a receiving step of receiving, according to the state of the client, one of the first encoded three-dimensional data and the second encoded three-dimensional data sent from the server.

[0120] Accordingly, the three-dimensional data decoding method can receive appropriate data according to the state of the client.

[0121] For example, it may also be that the state of the client includes the communication status of the client or the moving speed of the client.

[0122] For example, it may also be that the three-dimensional data decoding method further includes: a request step of requesting, from a server, one of the first encoded three-dimensional data and the second encoded three-dimensional data; and a receiving step of receiving, according to the request, one of the first encoded three-dimensional data and the second encoded three-dimensional data sent from the server.

[0123] Accordingly, the three-dimensional data decoding method can receive appropriate data corresponding to the use.

[0124] Moreover, a three-dimensional data encoding device according to an aspect of the present application includes: an extraction unit that extracts second three-dimensional data having a feature amount equal to or greater than a threshold value from first three-dimensional data; and a first encoding unit that generates first encoded three-dimensional data by encoding the second three-dimensional data.

[0125] Accordingly, the three-dimensional data encoding device generates first encoded three-dimensional data obtained by encoding data having a feature amount equal to or greater than the threshold value. Accordingly, compared with the case of directly encoding the first three-dimensional data, the amount of data can be reduced. Therefore, the three-dimensional data encoding device can reduce the amount of data during transmission.

[0126] Moreover, a three-dimensional data decoding device according to an aspect of the present application includes: a first decoding unit that decodes first encoded three-dimensional data by using a first decoding method, the first encoded three-dimensional data being obtained by encoding second three-dimensional data having a feature amount equal to or greater than a threshold value extracted from first three-dimensional data; and a second decoding unit that decodes second encoded three-dimensional data obtained by encoding the first three-dimensional data by using a second decoding method different from the first decoding method.

[0127] Accordingly, the three-dimensional data decoding device can selectively receive, for example, according to the use purpose, etc., the first encoded three-dimensional data and the second encoded three-dimensional data obtained by encoding data having a feature amount equal to or greater than the threshold value. Accordingly, the three-dimensional data decoding device can reduce the amount of data during transmission. Moreover, the three-dimensional data decoding device can adopt appropriate decoding methods for the first three-dimensional data and the second three-dimensional data, respectively.

[0128] Moreover, a method for producing three-dimensional data according to one aspect of the present application includes: a production step of producing first three-dimensional data based on information detected by a sensor; a reception step of receiving encoded three-dimensional data obtained by encoding second three-dimensional data; a decoding step of decoding the received encoded three-dimensional data to obtain the second three-dimensional data; and a synthesis step of producing third three-dimensional data by synthesizing the first three-dimensional data and the second three-dimensional data.

[0129] Accordingly, this method for producing three-dimensional data can produce detailed third three-dimensional data by using the produced first three-dimensional data and the received second three-dimensional data.

[0130] For example, it may also be that in the synthesis step, the third three-dimensional data having a higher density than the first three-dimensional data and the second three-dimensional data is produced by synthesizing the first three-dimensional data and the second three-dimensional data.

[0131] For example, it may also be that the second three-dimensional data is three-dimensional data generated by extracting data with a feature amount equal to or greater than a threshold value from fourth three-dimensional data.

[0132] Accordingly, this method for producing three-dimensional data can reduce the data amount of the three-dimensional data to be transmitted.

[0133] For example, it may also be that the method for producing three-dimensional data further includes a search step. In this search step, as the transmission device that is the source of the encoded three-dimensional data, in the reception step, the encoded three-dimensional data is received from the searched transmission device.

[0134] Accordingly, this method for producing three-dimensional data can, for example, determine the transmission device that holds the required three-dimensional data by searching.

[0135] For example, it may also be that the method for producing three-dimensional data further includes: a determination step of determining a request range, which is the range of the three-dimensional space for which three-dimensional data is requested; and a transmission step of transmitting information indicating the request range to the transmission device, and the second three-dimensional data includes the three-dimensional data within the request range.

[0136] Accordingly, this method for producing three-dimensional data can not only receive the required three-dimensional data, but also reduce the data amount of the three-dimensional data to be transmitted.

[0137] For example, it may also be that in the determination step, the space range including the occlusion area that cannot be detected by the sensor is determined as the request range.

[0138] The 3D data transmission method according to one aspect of the present application includes: a production step of producing 5th 3D data based on information detected by a sensor; an extraction step of producing 6th 3D data by extracting a part of the 5th 3D data; an encoding step of generating encoded 3D data by encoding the 6th 3D data; and a transmission step of transmitting the encoded 3D data.

[0139] Accordingly, this 3D data transmission method can not only transmit the 3D data produced by itself to other devices, but also reduce the amount of data of the 3D data to be transmitted.

[0140] For example, it may also be that in the production step, 7th 3D data is produced based on information detected by the sensor, and the 5th 3D data is produced by extracting data with a feature amount equal to or greater than a threshold value from the 7th 3D data.

[0141] Accordingly, this 3D data transmission method can reduce the amount of data of the 3D data to be transmitted.

[0142] For example, it may also be that the 3D data transmission method further includes a reception step, in which information indicating a request range, which is the range of the 3D space for which 3D data is requested, is received from a receiving device. In the extraction step, the 6th 3D data is produced by extracting the 3D data of the request range from the 5th 3D data. In the transmission step, the encoded 3D data is transmitted to the receiving device.

[0143] Accordingly, this 3D data transmission method can reduce the amount of data of the 3D data to be transmitted.

[0144] Moreover, the 3D data production device according to one aspect of the present application includes: a production unit that produces 1st 3D data based on information detected by a sensor; a reception unit that receives encoded 3D data obtained by encoding 2nd 3D data; a decoding unit that obtains the 2nd 3D data by decoding the received encoded 3D data; and a synthesis unit that produces 3rd 3D data by synthesizing the 1st 3D data and the 2nd 3D data.

[0145] Accordingly, this 3D data production device can produce detailed 3rd 3D data using the produced 1st 3D data and the received 2nd 3D data.

[0146] Furthermore, the three-dimensional data transmission device according to one aspect of the present application includes: a production unit that produces fifth three-dimensional data based on information detected by a sensor; an extraction unit that produces sixth three-dimensional data by extracting a part of the fifth three-dimensional data; an encoding unit that generates encoded three-dimensional data by encoding the sixth three-dimensional data; and a transmission unit that transmits the encoded three-dimensional data.

[0147] Accordingly, this three-dimensional data transmission device can not only transmit the three-dimensional data produced by itself to other devices, but also reduce the amount of data of the three-dimensional data to be transmitted.

[0148] Furthermore, the three-dimensional information processing method according to one aspect of the present application includes: an acquisition step of acquiring map data including first three-dimensional position information via a channel; a generation step of generating second three-dimensional position information based on information detected by a sensor; a determination step of determining whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information; a decision step of deciding on a response operation for the abnormality when it is determined that the first three-dimensional position information or the second three-dimensional position information is abnormal; and a work control step of performing control required for implementing the response operation.

[0149] Accordingly, this three-dimensional information processing method can detect an abnormality in the first three-dimensional position information or the second three-dimensional position information and can perform a response operation.

[0150] For example, it may be that the first three-dimensional position information is encoded in units of partial spaces having three-dimensional coordinate information, the first three-dimensional position information includes a plurality of random access units, each of the plurality of random access units is an aggregate of one or more partial spaces, and can be independently decoded.

[0151] Accordingly, this three-dimensional information processing method can reduce the amount of data of the acquired first three-dimensional position information.

[0152] For example, it may be that the first three-dimensional position information is data in which feature points where the three-dimensional feature amount exceeds a specified threshold are encoded.

[0153] Accordingly, this three-dimensional information processing method can reduce the amount of data of the acquired first three-dimensional position information.

[0154] For example, it may be that in the determination step, it is determined whether the first three-dimensional position information can be acquired via the channel, and when the first three-dimensional position information cannot be acquired via the channel, the first three-dimensional position information is determined to be abnormal.

[0155] Accordingly, in the case where the first three-dimensional position information cannot be constituted according to the communication status or the like, the three-dimensional information processing method performs appropriate response operations.

[0156] For example, it may also be that the three-dimensional information processing method further includes a self-position estimation step. In the self-position estimation step, the self-position of the moving body having the sensor is estimated by using the first three-dimensional position information and the second three-dimensional position information. In the determination step, it is predicted whether the moving body will enter a region with poor communication status. In the operation control step, in the case where it is predicted that the moving body will enter a region with poor communication status, before the moving body enters this region, the moving body obtains the first three-dimensional information.

[0157] Accordingly, in the case where there is a possibility that the first three-dimensional position information cannot be obtained, the three-dimensional information processing method can obtain the first three-dimensional position information in advance.

[0158] For example, it may also be that in the operation control step, in the case where the first three-dimensional position information cannot be obtained via the channel, the third three-dimensional position information with a narrower range than the first three-dimensional position information is obtained via the channel.

[0159] Accordingly, the three-dimensional information processing method can reduce the amount of data obtained via the channel, and can obtain three-dimensional position information even in the case of poor communication status.

[0160] For example, it may also be that the three-dimensional information processing method further includes a self-position estimation step. In this self-position estimation step, the self-position of the moving body having the sensor is estimated by using the first three-dimensional position information and the second three-dimensional position information. In the operation control step, in the case where the first three-dimensional position information cannot be obtained via the channel, map data including two-dimensional position information is obtained via the channel, and the self-position of the moving body having the sensor is estimated by using the two-dimensional position information and the second three-dimensional position information.

[0161] Accordingly, the three-dimensional information processing method can reduce the amount of data obtained via the channel, and thus can obtain three-dimensional position information even in the case of poor communication status.

[0162] For example, it may also be that the three-dimensional information processing method further includes an automatic driving step. In this automatic driving step, the moving body performs automatic driving by using the result of the self-position estimation. In the determination step, further, according to the moving environment of the moving body, it is determined whether to perform the automatic driving of the moving body by using the result of the self-position estimation of the moving body performed according to the two-dimensional position information and the second three-dimensional position information.

[0163] Accordingly, the three-dimensional information processing method can determine whether to continue with autonomous driving according to the moving environment of the moving body.

[0164] For example, it may also be that the three-dimensional information processing method further includes an autonomous driving step, in which, using the result of the self-position estimation, the moving body is made to perform autonomous driving, and in the work control step, according to the moving environment of the moving body, the mode of the autonomous driving is switched.

[0165] Accordingly, the three-dimensional information processing method can set an appropriate autonomous driving mode according to the moving environment of the moving body.

[0166] For example, it may also be that in the determination step, it is determined whether the data of the first three-dimensional position information is complete, and in the case where the data of the first three-dimensional position information is incomplete, the first three-dimensional position information is determined to be abnormal.

[0167] Accordingly, the three-dimensional information processing method can perform appropriate response operations, for example, in the case where the first three-dimensional position information is damaged.

[0168] For example, in the determination step, it is determined whether the generation accuracy of the data of the second three-dimensional position information is equal to or higher than the reference value, and in the case where the generation accuracy of the data of the second three-dimensional position information is not equal to or higher than the reference value, the second three-dimensional position information is determined to be abnormal.

[0169] Accordingly, the three-dimensional information processing method can perform appropriate response operations in the case where the accuracy of the second three-dimensional position information is low.

[0170] For example, it may also be that in the work control step, in the case where the generation accuracy of the data of the second three-dimensional position information is not equal to or higher than the reference value, the fourth three-dimensional position information is generated according to the information detected by a substitute sensor different from the sensor.

[0171] Accordingly, the three-dimensional information processing method can obtain three-dimensional position information using a substitute sensor, for example, in the case where the sensor fails.

[0172] For example, it may also be that the three-dimensional information processing method further includes: a self-position estimation step of estimating the self-position of the moving body having the sensor using the first three-dimensional position information and the second three-dimensional position information; and an autonomous driving step of making the moving body perform autonomous driving using the result of the self-position estimation, and in the work control step, in the case where the generation accuracy of the data of the second three-dimensional position information is not equal to or higher than the reference value, the mode of the autonomous driving is switched.

[0173] Accordingly, when the accuracy of the second three-dimensional position information is low, the three-dimensional information processing method performs appropriate response operations.

[0174] For example, it may also be that, in the operation control step, when the generation accuracy of the data of the second three-dimensional position information is not above the reference value, the operation of the sensor is corrected.

[0175] Accordingly, when the accuracy of the second three-dimensional position information is low, the three-dimensional information processing method can improve the accuracy of the second three-dimensional position information.

[0176] Moreover, the three-dimensional information processing apparatus according to one aspect of the present application includes: an acquisition unit that acquires map data including first three-dimensional position information via a channel; a generation unit that generates second three-dimensional position information based on information detected by a sensor; a determination unit that determines whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information; a decision unit that determines a response operation for the abnormality when it is determined that the first three-dimensional position information or the second three-dimensional position information is abnormal; and an operation control unit that executes control required for the implementation of the response operation.

[0177] Accordingly, the three-dimensional information processing apparatus can detect an abnormality in the first three-dimensional position information or the second three-dimensional position information and can perform a response operation.

[0178] Moreover, the three-dimensional data production method according to one aspect of the present application is applied to a moving body including a sensor and a communication unit that transmits and receives three-dimensional data to and from the outside. The three-dimensional data production method includes: a three-dimensional data production step of producing second three-dimensional data based on information detected by the sensor and first three-dimensional data received by the communication unit; and a transmission step of transmitting third three-dimensional data, which is part of the second three-dimensional data, to the outside.

[0179] Accordingly, the three-dimensional data production method can generate three-dimensional data in a range that cannot be detected by the moving body. That is, the three-dimensional data production method can generate detailed three-dimensional data. Moreover, the three-dimensional data production method can transmit three-dimensional data in a range that cannot be detected by other moving bodies or the like to those other moving bodies or the like.

[0180] For example, it may also be that the three-dimensional data production step and the transmission step are repeatedly executed, and the third three-dimensional data is three-dimensional data of a small space having a predetermined size, and the small space is located at a position that is a predetermined distance in front of the current position of the moving body in the moving direction of the moving body.

[0181] Accordingly, the data volume of the third 3D data to be transmitted can be reduced.

[0182] For example, the specified distance may vary according to the moving speed of the moving body.

[0183] Accordingly, this method for producing 3D data can set an appropriate small space according to the moving speed of the moving body, and can transmit the 3D data of this small space to other moving bodies or the like.

[0184] For example, the specified size may vary according to the moving speed of the moving body.

[0185] Accordingly, this method for producing 3D data can set an appropriate small space according to the moving speed of the moving body, and can transmit the 3D data of this small space to other moving bodies or the like.

[0186] For example, the method for producing 3D data may further determine whether the second 3D data of the small space corresponding to the third 3D data that has been transmitted has changed. If it has changed, the fourth 3D data, which is at least a part of the changed second 3D data, is transmitted to the outside.

[0187] Accordingly, this method for producing 3D data can transmit the fourth 3D data of the changed space to other moving bodies or the like.

[0188] For example, the fourth 3D data may be transmitted prior to the third 3D data.

[0189] Accordingly, since this method for producing 3D data can preferentially transmit the fourth 3D data of the changed space to other moving bodies or the like, other moving bodies or the like can, for example, quickly make judgments based on the 3D data.

[0190] For example, in the case of such a change, the fourth 3D data is transmitted first before transmitting the third 3D data.

[0191] For example, the fourth 3D data may show the difference between the second 3D data of the small space corresponding to the transmitted third 3D data and the changed second 3D data.

[0192] Accordingly, this method for producing 3D data can reduce the data volume of the 3D data to be transmitted.

[0193] For example, in the transmitting step, if there is no difference between the third 3D data of the small space and the first 3D data of the small space, the third 3D data is not transmitted.

[0194] Accordingly, the data amount of the third three-dimensional data to be transmitted can be reduced.

[0195] For example, it may also be that, in the sending step, when there is no difference between the third three-dimensional data of the small space and the first three-dimensional data of the small space, information indicating that there is no difference between the third three-dimensional data of the small space and the first three-dimensional data of the small space is sent to the outside.

[0196] For example, it may also be that the information detected by the sensor is three-dimensional data.

[0197] Moreover, the three-dimensional data production device according to one aspect of the present application is mounted on a moving body, and the three-dimensional data production device includes: a sensor; a receiving unit that receives first three-dimensional data from the outside; a production unit that produces second three-dimensional data based on the information detected by the sensor and the first three-dimensional data; and a sending unit that sends third three-dimensional data, which is a part of the second three-dimensional data, to the outside.

[0198] Accordingly, the three-dimensional data production device can generate three-dimensional data in a range that cannot be detected by the moving body. That is, the three-dimensional data production device can generate detailed three-dimensional data. Moreover, the three-dimensional data production device can send three-dimensional data in a range that cannot be detected by other moving bodies or the like to these other moving bodies or the like.

[0199] In addition, these general or specific aspects can be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and can be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0200] Hereinafter, embodiments will be specifically described with reference to the drawings. In addition, all the embodiments to be described below are specific examples showing one aspect of the present application. The numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of the constituent elements, steps, order of steps, etc. shown in the following embodiments are all examples, and the gist thereof is not to limit the present application. Moreover, among the constituent elements of the following embodiments, the constituent elements not described in the technical solution showing the uppermost concept are described as arbitrary constituent elements.

[0201] (Embodiment 1)

[0202] First, the data structure of the encoded three-dimensional data (hereinafter also referred to as encoded data) according to the present embodiment will be described. Figure 1 The configuration of the encoded three-dimensional data according to the present embodiment is shown.

[0203] In the present embodiment, a three-dimensional space is divided into spaces (SPCs) corresponding to pictures in the encoding of moving images, and three-dimensional data is encoded in units of space. The space is further divided into volumes (VLMs) corresponding to macroblocks and the like in the encoding of moving images, and prediction and transformation are performed in units of VLM. A volume includes a plurality of voxels (VXLs) which are the smallest units corresponding to position coordinates. In addition, prediction means, similar to the prediction performed in two-dimensional images, referring to other processing units, generating predicted three-dimensional data similar to the processing unit of the object to be processed, and encoding the difference between the predicted three-dimensional data and the processing unit of the object to be processed. Moreover, this prediction includes not only spatial prediction referring to other prediction units at the same time but also temporal prediction referring to prediction units at different times.

[0204] For example, when a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes a three-dimensional space represented by point cloud data or the like of a point group, according to the size of voxels, each point of the point group or a plurality of points included in a voxel is encoded together. If the voxels are subdivided, the three-dimensional shape of the point group can be represented with high precision, and if the size of the voxels is increased, the three-dimensional shape of the point group can be represented roughly.

[0205] In addition, although the case where the three-dimensional data is point cloud data is described as an example below, the three-dimensional data is not limited to point cloud data and can be three-dimensional data in any form.

[0206] Moreover, hierarchical voxels can be used. In this case, in the n-th hierarchy, it is possible to sequentially show whether there are sampling points in the hierarchies below the (n - 1)-th hierarchy (the lower layer of the n-th hierarchy). For example, when only the n-th hierarchy is decoded, when there are sampling points in the hierarchies below the (n - 1)-th hierarchy, it is possible to perform decoding by regarding that there are sampling points at the center of the voxels in the n-th hierarchy.

[0207] Moreover, the encoding device obtains point cloud data through a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor, etc.

[0208] Regarding the space, similar to the encoding of moving images, it is at least classified into any one of the following three prediction structures: an intra-frame space (I-SPC) that can be decoded independently, a prediction space (P-SPC) that can only be referred to unidirectionally, and a bidirectional space (B-SPC) that can be referred to bidirectionally. Moreover, the space has two types of time information: a decoding time and a display time.

[0209] Moreover, as Figure 1As shown, as a processing unit including multiple spaces, there is a GOS (Group Of Space) which is a random access unit. Moreover, as a processing unit including multiple GOSs, there is a world space (WLD).

[0210] The space area occupied by the world space is associated with the absolute position on the earth through GPS or latitude and longitude information, etc. This position information is stored as meta information. In addition, the meta information can be included in the encoded data or transmitted separately from the encoded data.

[0211] Also, within a GOS, all SPCs can be three-dimensionally adjacent, or there can be SPCs that are not three-dimensionally adjacent to other SPCs.

[0212] In addition, hereinafter, processes such as encoding, decoding, or referencing corresponding to the three-dimensional data included in processing units such as GOS, SPC, or VLM are also simply referred to as encoding, decoding, or referencing the processing unit. And the three-dimensional data included in the processing unit includes at least one group of a spatial position such as three-dimensional coordinates and characteristic values such as color information, for example.

[0213] Next, the prediction structure of SPCs in a GOS will be described. Multiple SPCs within the same GOS, or multiple VLMs within the same SPC, although occupying different spaces from each other, hold the same time information (decoding time and display time).

[0214] Also, within a GOS, the SPC that is the first in the decoding order is an I-SPC. And there are two types of GOSs in a GOS: a closed GOS and an open GOS. A closed GOS is a GOS that can decode all SPCs within the GOS when starting to decode from the first I-SPC. In an open GOS, within the GOS, a part of the SPCs whose display time is earlier than that of the first I-SPC refer to different GOSs and can only be decoded in that GOS.

[0215] In addition, in the encoded data such as map information, there is a case where the WLD is decoded in the direction opposite to the encoding order. If there is a dependency between GOSs, it is difficult to perform reverse regeneration. Therefore, in this case, a closed GOS is basically adopted.

[0216] Also, a GOS has a layer structure in the height direction, and encoding or decoding is performed sequentially starting from the SPCs in the bottom layer.

[0217] Figure 2 An example of the prediction structure between SPCs belonging to the bottommost layer of a GOS is shown. Figure 3 An example of the prediction structure between layers is shown.

[0218] There is more than one I-SPC in the GOS. Although there are objects such as people, animals, cars, bicycles, traffic lights, or buildings that serve as land marks in the three-dimensional space, it is effective especially when encoding small-sized objects as I-SPCs. For example, when a three-dimensional data decoding device (hereinafter also referred to as the decoding device) decodes the GOS with a low processing volume or at high speed, it only decodes the I-SPCs in the GOS.

[0219] Moreover, the encoding device can switch the encoding interval or the occurrence frequency of the I-SPC according to the density of the objects in the WLD.

[0220] And, in Figure 3 In the configuration shown, the encoding device or the decoding device performs encoding or decoding on multiple layers sequentially starting from the lower layer (layer 1). Accordingly, for example, for an automatically moving vehicle or the like, it is possible to give higher priority to the data near the ground where there is a large amount of information.

[0221] In addition, in the encoded data used in a drone or the like, in the GOS, encoding or decoding can be performed sequentially starting from the SPC of the upper layer in the height direction.

[0222] Moreover, the encoding device or the decoding device can also perform encoding or decoding on multiple layers in such a way that the decoding device roughly grasps the GOS and can gradually increase the resolution. For example, the encoding device or the decoding device can perform encoding or decoding in the order of layer 3, 8, 1, 9...

[0223] Next, a method for corresponding static objects and dynamic objects will be described.

[0224] In the three-dimensional space, there are static objects or scenes such as buildings or roads (hereinafter collectively referred to as static objects), and dynamic objects such as vehicles or people (hereinafter referred to as dynamic objects). Detection of the objects can be performed separately by extracting feature points from the data of the point cloud data, or the images captured by a stereo camera or the like. Here, an example of the encoding method for dynamic objects will be described.

[0225] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects by identification information.

[0226] For example, the GOS is used as an identification unit. In this case, the GOS including the SPCs constituting the static object and the GOS including the SPCs constituting the dynamic object are distinguished in the encoded data, or by identification information stored separately from the encoded data.

[0227] Alternatively, the SPC is used as the identification unit. In this case, only the SPCs that constitute the VLMs of the static object and the SPCs that include the VLMs that constitute the dynamic object are distinguished by the above identification information.

[0228] Alternatively, the VLM or VXL can be used as the identification unit. In this case, the VLMs or VXLs that include static objects and the VLMs or VXLs that include dynamic objects are distinguished by the above identification information.

[0229] Moreover, the encoding device can encode the dynamic object as one or more VLMs or SPCs, and encode the VLMs or SPCs that include static objects and the SPCs that include dynamic objects as different GOSs. Moreover, when the size of the GOS becomes variable according to the size of the dynamic object, the encoding device stores the size of the GOS as meta-information separately.

[0230] Moreover, the encoding device encodes the static object and the dynamic object independently of each other, and for the world space composed of static objects, the dynamic object can be overlapped. At this time, the dynamic object is composed of one or more SPCs, and each SPC corresponds to one or more SPCs of the static object that overlaps the SPC. In addition, the dynamic object may not be represented by SPCs, but may be represented by one or more VLMs or VXLs.

[0231] Moreover, the encoding device can encode the static object and the dynamic object as different streams.

[0232] Moreover, the encoding device can also generate a GOS that includes one or more SPCs that constitute the dynamic object. Moreover, the encoding device can set the GOS (GOS_M) that includes the dynamic object and the GOS of the static object corresponding to the spatial region of GOS_M to have the same size (occupy the same spatial region). In this way, the overlapping process can be performed in units of GOS.

[0233] The P-SPC or B-SPC that constitutes the dynamic object can also refer to the SPCs included in different encoded GOSs. The position of the dynamic object changes over time. In the case where the same dynamic object is encoded as GOSs at different times, the cross-GOS reference is effective from the viewpoint of the compression ratio.

[0234] Moreover, it is also possible to switch between the above first method and second method according to the use of the encoded data. For example, when the encoded three-dimensional data is applied as a map, since it is desired to be separated from the dynamic object, the encoding device adopts the second method. In addition, when the encoding device encodes the three-dimensional data of activities such as concerts or sports, if there is no need to separate the dynamic object, the first method is adopted.

[0235] Moreover, the decoding time and display time of GOS or SPC can be stored in the encoded data or as meta-information. Also, the time information of static objects can all be the same. In this case, the actual decoding time and display time can be determined by the decoding device. Alternatively, as the decoding time, different values can be assigned for each GOS or SPC, and as the display time, the same value can be assigned to all. Also, as shown in the decoder mode in video coding such as HRD (Hypothetical Reference Decoder) of HEVC, the decoder has a buffer of a specified size. As long as the bitstream is read at a specified bit rate according to the decoding time, a model that will not be damaged and can be guaranteed to be decoded can be imported.

[0236] Next, the configuration of GOS in the world space will be described. The coordinates of the three-dimensional space in the world space are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, z-axis). By setting a specified rule in the encoding order of GOS, GOS that are adjacent in space can be encoded continuously in the encoded data. For example, in Figure 4 the example shown, the GOS in the xz plane are encoded continuously. After the encoding of all GOS in one xz plane is completed, the value of the y-axis is updated. That is, as the encoding progresses, the world space extends in the y-axis direction. Also, the index number of GOS is set as the encoding order.

[0237] Here, the three-dimensional space of the world space corresponds one-to-one with GPS or geographical absolute coordinates such as latitude and longitude. Alternatively, the three-dimensional space can be represented by the relative position with respect to a preset reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, etc., and this direction vector is stored together with the encoded data as meta-information.

[0238] Moreover, the size of GOS is set to be fixed, and the encoding device stores this size as meta-information. Also, the size of GOS can be switched according to, for example, whether it is in the city or indoors or outdoors. That is, the size of GOS can be switched according to the amount or nature of the object having the value as information. Alternatively, the encoding device can appropriately switch the size of GOS or the interval of I-SPC within GOS according to the density of the object, etc. in the same world space. For example, the encoding device sets the size of GOS to be smaller and the interval of I-SPC within GOS to be shorter when the density of the object is higher.

[0239] In Figure 5In the example, in the region from the 3rd to the 10th GOS, due to the high density of objects, in order to achieve random access with a fine granularity, the GOS is subdivided. Also, from the 7th to the 10th GOS are respectively located on the back side of the 3rd to 6th GOS.

[0240] Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be described. Figure 6 FIG. is a block diagram of the three-dimensional data encoding device 100 according to this embodiment. Figure 7 FIG. is a flowchart showing an operation example of the three-dimensional data encoding device 100.

[0241] Figure 6 The three-dimensional data encoding device 100 shown generates encoded three-dimensional data 112 by encoding three-dimensional data 111. This three-dimensional data encoding device 100 includes: an acquisition unit 101, an encoding region determination unit 102, a division unit 103, and an encoding unit 104.

[0242] As Figure 7 shown, first, the acquisition unit 101 acquires three-dimensional data 111 as point cloud data (S101).

[0243] Next, the encoding region determination unit 102 determines the region to be encoded from the spatial region corresponding to the acquired point cloud data (S102). For example, the encoding region determination unit 102 determines the spatial region around that position as the region to be encoded according to the position of the user or the vehicle.

[0244] Next, the division unit 103 divides the point cloud data included in the region to be encoded into respective processing units. Here, the processing units are the above-mentioned GOS and SPC, etc. And the region to be encoded corresponds to the above-mentioned world space, for example. Specifically, the division unit 103 divides the point cloud data into processing units according to the size of the GOS set in advance, the presence or absence or size of dynamic objects (S103). And the division unit 103 determines the start position of the SPC that becomes the beginning in the encoding order in each GOS.

[0245] Next, the encoding unit 104 generates encoded three-dimensional data 112 by sequentially encoding a plurality of SPCs in each GOS (S104).

[0246] In addition, here, after dividing the region to be encoded into GOS and SPC, although an example of encoding each GOS is shown, the order of processing is not limited to the above. For example, after determining the configuration of one GOS, it is possible to encode that GOS, and then determine the configuration of the GOS, etc. in this order.

[0247] In this way, the three-dimensional data encoding device 100 generates the encoded three-dimensional data 112 by encoding the three-dimensional data 111. Specifically, the three-dimensional data encoding device 100 divides the three-dimensional data into random access units, that is, divides it into first processing units (GOS) corresponding to three-dimensional coordinates respectively, divides the first processing units (GOS) into a plurality of second processing units (SPC), and divides the second processing units (SPC) into a plurality of third processing units (VLM). Moreover, the third processing unit (VLM) includes one or more voxels (VXL), and the voxel (VXL) is the smallest unit corresponding to the position information.

[0248] Next, the three-dimensional data encoding device 100 generates the encoded three-dimensional data 112 by encoding each of the plurality of first processing units (GOS). Specifically, the three-dimensional data encoding device 100 encodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Moreover, the three-dimensional data encoding device 100 encodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0249] For example, when the first processing unit (GOS) of the processing object is a closed GOS, for the second processing unit (SPC) of the processing object included in the first processing unit (GOS) of the processing object, encoding is performed with reference to other second processing units (SPC) included in the first processing unit (GOS) of the processing object. That is, the three-dimensional data encoding device 100 does not refer to the second processing units (SPC) included in the first processing units (GOS) different from the first processing unit (GOS) of the processing object.

[0250] Moreover, when the first processing unit (GOS) of the processing object is an open GOS, for the second processing unit (SPC) of the processing object included in the first processing unit (GOS) of the processing object, encoding is performed with reference to other second processing units (SPC) included in the first processing unit (GOS) of the processing object or the second processing units (SPC) included in the first processing units (GOS) different from the first processing unit (GOS) of the processing object.

[0251] Moreover, the three-dimensional data encoding device 100 selects one from the first type (I-SPC) of not referring to other second processing units (SPC), the second type (P-SPC) of referring to one other second processing unit (SPC), and the third type of referring to two other second processing units (SPC) as the type of the second processing unit (SPC) of the processing object, and encodes the second processing unit (SPC) of the processing object according to the selected type.

[0252] Next, the configuration and operation process of the three-dimensional data decoding device according to this embodiment will be described. Figure 8 is a block diagram of the three-dimensional data decoding device 200 according to this embodiment. Figure 9 is a flowchart showing an operation example of the three-dimensional data decoding device 200.

[0253] Figure 8 The three-dimensional data decoding device 200 shown generates decoded three-dimensional data 212 by decoding the encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. The three-dimensional data decoding device 200 includes: an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0254] First, the acquisition unit 201 acquires the encoded three-dimensional data 211 (S201). Next, the decoding start GOS determination unit 202 determines the GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to the meta information in the encoded three-dimensional data 211 or stored separately from the encoded three-dimensional data, and determines the GOS including the spatial position, object, or SPC corresponding to the time at which decoding starts as the GOS to be decoded.

[0255] Next, the decoding SPC determination unit 203 determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the decoding SPC determination unit 203 determines (1) whether to decode only I-SPC, (2) whether to decode I-SPC and P-SPC, and (3) whether to decode all types. In addition, in the case where the type of the SPC to be decoded, such as all SPCs, is specified in advance, this step may not be performed.

[0256] Next, the decoding unit 204 acquires the SPC that is the first in the decoding order (the same as the encoding order) within the GOS, the address position where it starts in the encoded three-dimensional data 211, acquires the encoded data of the first SPC from this address position, and decodes each SPC in sequence from the first SPC (S204). And the above address position is stored in meta information or the like.

[0257] In this way, the three-dimensional data decoding device 200 decodes the decoded three-dimensional data 212. Specifically, the three-dimensional data decoding device 200 generates the decoded three-dimensional data 212 of the first processing unit (GOS) as a random access unit by decoding each of the encoded three-dimensional data 211 of the first processing unit (GOS) corresponding to the three-dimensional coordinates respectively. More specifically, the three-dimensional data decoding device 200 decodes each of the plurality of second processing units (SPC) in each of the first processing units (GOS). Further, the three-dimensional data decoding device 200 decodes each of the plurality of third processing units (VLM) in each of the second processing units (SPC).

[0258] The meta information for random access will be described below. This meta information is generated by the three-dimensional data encoding device 100 and is included in the encoded three-dimensional data 112 (211).

[0259] In the random access of conventional two-dimensional moving images, decoding starts from the first frame of the random access unit near the specified time. However, in the world space, random access is also envisioned for (coordinates, objects, etc.) in addition to time.

[0260] Therefore, in order to achieve random access to at least the three elements of coordinates, objects, and time, a table in which the index numbers of each element are associated with the GOS is prepared. Moreover, the index number of the GOS is associated with the address of the I-SPC that is the start of the GOS. Figure 10 An example of the table included in the meta information is shown. Additionally, it is not necessary to use Figure 10 all of the shown tables, and at least one table can be used.

[0261] As an example below, random access starting from coordinates will be described. When accessing the coordinates (x2, y2, z2), first referring to the coordinate-GOS table, it can be known that the location with the coordinates (x2, y2, z2) is included in the second GOS. Then, referring to the GOS address table, since it can be known that the address of the I-SPC at the start of the second GOS is addr(2), the decoding unit 204 obtains data from this address and starts decoding.

[0262] In addition, the address can be an address in the logical format or a physical address of an HDD or a memory. Also, information for determining a file segment can be used instead of the address. For example, a file segment is a unit obtained by segmenting one or more GOSs, etc.

[0263] Also, when the object spans multiple GOSs, the GOSs to which the multiple objects belong can also be shown in the object GOS table. If the multiple GOSs are closed GOSs, the encoding device and the decoding device can perform encoding or decoding in parallel. Additionally, if the multiple GOSs are open GOSs, by cross-referencing between the multiple GOSs, the compression efficiency can be further improved.

[0264] Examples of the object include a person, an animal, a car, a bicycle, a traffic signal, or a building serving as a land mark. For example, when the three-dimensional data encoding device 100 encodes in the world space, it extracts feature points unique to the object from three-dimensional point cloud data or the like, detects the object based on the feature points, and can set the detected object as a random access point.

[0265] In this way, the three-dimensional data encoding device 100 generates first information that shows multiple first processing units (GOSs) and three-dimensional coordinates corresponding to each of the multiple first processing units (GOSs). And encoding the three-dimensional data 112(211) includes this first information. And the first information further shows at least one of the object, the time, and the data storage destination corresponding to each of the multiple first processing units (GOSs).

[0266] The three-dimensional data decoding device 200 obtains the first information from the encoded three-dimensional data 211, uses the first information to determine the encoded three-dimensional data 211 of the first processing unit corresponding to the specified three-dimensional coordinates, object, or time, and decodes the encoded three-dimensional data 211.

[0267] Examples of other meta-information will be described below. In addition to the meta-information for random access, the three-dimensional data encoding device 100 can also generate and store the following meta-information. And the three-dimensional data decoding device 200 can also use this meta-information during decoding.

[0268] In cases such as when using the three-dimensional data as map information, a profile is specified according to the use, and the information showing the profile can be included in the meta-information. For example, a profile for urban areas or suburbs is specified, or a profile for flying objects is specified, and the maximum or minimum size of the world space, SPC, or VLM is defined respectively. For example, in the profile for urban areas, more detailed information is required than in the suburbs, so the minimum size of the VLM is set smaller.

[0269] The meta-information may also include a tag value indicating the type of the object. The tag value corresponds to the VLM, SPC, or GOS constituting the object. The tag value can be set according to the type of the object, etc. For example, the tag value "0" represents "person", the tag value "1" represents "car", and the tag value "2" represents "traffic signal". Alternatively, in a case where it is difficult to determine or unnecessary to determine the type of the object, a tag value indicating properties such as size, or whether the object is a dynamic object or a static object may be used.

[0270] Furthermore, the meta-information may also include information indicating the range of the spatial region occupied by the world space.

[0271] Furthermore, the meta-information may store the size of the SPC or VXL as the entire stream of encoded data or as header information shared by multiple SPCs such as SPCs within the GOS.

[0272] Furthermore, the meta-information may also include identification information such as a distance sensor or a camera used in the generation of the point cloud data, or may include information indicating the position accuracy of the point group within the point cloud data.

[0273] Furthermore, the meta-information may include information indicating whether the world space is composed of only static objects or contains dynamic objects.

[0274] A modification example of the present embodiment will be described below.

[0275] The encoding device or the decoding device may encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel can be determined based on the meta-information indicating the spatial position of the GOSs, etc.

[0276] In a case where the three-dimensional data is used as a spatial map when a vehicle or a flying object moves, or in a case where such a spatial map is generated, etc., the encoding device or the decoding device may encode or decode the GOS or SPC included in the space determined based on GPS, path information, or zoom ratio, etc.

[0277] Furthermore, the decoding device may also start decoding sequentially from the space close to its own position or the walking path. The encoding device or the decoding device may also perform encoding or decoding by making the priority of the space far from its own position or the walking path lower than that of the close space. Here, reducing the priority means reducing the processing order, reducing the resolution (post-filtering), or reducing the image quality (improving the encoding efficiency. For example, increasing the quantization step size), etc.

[0278] Furthermore, when decoding the encoded data hierarchically encoded within the space, the decoding device may also decode only the lower hierarchy.

[0279] Further, the decoding device may also start decoding from the lower layer according to the zoom ratio or use of the map.

[0280] Further, in applications such as self-position estimation or object recognition performed during the automatic driving of a vehicle or a robot, the encoding device or the decoding device may also reduce the resolution of areas outside the area within a specified height from the road surface (the area to be recognized) for encoding or decoding.

[0281] Further, the encoding device may also encode the point cloud data representing the spatial shapes of the indoor and outdoor spaces independently. For example, by separating the GOS representing the indoor (indoor GOS) from the GOS representing the outdoor (outdoor GOS), the decoding device can select the GOS to be decoded according to the viewpoint position when using the encoded data.

[0282] Further, the encoding device may make the indoor GOS and the outdoor GOS with adjacent coordinates adjacent in the encoding stream for encoding. For example, the encoding device correlates their identifiers and stores the information indicating the correlated identifiers in the encoding stream or in separately stored meta-information. Accordingly, the decoding device can identify the indoor GOS and the outdoor GOS with adjacent coordinates by referring to the information in the meta-information.

[0283] Further, the encoding device may also switch the size of the GOS or SPC between the indoor GOS and the outdoor GOS. For example, the encoding device sets the size of the GOS to be smaller indoors than outdoors. Further, the encoding device may also change the accuracy when extracting feature points from the point cloud data or the accuracy of object detection, etc., between the indoor GOS and the outdoor GOS.

[0284] Further, the encoding device may attach information for the decoding device to distinguish and display dynamic objects from static objects to the encoded data. Accordingly, the decoding device can combine and represent dynamic objects with a red frame or explanatory text, etc. In addition, the decoding device may also represent only with a red frame or explanatory text instead of dynamic objects. And the decoding device can represent more detailed object categories. For example, a car may use a red frame and a person may use a yellow frame.

[0285] Further, the encoding device or the decoding device may determine whether to encode or decode dynamic objects and static objects as different SPCs or GOSs according to the appearance frequency of the dynamic objects, or the ratio of static objects to dynamic objects, etc. For example, when the appearance frequency or ratio of the dynamic objects exceeds the threshold, the SPC or GOS in which the dynamic objects and the static objects are mixed is allowed, and when the appearance frequency or ratio of the dynamic objects does not exceed the threshold, the SPC or GOS in which the dynamic objects and the static objects are mixed is not allowed.

[0286] When a dynamic object is detected not from point cloud data but from two-dimensional image information of a camera, the encoding device can separately obtain information (such as a frame or text) for identifying the detection result and the object position, and encode these as part of three-dimensional encoded data. In this case, the decoding device overlays and displays auxiliary information (frame or text) representing the dynamic object on the decoding result of the static object.

[0287] Moreover, the encoding device can change the density of VXL or VLM according to the complexity of the shape of the static object, etc. For example, the more complex the shape of the static object, the denser the encoding device sets VXL or VLM. Also, the encoding device can determine the quantization step size, etc. when quantifying spatial position or color information according to the density of VXL or VLM. For example, the denser VXL or VLM, the smaller the encoding device sets the quantization step size.

[0288] As described above, the encoding device or decoding device according to this embodiment performs spatial encoding or decoding in a spatial unit having coordinate information.

[0289] Furthermore, the encoding device and the decoding device perform encoding or decoding in a volume unit within the space. The volume includes voxels, which are the smallest units corresponding to position information.

[0290] Moreover, the encoding device and the decoding device establish correspondences between arbitrary elements by using a table in which each element of spatial information including coordinates, objects, and time, etc. is associated with a GOP, or a table corresponding between each element, and perform encoding or decoding. And the decoding device uses the value of the selected element to determine the coordinates, and determines the volume, voxel, or space according to the coordinates, and decodes the space including the volume or voxel, or the determined space.

[0291] Moreover, the encoding device determines the volume, voxel, or space that can be selected by an element through feature point extraction or object recognition, and encodes it as a volume, voxel, or space that can be randomly accessed.

[0292] The space is divided into three types, namely: I-SPC that can be encoded or decoded by the space alone, P-SPC that encodes or decodes with reference to any one processed space, and B-SPC that encodes or decodes with reference to any two processed spaces.

[0293] One or more volumes correspond to static objects or dynamic objects. The space containing the static object and the space containing the dynamic object are encoded or decoded as different GOSs. That is, the SPC containing the static object and the SPC containing the dynamic object are assigned to different GOSs.

[0294] Dynamic objects are encoded or decoded on a per-object basis, corresponding to more than one space that contains only static objects. That is, multiple dynamic objects are encoded separately, and the encoded data of the multiple dynamic objects corresponds to the SPC that contains only static objects.

[0295] The encoding device and the decoding device increase the priority of the I-SPC in the GOS to perform encoding or decoding. For example, the encoding device performs encoding in a manner that reduces the degradation of the I-SPC (after decoding, the original three-dimensional data can be reproduced more faithfully). Also, the decoding device decodes only the I-SPC, for example.

[0296] The encoding device can change the frequency of using the I-SPC according to the density or value (quantity) of the objects in the world space to perform encoding. That is, the encoding device changes the frequency of selecting the I-SPC according to the quantity or density of the objects included in the three-dimensional data. For example, the encoding device increases the usage frequency of the I-space as the density of the objects in the world space is greater.

[0297] Also, the encoding device sets random access points in units of GOS, and stores the information indicating the spatial region corresponding to the GOS in the header information.

[0298] The encoding device uses a default value as the spatial size of the GOS, for example. In addition, the encoding device can also change the size of the GOS according to the value (quantity) or density of the objects or dynamic objects. For example, the encoding device sets the spatial size of the GOS to be smaller as the objects or dynamic objects are denser or more numerous.

[0299] Also, the space or volume includes a group of feature points derived using information obtained by sensors such as depth sensors, gyroscopes, or cameras. The coordinates of the feature points are set as the center positions of the voxels. And through the subdivision of the voxels, high-precision position information can be achieved.

[0300] The group of feature points is derived using multiple pictures. The multiple pictures have at least the following two types of time information, namely: actual time information, and the same time information in the multiple pictures corresponding to the space (for example, the encoding time for rate control, etc.).

[0301] Also, encoding or decoding is performed in units of GOS that includes more than one space.

[0302] The encoding device and the decoding device predict the P space or B space in the GOS to be processed with reference to the space in the processed GOS.

[0303] Alternatively, the encoding device and the decoding device do not refer to different GOSs, and use the processed space within the GOS of the object to be processed to predict the P space or B space within the GOS of the object to be processed.

[0304] Furthermore, the encoding device and the decoding device send or receive an encoded stream in units of a world space including one or more GOSs.

[0305] Moreover, the GOS has a layer structure at least in one direction within the world space, and the encoding device and the decoding device perform encoding or decoding starting from the lower layer. For example, a GOS that can be randomly accessed belongs to the lowest layer. A GOS belonging to an upper layer only refers to GOSs belonging to layers below the same layer. That is, the GOS is spatially divided in a predefined direction and includes multiple layers each having one or more SPCs. The encoding device and the decoding device perform encoding or decoding for each SPC by referring to SPCs included in the same layer as or a lower layer than the SPC.

[0306] In addition, the encoding device and the decoding device continuously perform encoding or decoding on GOSs within a world space unit including multiple GOSs. The encoding device and the decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. That is, the encoded data includes information indicating the encoding order of multiple GOSs.

[0307] Also, the encoding device and the decoding device perform encoding or decoding on two or more different spaces or GOSs in parallel.

[0308] Moreover, the encoding device and the decoding device encode or decode the spatial information (coordinates, size, etc.) of the space or GOS.

[0309] Furthermore, the encoding device and the decoding device encode or decode the space or GOS included in a specific space determined according to external information such as GPS, path information, or magnification related to its own position or / and area size.

[0310] The encoding device or the decoding device performs encoding or decoding by setting the priority of a space far from its own position to be lower than that of a space close to its own position.

[0311] The encoding device sets one direction in the world space according to magnification or use, and encodes the GOS having a layer structure in that direction. And the decoding device preferentially performs decoding starting from the lower layer for the GOS having a layer structure in one direction in the world space set according to magnification or use.

[0312] The encoding device changes the extraction of feature points, the accuracy of object recognition, or the size of the spatial region, etc. included in the indoor and outdoor spaces. However, the encoding device and the decoding device encode or decode by making the indoor GOS and the outdoor GOS with close coordinates adjacent in the world space, and also encode or decode by corresponding these identifiers.

[0313] (Embodiment 2)

[0314] When using the encoded data of the point cloud data for an actual device or service, in order to suppress the network bandwidth, it is desired to transmit and receive the required information according to the usage. However, such a function does not exist in the encoding structure of the three-dimensional data so far, and thus there is no corresponding encoding method either.

[0315] What will be described in this embodiment is a three-dimensional data encoding method and a three-dimensional data encoding device for providing a function of transmitting and receiving the required information according to the usage in the encoded data of the three-dimensional point cloud data, and also a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data.

[0316] A voxel (VXL) having a feature amount equal to or more than a certain value is defined as a feature voxel (FVXL), and a world space (WLD) composed of FVXLs is defined as a sparse world space (SWLD). Figure 11 A configuration example of the sparse world space and the world space is shown. In the SWLD, there are included: FGOS, which is a GOS composed of FVXLs; FSPC, which is an SPC composed of FVXLs; and FVLM, which is a VLM composed of FVXLs. The data structure and prediction structure of FGOS, FSPC, and FVLM can be the same as those of GOS, SPC, and VLM.

[0317] The feature amount refers to a feature amount that represents the three-dimensional position information of the VXL or the visible light information at the VXL position, and in particular, a feature amount that can detect more features such as the corners and edges of a three-dimensional object. Specifically, although this feature amount is the three-dimensional feature amount or the visible light feature amount described below, as long as it is a feature amount that represents the position, brightness, or color information, etc. of the VXL, it can be any feature amount.

[0318] As the three-dimensional feature amount, the SHOT feature amount (Signature of Histograms of OrienTations), the PFH feature amount (Point Feature Histograms), or the PPF feature amount (Point Pair Feature) is adopted.

[0319] The SHOT feature quantity is obtained by dividing the periphery of the VXL, calculating the inner product of the reference point and the normal vector of the divided area, and performing histogramming. This SHOT feature quantity has the characteristics of high dimensionality and high feature expressiveness.

[0320] The PFH feature quantity is obtained by selecting multiple two-point groups near the VXL, calculating the normal vector, etc. based on these two points, and performing histogramming. Since this PFH feature quantity is a histogram feature, it is robust against a small amount of interference and has the characteristic of high feature expressiveness.

[0321] The PPF feature quantity is a feature quantity calculated using the normal vector, etc. according to two VXLs. In this PPF feature quantity, since all VXLs are used, it is robust against occlusion.

[0322] Moreover, as a feature quantity of visible light, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients), etc. that adopt information such as the luminance gradient information of an image can be used.

[0323] The SWLD is generated by calculating the above-mentioned feature quantity from each VXL of the WLD and extracting the FVXL. Here, the SWLD can be updated each time the WLD is updated, or it can be updated periodically after a certain period of time regardless of the update timing of the WLD.

[0324] The SWLD can be generated for each feature quantity. For example, as shown by the SWLD1 based on the SHOT feature quantity and the SWLD2 based on the SIFT feature quantity, the SWLD can be generated separately for each feature quantity, and the SWLD can be distinguished and used according to the application. Also, the feature quantity of each calculated FVXL can be held as feature quantity information in each FVXL.

[0325] Next, the method of using the sparse world space (SWLD) will be described. Since the SWLD only contains feature voxels (FVXL), generally, the data size is smaller compared to the WLD that includes all VXLs.

[0326] In an application that uses feature quantities to achieve a certain purpose, by using the information of SWLD instead of WLD, the read time from the hard disk can be suppressed, and the bandwidth and transmission time during network transmission can be suppressed. For example, as map information, WLD and SWLD are stored in the server in advance, and by switching the transmitted map information to WLD or SWLD according to the requirements from the client, the network bandwidth and transmission time can be suppressed. The following shows specific examples.

[0327] Figure 12 And Figure 13 Examples of the use of SWLD and WLD are shown. As Figure 12 shown, when the client 1 as a vehicle-mounted device needs map information for its own position judgment, the client 1 sends a request for obtaining map data for its own position estimation to the server (S301). The server sends the SWLD to the client 1 according to this acquisition request (S302). The client 1 uses the received SWLD to judge its own position (S303). At this time, the client 1 obtains VXL information around the client 1 by various methods such as a distance sensor such as a rangefinder, a stereo camera, or a combination of multiple monocular cameras, and estimates its own position information based on the obtained VXL information and SWLD. Here, the own position information includes the three-dimensional position information and the orientation of the client 1, etc.

[0328] As Figure 13 shown, when the client 2 as a vehicle-mounted device needs map information for map drawing such as a three-dimensional map, the client 2 sends a request for obtaining map data for map drawing to the server (S311). The server sends the WLD to the client 2 according to this acquisition request (S312). The client 2 uses the received WLD to perform map drawing (S313). At this time, the client 2, for example, uses an image taken by its own visible light camera, etc., and the WLD obtained from the server to create a concept image, and depicts the created image on a screen such as a car navigation.

[0329] As shown above, the server sends the SWLD to the client in applications that mainly require the feature quantities of each VXL for its own position estimation, and sends the WLD to the client when detailed VXL information is required, such as map drawing. Accordingly, the map data can be efficiently sent and received.

[0330] In addition, the client can judge which of SWLD and WLD it needs and request the server to send SWLD or WLD. And the server can judge which of SWLD or WLD should be sent according to the condition of the client or the network.

[0331] Next, a method for switching between the reception and transmission of the Sparse World Space (SWLD) and the World Space (WLD) will be described.

[0332] The reception of the WLD or SWLD can be switched according to the network bandwidth. Figure 14 A working example in this case is shown. For example, when a low-speed network with a network bandwidth such as in an LTE (Long Term Evolution) environment is used, when the client accesses the server via the low-speed network (S321), the client obtains the SWLD as map information from the server (S322). Additionally, when a high-speed network with a surplus network bandwidth such as in a WiFi environment is used, the client accesses the server via the high-speed network (S323) and obtains the WLD from the server (S324). Accordingly, the client can obtain appropriate map information according to the network bandwidth of the client.

[0333] Specifically, the client receives the SWLD via LTE outdoors, and when entering indoors such as in a facility, obtains the WLD via WiFi. Accordingly, the client can obtain more detailed map information of the interior.

[0334] In this way, the client can request the WLD or SWLD from the server according to the frequency band of the network it uses. Alternatively, the client can send information indicating the frequency band of the network it uses to the server, and the server sends appropriate data (WLD or SWLD) to the client according to this information. Or, the server can determine the network bandwidth of the client and send appropriate data (WLD or SWLD) to the client.

[0335] Moreover, the reception of the WLD or SWLD can be switched according to the moving speed. Figure 15 A working example in this case is shown. For example, when the client is moving at high speed (S331), the client receives the SWLD from the server (S332). Additionally, when the client is moving at low speed (S333), the client receives the WLD from the server (S334). Accordingly, the client can both suppress the network bandwidth and obtain map information according to the speed. Specifically, when the client is driving on a highway, by receiving the SWLD with a small amount of data, the map information can be updated at an appropriate speed approximately. Additionally, when the client is driving on an ordinary road, by receiving the WLD, more detailed map information can be obtained.

[0336] In this way, the client can request the WLD or SWLD from the server according to its own moving speed. Alternatively, the client can send the information indicating its own moving speed to the server, and the server sends appropriate data (WLD or SWLD) to the client according to this information. Alternatively, the server can determine the moving speed of the client and send appropriate data (WLD or SWLD) to the client.

[0337] Also, it can be that the client first obtains the SWLD from the server and then obtains the WLD of the important areas therein. For example, when the client obtains map data, it first obtains the general map information in the form of SWLD, filters out the areas where there are many features such as buildings, signs, or people, and then obtains the WLD of the filtered areas. Accordingly, the client can both suppress the amount of received data from the server and obtain the detailed information of the required areas.

[0338] Also, it can be that the server separately creates the SWLD for each object according to the WLD, and the client receives them separately according to the usage. Accordingly, the network bandwidth can be suppressed. For example, the server pre-identifies people or vehicles from the WLD and creates the SWLD for people and the SWLD for vehicles. When the client wants to obtain information about the people around, it receives the SWLD for people, and when it wants to obtain information about vehicles, it receives the SWLD for vehicles. And the types of such SWLD can be distinguished according to the information (flags or types, etc.) attached to the head or the like.

[0339] Next, the configuration and the working process of the three-dimensional data encoding device (such as a server) according to the present embodiment will be described. Figure 16 It is a block diagram of the three-dimensional data encoding device 400 according to the present embodiment. Figure 17 It is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device 400.

[0340] Figure 16 The three-dimensional data encoding device 400 shown generates encoded three-dimensional data 413 and 414 as an encoded stream by encoding the input three-dimensional data 411. Here, the encoded three-dimensional data 413 is the encoded three-dimensional data corresponding to the WLD, and the encoded three-dimensional data 414 is the encoded three-dimensional data corresponding to the SWLD. The three-dimensional data encoding device 400 includes: an acquisition unit 401, an encoding area determination unit 402, an SWLD extraction unit 403, a WLD encoding unit 404, and an SWLD encoding unit 405.

[0341] As Figure 17 shown, first, the acquisition unit 401 acquires the input three-dimensional data 411 as point cloud data in a three-dimensional space (S401).

[0342] Next, the coding area determination unit 402 determines the spatial area of the coding object based on the spatial area where the point cloud data exists (S402).

[0343] Next, the SWLD extraction unit 403 defines the spatial area of the coding object as the WLD, and calculates the feature amount based on each VXL included in the WLD. Further, the SWLD extraction unit 403 extracts the VXL whose feature amount is equal to or greater than a preset threshold value, defines the extracted VXL as the FVXL, and generates the extracted three-dimensional data 412 by adding the FVXL to the SWLD (S403). That is, the extracted three-dimensional data 412 whose feature amount is equal to or greater than the threshold value is extracted from the input three-dimensional data 411.

[0344] Next, the WLD coding unit 404 generates the coded three-dimensional data 413 corresponding to the WLD by coding the input three-dimensional data 411 corresponding to the WLD (S404). At this time, the WLD coding unit 404 attaches information for distinguishing that the coded three-dimensional data 413 is a stream including the WLD to the head of the coded three-dimensional data 413.

[0345] Further, the SWLD coding unit 405 generates the coded three-dimensional data 414 corresponding to the SWLD by coding the extracted three-dimensional data 412 corresponding to the SWLD (S405). At this time, the SWLD coding unit 405 attaches information for distinguishing that the coded three-dimensional data 414 is a stream including the SWLD to the head of the coded three-dimensional data 414.

[0346] Further, the processing order of the process of generating the coded three-dimensional data 413 and the process of generating the coded three-dimensional data 414 may be opposite to the above. Further, a part or all of the above processes may be executed in parallel.

[0347] The information given to the heads of the coded three-dimensional data 413 and 414 is defined as a parameter such as "world_type", for example. When world_type = 0, it indicates that the stream includes the WLD, and when world_type = 1, it indicates that the stream includes the SWLD. When defining more other categories, the assigned value can be increased as in world_type = 2. Further, a specific flag may be included in one of the coded three-dimensional data 413 and 414. For example, the coded three-dimensional data 414 may be given a flag indicating that the stream includes the SWLD. In this case, the decoding device can determine whether the stream includes the WLD or the SWLD based on the presence or absence of the flag.

[0348] Further, the coding method used by the WLD coding unit 404 when coding the WLD may be different from the coding method used by the SWLD coding unit 405 when coding the SWLD.

[0349] For example, since SWLD data is selected, its correlation with surrounding data may be lower compared to WLD. Therefore, in the encoding method for SWLD, among intra prediction and inter prediction, inter prediction is prioritized compared to the encoding method for WLD.

[0350] Also, the representation methods of three-dimensional positions may be different between the encoding method for SWLD and the encoding method for WLD. For example, it may be that in FWLD, the three-dimensional position of FVXL is represented by three-dimensional coordinates, and in WLD, the three-dimensional position is represented by an octree described later, and vice versa.

[0351] Moreover, the SWLD encoding unit 405 encodes in such a way that the data size of the encoded three-dimensional data 414 of SWLD is smaller than the data size of the encoded three-dimensional data 413 of WLD. As described above, for example, the correlation between data in SWLD may be lower than that in WLD. Accordingly, the encoding efficiency decreases, and the data size of the encoded three-dimensional data 414 may be larger than the data size of the encoded three-dimensional data 413 of WLD. Therefore, when the data size of the obtained encoded three-dimensional data 414 of SWLD is larger than the data size of the encoded three-dimensional data 413 of WLD, the SWLD encoding unit 405 performs re-encoding to regenerate the encoded three-dimensional data 414 with a reduced data size.

[0352] For example, the SWLD extraction unit 403 regenerates the extracted three-dimensional data 412 with a reduced number of extracted feature points, and the SWLD encoding unit 405 encodes the extracted three-dimensional data 412. Or, the quantization level in the SWLD encoding unit 405 can be made coarser. For example, in the octree structure described later, by rounding the data in the bottom layer, the quantization level can be made coarser.

[0353] Also, when the SWLD encoding unit 405 cannot make the data size of the encoded three-dimensional data 414 of SWLD smaller than the data size of the encoded three-dimensional data 413 of WLD, it may not generate the encoded three-dimensional data 414 of SWLD. Or, the encoded three-dimensional data 413 of WLD can be copied to the encoded three-dimensional data 414 of SWLD. That is, as the encoded three-dimensional data 414 of SWLD, the encoded three-dimensional data 413 of WLD can be directly used.

[0354] Next, the configuration and operation flow of the three-dimensional data decoding device (such as a client) according to this embodiment will be described. Figure 18 It is a block diagram of the three-dimensional data decoding device 500 according to this embodiment. Figure 19 It is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding device 500.

[0355] Figure 18 The three-dimensional data decoding device 500 shown generates decoded three-dimensional data 512 or 513 by decoding the encoded three-dimensional data 511. Here, the encoded three-dimensional data 511 is, for example, the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400.

[0356] The three-dimensional data decoding device 500 includes: an acquisition unit 501, a header analysis unit 502, a WLD decoding unit 503, and an SWLD decoding unit 504.

[0357] As Figure 19 shown, first, the acquisition unit 501 acquires the encoded three-dimensional data 511 (S501). Next, the header analysis unit 502 analyzes the header of the encoded three-dimensional data 511 to determine whether the encoded three-dimensional data 511 is a stream containing WLD or a stream containing SWLD (S502). For example, the determination is made with reference to the parameter of world_type described above.

[0358] In the case where the encoded three-dimensional data 511 is a stream containing WLD (Yes in S503), the WLD decoding unit 503 decodes the encoded three-dimensional data 511 to generate the decoded three-dimensional data 512 of WLD (S504). In addition, in the case where the encoded three-dimensional data 511 is a stream containing SWLD (No in S503), the SWLD decoding unit 504 decodes the encoded three-dimensional data 511 to generate the decoded three-dimensional data 513 of SWLD (S505).

[0359] And, similar to the encoding device, the decoding method used by the WLD decoding unit 503 when decoding WLD and the decoding method used by the SWLD decoding unit 504 when decoding SWLD can be different. For example, in the decoding method for SWLD, inter-frame prediction in intra-frame prediction and inter-frame prediction can be given priority compared to the decoding method for WLD.

[0360] And, in the decoding method for SWLD and the decoding method for WLD, the expression methods of three-dimensional positions can be different. For example, in SWLD, the three-dimensional position of FVXL can be expressed by three-dimensional coordinates, and in WLD, the three-dimensional position can be expressed by an octree described later, and vice versa.

[0361] Next, the octree representation as an expression method of three-dimensional positions will be described. The VXL data included in the three-dimensional data is encoded after being converted into an octree structure. Figure 20 An example of VXL of WLD is shown. Figure 21 Shows Figure 20 The octree structure of the WLD shown. InFigure 20 In the example shown, there are three VXLs (hereinafter, valid VXLs) that include point groups, namely VXL1 to VXL3. As Figure 21 shown, the octree structure is composed of nodes and leaves. Each node has a maximum of 8 nodes or leaves. Each leaf has VXL information. Here, Figure 21 among the leaves shown, leaves 1, 2, and 3 respectively represent Figure 20 the VXL1, VXL2, and VXL3 shown.

[0362] Specifically, each node and leaf correspond to a three-dimensional position. Node 1 corresponds to Figure 20 all the blocks shown. The block corresponding to Node 1 is divided into 8 blocks. Among the 8 blocks, the blocks including valid VXLs are set as nodes, and the other blocks are set as leaves. The block corresponding to the node is further divided into 8 nodes or leaves, and the number of times this process is repeated is the same as the number of levels in the tree structure. And all the blocks in the bottom layer are set as leaves.

[0363] And, Figure 22 an example of the SWLD generated from the Figure 20 shown WLD is shown. Figure 20 The results of feature quantity extraction of the VXL1 and VXL2 shown are judged as FVXL1 and FVXL2 and are added to the SWLD. In addition, since VXL3 is not judged as FVXL, it is not included in the SWLD. Figure 23 An example of the octree structure of the Figure 22 shown SWLD is shown. In the Figure 23 octree structure shown, Figure 21 leaf 3, which corresponds to VXL3 and is shown, is deleted. Accordingly, Figure 21 node 3 shown has no valid VXL and is changed to a leaf. In general, the number of leaves in the SWLD is smaller than the number of leaves in the WLD, and the encoded three-dimensional data of the SWLD is also smaller than the encoded three-dimensional data of the WLD.

[0364] The following describes a modification example of this embodiment.

[0365] For example, it can also be the case where, when a client such as a vehicle-mounted device estimates its own position, it receives the SWLD from the server, uses the SWLD to estimate its own position, and performs obstacle detection. In this case, various methods such as a distance sensor such as a rangefinder, a stereo camera, or a combination of multiple monocular cameras are used to perform obstacle detection based on the three-dimensional information of the surrounding area obtained by itself.

[0366] Also, generally speaking, it is difficult to include VXL data of flat areas in the SWLD. For this reason, the server maintains a sub-sampled world space (subWLD) obtained by downsampling the WLD for detecting stationary obstacles, and can send the SWLD and the subWLD to the client. Accordingly, both the network bandwidth can be suppressed, and the client side can perform self-position estimation and obstacle detection.

[0367] Also, when quickly depicting three-dimensional map data on the client side, it is convenient if the map information has a grid structure. Thus, the server can generate a grid based on the WLD and maintain it in advance as a grid world space (MWLD). For example, when the client needs to perform rough three-dimensional depiction, it receives the MWLD, and when it needs to perform detailed three-dimensional depiction, it receives the WLD. Accordingly, the network bandwidth can be suppressed.

[0368] Also, although the server sets the VXLs with feature amounts above the threshold as FVXLs from each VXL, the FVXLs can also be calculated by different methods. For example, if the server determines that VXLs, VLMs, SPCs, or GOSs that make up signals or intersections, etc. are required for self-position estimation, driving assistance, or autonomous driving, etc., they can be included in the SWLD as FVXLs, FVLMs, FSPCs, and FGOSs. And the above determination can be made manually. In addition, the FVXLs obtained by the above method can be added to the FVXLs, etc. set based on the feature amount. That is, the SWLD extraction unit 403 can further extract data corresponding to an object having a predetermined attribute from the input three-dimensional data 411 as the extracted three-dimensional data 412.

[0369] Also, different labels can be assigned to the situations that are required for these uses, different from the feature amounts. The server can separately maintain the FVXLs required for self-position estimation, driving assistance, or autonomous driving, such as signals or intersections, as the upper layer of the SWLD (for example, the lane world space).

[0370] Also, the server can attach attributes to the VXLs in the WLD in units of random access or specified units. The attributes include, for example, information indicating whether it is required or not required for self-position estimation, or information indicating whether it is important as traffic information such as a signal or an intersection. And the attributes can also include the correspondence relationship with Features (intersections or roads, etc.) in lane information (GDF: Geographic Data Files, etc.).

[0371] Also, as a method for updating the WLD or SWLD, the following method can be adopted.

[0372] Update information such as changes in people, construction, or street trees (facing the track) is loaded into the server as a point cloud or metadata. Based on this load, the server updates the WLD, and after that, uses the updated WLD to update the SWLD.

[0373] Moreover, when the client detects a mismatch between the three-dimensional information generated by itself during self-position estimation and the three-dimensional information received from the server, it can send the three-dimensional information generated by itself to the server together with an update notification. In this case, the server uses the WLD to update the SWLD. If the SWLD is not updated, the server determines that the WLD itself is old.

[0374] Also, as header information of the encoded stream, information for distinguishing between WLD and SWLD is attached. For example, in the case where there are multiple world spaces such as a grid world space or a lane world space, information for distinguishing them can be attached to the header information. And when there are multiple SWLDs with different feature amounts, information for distinguishing them separately can also be attached to the header information.

[0375] Moreover, although the SWLD is composed of FVXLs, it can also include VXLs that are not determined to be FVXLs. For example, the SWLD can include adjacent VXLs used when calculating the feature amounts of FVXLs. Accordingly, even when no feature amount information is attached to each FVXL of the SWLD, the client can calculate the feature amounts of FVXLs when receiving the SWLD. Also, at this time, the SWLD can include information for distinguishing whether each VXL is an FVXL or a VXL.

[0376] As described above, the three-dimensional data encoding device 400 extracts the extracted three-dimensional data 412 (second three-dimensional data) whose feature amount is above the threshold from the input three-dimensional data 411 (first three-dimensional data), and generates the encoded three-dimensional data 414 (first encoded three-dimensional data) by encoding the extracted three-dimensional data 412.

[0377] Accordingly, the three-dimensional data encoding device 400 generates the encoded three-dimensional data 414 obtained by encoding the data whose feature amount is above the threshold. In this way, compared with the case of directly encoding the input three-dimensional data 411, the data amount can be reduced. Therefore, the three-dimensional data encoding device 400 can reduce the data amount during transmission.

[0378] Moreover, the three-dimensional data encoding device 400 further generates the encoded three-dimensional data 413 (second encoded three-dimensional data) by encoding the input three-dimensional data 411.

[0379] Accordingly, the three-dimensional data encoding device 400 can selectively transmit the encoded three-dimensional data 413 and the encoded three-dimensional data 414 according to the usage purpose or the like.

[0380] Moreover, the extracted three-dimensional data 412 is encoded by the first encoding method, and the input three-dimensional data 411 is encoded by the second encoding method different from the first encoding method.

[0381] Accordingly, the three-dimensional data encoding device 400 can adopt appropriate encoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0382] Moreover, in the first encoding method, among intra prediction and inter prediction, inter prediction is prioritized compared with the second encoding method.

[0383] Accordingly, the three-dimensional data encoding device 400 can increase the priority of inter prediction for the extracted three-dimensional data 412 where the correlation between adjacent data is likely to become low.

[0384] Moreover, in the first encoding method and the second encoding method, the expression methods of three-dimensional positions are different. For example, in the second encoding method, the three-dimensional position is expressed by an octree, and in the first encoding method, the three-dimensional position is expressed by three-dimensional coordinates.

[0385] Accordingly, the three-dimensional data encoding device 400 can adopt a more appropriate expression method of three-dimensional positions for three-dimensional data with different numbers of data (the number of VXL or FVXL).

[0386] Moreover, at least one of the encoded three-dimensional data 413 and 414 includes an identifier indicating whether the encoded three-dimensional data is the encoded three-dimensional data obtained by encoding the input three-dimensional data 411 or the encoded three-dimensional data obtained by encoding a part of the input three-dimensional data 411. That is, this identifier indicates whether the encoded three-dimensional data is the encoded three-dimensional data 413 of WLD or the encoded three-dimensional data 414 of SWLD.

[0387] Accordingly, the decoding device can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

[0388] Moreover, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 in such a way that the data amount of the encoded three-dimensional data 414 is less than the data amount of the encoded three-dimensional data 413.

[0389] Accordingly, the three-dimensional data encoding device 400 can make the data amount of the encoded three-dimensional data 414 less than the data amount of the encoded three-dimensional data 413.

[0390] Furthermore, the three-dimensional data encoding device 400 further extracts data corresponding to an object having a predetermined attribute from the input three-dimensional data 411 as the extracted three-dimensional data 412. For example, an object having a predetermined attribute refers to an object required in self-position estimation, driving assistance, or autonomous driving, etc., such as a signal or an intersection.

[0391] Accordingly, the three-dimensional data encoding device 400 can generate the encoded three-dimensional data 414 including the data required by the decoding device.

[0392] Furthermore, the three-dimensional data encoding device 400 (server) further sends either the encoded three-dimensional data 413 or 414 to the client according to the state of the client.

[0393] Accordingly, the three-dimensional data encoding device 400 can send appropriate data according to the state of the client.

[0394] Furthermore, the state of the client includes the communication status of the client (e.g., network bandwidth) or the moving speed of the client.

[0395] Furthermore, the three-dimensional data encoding device 400 further sends either the encoded three-dimensional data 413 or 414 to the client according to the request of the client.

[0396] Accordingly, the three-dimensional data encoding device 400 can send appropriate data according to the request of the client.

[0397] Furthermore, the three-dimensional data decoding device 500 according to the present embodiment decodes the encoded three-dimensional data 413 or 414 generated by the above three-dimensional data encoding device 400.

[0398] That is, the three-dimensional data decoding device 500 decodes the encoded three-dimensional data 414 obtained by encoding the extracted three-dimensional data 412 whose feature amount extracted from the input three-dimensional data 411 is above a threshold value by the first decoding method. And the three-dimensional data decoding device 500 decodes the encoded three-dimensional data 413 obtained by encoding the input three-dimensional data 411 by using a second decoding method different from the first decoding method.

[0399] Accordingly, the three-dimensional data decoding device 500 can selectively receive, for example, according to the usage purpose, etc., the encoded three-dimensional data 414 and the encoded three-dimensional data 413 obtained by encoding the data whose feature amount is above the threshold value. Accordingly, the three-dimensional data decoding device 500 can reduce the amount of data during transmission. Moreover, the three-dimensional data decoding device 500 can adopt appropriate decoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0400] Also, in the first decoding method, among intra prediction and inter prediction, inter prediction is prioritized compared to the second decoding method.

[0401] Accordingly, the three-dimensional data decoding device 500 can increase the priority of inter prediction for the extracted three-dimensional data where the correlation between adjacent data is likely to be low.

[0402] Also, the representation methods of three-dimensional positions are different between the first decoding method and the second decoding method. For example, in the second decoding method, the three-dimensional position is represented by an octree, and in the first decoding method, the three-dimensional position is represented by three-dimensional coordinates.

[0403] Accordingly, the three-dimensional data decoding device 500 can adopt a more appropriate representation method of three-dimensional positions for three-dimensional data with different numbers of data (the number of VXLs or FVXLs).

[0404] Also, at least one of the encoded three-dimensional data 413 and 414 includes an identifier that indicates whether the encoded three-dimensional data is the encoded three-dimensional data obtained by encoding the input three-dimensional data 411 or the encoded three-dimensional data obtained by encoding a part of the input three-dimensional data 411. The three-dimensional data decoding device 500 refers to this identifier to identify the encoded three-dimensional data 413 and 414.

[0405] Accordingly, the three-dimensional data decoding device 500 can easily determine whether the obtained encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

[0406] Also, the three-dimensional data decoding device 500 further notifies the server of the state of the client (the three-dimensional data decoding device 500). The three-dimensional data decoding device 500 receives one of the encoded three-dimensional data 413 and 414 sent from the server according to the state of the client.

[0407] Accordingly, the three-dimensional data decoding device 500 can receive appropriate data according to the state of the client.

[0408] Also, the state of the client includes the communication status of the client (e.g., network bandwidth) or the moving speed of the client.

[0409] Also, the three-dimensional data decoding device 500 further requests one of the encoded three-dimensional data 413 and 414 from the server and receives one of the encoded three-dimensional data 413 and 414 sent from the server according to the request.

[0410] Accordingly, the three-dimensional data decoding device 500 can receive appropriate data corresponding to the use.

[0411] (Embodiment 3)

[0412] In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described.

[0413] Figure 24 It is a schematic diagram showing the state of transmission and reception of three-dimensional data 607 between the host vehicle 600 and surrounding vehicles 601.

[0414] When acquiring three-dimensional data by sensors (distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras, etc.) mounted on the host vehicle 600, due to the influence of obstacles such as surrounding vehicles 601, although within the sensor detection range 602 of the host vehicle 600, there will be areas where three-dimensional data cannot be created (hereinafter referred to as occlusion areas 604). Also, although the larger the space for obtaining three-dimensional data, the higher the accuracy of autonomous operation, the sensor detection range of only the host vehicle 600 is limited.

[0415] The sensor detection range 602 of the host vehicle 600 includes an area 603 where three-dimensional data can be obtained and an occlusion area 604. The range in which the host vehicle 600 wants to obtain three-dimensional data includes the sensor detection range 602 of the host vehicle 600 and areas other than this. Also, the sensor detection range 605 of the surrounding vehicle 601 includes the occlusion area 604 and an area 606 not included in the sensor detection range 602 of the host vehicle 600.

[0416] The surrounding vehicle 601 transmits the information detected by the surrounding vehicle 601 to the host vehicle 600. The host vehicle 600 can obtain three-dimensional data 607 of the occlusion area 604 and the area 606 outside the sensor detection range 602 of the host vehicle 600 by obtaining the information detected by surrounding vehicles 601 such as the vehicle traveling ahead. The host vehicle 600 uses the information obtained by the surrounding vehicle 601 to supplement the three-dimensional data of the occlusion area 604 and the area 606 outside the sensor detection range.

[0417] The uses of three-dimensional data in the autonomous operation of vehicles or robots are for self-position estimation, detection of the surrounding situation, or both. For example, in self-position estimation, three-dimensional data generated by the host vehicle 600 is used based on the sensor information of the host vehicle 600. In the detection of the surrounding situation, in addition to the three-dimensional data generated by the host vehicle 600, three-dimensional data obtained by the surrounding vehicle 601 is also used.

[0418] Surrounding vehicles 601 that transfer three-dimensional data 607 to the host vehicle 600 can be determined according to the state of the host vehicle 600. For example, the surrounding vehicle 601 is a vehicle traveling ahead when the host vehicle 600 goes straight, an oncoming vehicle when the host vehicle 600 turns right, or a vehicle behind when the host vehicle 600 reverses. Also, it can be that the driver of the host vehicle 600 directly designates a surrounding vehicle 601 that transfers three-dimensional data 607 to the host vehicle 600.

[0419] Also, the host vehicle 600 can search for surrounding vehicles 601 that hold three-dimensional data of areas that the host vehicle 600 cannot obtain within the space where it wants to obtain three-dimensional data 607. Areas that the host vehicle 600 cannot obtain refer to, for example, the occlusion area 604 or the area 606 outside the sensor detection range 602.

[0420] Also, the host vehicle 600 can determine the occlusion area 604 based on the sensor information of the host vehicle 600. For example, the host vehicle 600 determines an area within the sensor detection range 602 of the host vehicle 600 that cannot generate three-dimensional data as the occlusion area 604.

[0421] The following describes a working example in the case where the vehicle traveling ahead transfers three-dimensional data 607. Figure 25 An example of the three-dimensional data transmitted in this case is shown.

[0422] As Figure 25 shown, the three-dimensional data 607 transmitted from the vehicle traveling ahead is, for example, a sparse world space (SWLD) of point cloud data. That is, the vehicle traveling ahead creates three-dimensional data (point cloud data) of the WLD based on the information detected by the sensors of the vehicle traveling ahead, and creates three-dimensional data (point cloud data) of the SWLD by extracting data with a feature amount above a threshold from the three-dimensional data of the WLD. Then, the vehicle traveling ahead transfers the created three-dimensional data of the SWLD to the host vehicle 600.

[0423] The host vehicle 600 receives the SWLD and merges the received SWLD into the point cloud data created by the host vehicle 600.

[0424] The transmitted SWLD has information on absolute coordinates (the position of the SWLD in the coordinate system of the three-dimensional map). The host vehicle 600 superimposes the point cloud data generated by the host vehicle 600 according to the absolute coordinates, thereby enabling the merge process.

[0425] The SWLD transmitted from the surrounding vehicle 601 may be the SWLD of the area 606 outside the sensor detection range 602 of the host vehicle 600 and within the sensor detection range 605 of the surrounding vehicle 601, or may be the SWLD of the occlusion area 604 relative to the host vehicle 600, or may be the SWLD of both. Also, the transmitted SWLD may be the SWLD of the area used by the surrounding vehicle 601 when detecting the surrounding situation among the above-mentioned SWLDs.

[0426] Also, the surrounding vehicle 601 may change the density of the transmitted point cloud data according to the communicable time based on the speed difference between the host vehicle 600 and the surrounding vehicle 601. For example, when the speed difference is large and the communicable time is short, the surrounding vehicle 601 may reduce the density (data volume) of the point cloud data by extracting three-dimensional points with large feature amounts from the SWLD.

[0427] Also, the detection of the surrounding situation means determining whether there are people, vehicles, road construction equipment, etc., identifying their types, and detecting their positions, moving directions, moving speeds, etc.

[0428] Also, it may be that the host vehicle 600 obtains the braking information of the surrounding vehicle 601 in place of the three-dimensional data 607 generated by the surrounding vehicle 601, or in addition to the three-dimensional data 607, also obtains the braking information of the surrounding vehicle 601. Here, the braking information of the surrounding vehicle 601 is, for example, information indicating that the accelerator or brake of the surrounding vehicle 601 is depressed, or the degree of depression.

[0429] Also, for the point cloud data generated by each vehicle, considering low-latency communication between vehicles, the three-dimensional space is subdivided into random access units. In addition, for the map data downloaded from the server, compared with the case of vehicle-to-vehicle communication, three-dimensional maps, etc. are divided into larger random access units.

[0430] Data of areas that are likely to be occlusion areas, such as the area in front of the vehicle traveling ahead or the area behind the vehicle traveling behind, are divided into small random access units as low-latency-oriented data.

[0431] When traveling at high speed, since the importance of the front increases, each vehicle creates the SWLD of a narrow viewing angle range in small random access units when traveling at high speed.

[0432] When the area where the host vehicle 600 can obtain point cloud data is included in the SWLD created by the vehicle traveling ahead for transmission, the vehicle traveling ahead can reduce the transmission volume by removing the point cloud data in this area.

[0433] Next, the configuration and operation of the three-dimensional data creation device 620, which is the three-dimensional data receiving device according to this embodiment, will be described.

[0434] Figure 26 FIG. 4 is a block diagram of the three-dimensional data creation device 620 according to this embodiment. The three-dimensional data creation device 620 is included in the own vehicle 600 described above, for example, and creates a denser third three-dimensional data 636 by synthesizing the received second three-dimensional data 635 and the first three-dimensional data 632 created by the three-dimensional data creation device 620.

[0435] The three-dimensional data creation device 620 includes: a three-dimensional data creation unit 621, a request range determination unit 622, a search unit 623, a reception unit 624, a decoding unit 625, and a synthesis unit 626. Figure 27 FIG. 5 is a flowchart showing the operation of the three-dimensional data creation device 620.

[0436] First, the three-dimensional data creation unit 621 creates the first three-dimensional data 632 using the sensor information 631 detected by the sensors included in the own vehicle 600 (S621). Next, the request range determination unit 622 determines a request range, which is a three-dimensional space range where the data in the created first three-dimensional data 632 is insufficient (S622).

[0437] Next, the search unit 623 searches for surrounding vehicles 601 that hold three-dimensional data within the request range, and transmits request range information 633 indicating the request range to the surrounding vehicles 601 determined by the search (S623). Next, the reception unit 624 receives the encoded three-dimensional data 634, which is an encoded stream of the request range, from the surrounding vehicles 601 (S624). In addition, the search unit 623 can issue requests to all vehicles existing within the determined range without discrimination, and receive the encoded three-dimensional data 634 from the responding party. Also, the search unit 623 is not limited to vehicles, and can also issue requests to objects such as traffic lights or signs, and receive the encoded three-dimensional data 634 from the object.

[0438] Next, the received encoded three-dimensional data 634 is decoded by the decoding unit 625 to obtain the second three-dimensional data 635 (S625). Next, the first three-dimensional data 632 and the second three-dimensional data 635 are synthesized by the synthesis unit 626 to create a denser third three-dimensional data 636 (S626).

[0439] Next, the configuration and operation of the three-dimensional data transmission device 640 according to this embodiment will be described. Figure 28 FIG. 6 is a block diagram of the three-dimensional data transmission device 640.

[0440] The three-dimensional data transmission device 640 is included, for example, in the surrounding vehicle 601 mentioned above. It processes the fifth three-dimensional data 652 created by the surrounding vehicle 601 into the sixth three-dimensional data 654 requested by the host vehicle 600, generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654, and transmits the encoded three-dimensional data 634 to the host vehicle 600.

[0441] The three-dimensional data transmission device 640 includes: a three-dimensional data creation unit 641, a reception unit 642, an extraction unit 643, an encoding unit 644, and a transmission unit 645. Figure 29 It is a flowchart showing the operation of the three-dimensional data transmission device 640.

[0442] First, the three-dimensional data creation unit 641 creates the fifth three-dimensional data 652 using the sensor information 651 detected by the sensors included in the surrounding vehicle 601 (S641). Next, the reception unit 642 receives the requested range information 633 transmitted from the host vehicle 600 (S642).

[0443] Next, the extraction unit 643 extracts the three-dimensional data within the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652, and processes the fifth three-dimensional data 652 into the sixth three-dimensional data 654 (S643). Next, the encoding unit 644 encodes the sixth three-dimensional data 654 to generate the encoded three-dimensional data 634 as an encoded stream (S644). Then, the transmission unit 645 transmits the encoded three-dimensional data 634 to the host vehicle 600 (S645).

[0444] In addition, here, although an example in which the host vehicle 600 is equipped with the three-dimensional data creation device 620 and the surrounding vehicle 601 is equipped with the three-dimensional data transmission device 640 has been described, each vehicle may also have the functions of the three-dimensional data creation device 620 and the three-dimensional data transmission device 640.

[0445] Hereinafter, the configuration and operation in the case where the three-dimensional data creation device 620 is a surrounding condition detection device that realizes the detection process of the surrounding conditions of the host vehicle 600 will be described. Figure 30 It is a block diagram showing the configuration of the three-dimensional data creation device 620A in this case. Figure 30 The shown three-dimensional data creation device 620A in addition to Figure 26 the configuration of the shown three-dimensional data creation device 620, further includes: a detection area determination unit 627, a surrounding condition detection unit 628, and an autonomous operation control unit 629. And the three-dimensional data creation device 620A is included in the host vehicle 600.

[0446] Figure 31It is a flowchart for the surrounding condition detection process of the host vehicle 600 by the three-dimensional data production device 620A.

[0447] First, the three-dimensional data production unit 621 produces the first three-dimensional data 632 as point cloud data by using the sensor information 631 of the detection range of the host vehicle 600 detected by the sensors equipped on the host vehicle 600 (S661). In addition, the three-dimensional data production device 620A can also use the sensor information 631 to estimate its own position.

[0448] Next, the detection area determination unit 627 determines the detection target range as the spatial area for which the surrounding conditions are to be detected (S662). For example, the detection area determination unit 627 calculates the area required for the surrounding condition detection for safe autonomous operation according to the driving direction and speed of the host vehicle 600 and other autonomous operation (automatic driving) conditions, and determines this area as the detection target range.

[0449] Next, the request range determination unit 622 determines the occlusion area 604 and the spatial area that is outside the detection range of the sensors of the host vehicle 600 but is required for the surrounding condition detection as the request range (S663).

[0450] When there is a request range determined in step S663 (Yes in S664), the search unit 623 searches for surrounding vehicles that hold information related to the request range. For example, the search unit 623 can ask the surrounding vehicles whether they hold information related to the request range, and judge whether the surrounding vehicle holds information related to the request range according to the request range and the position of the surrounding vehicle. Next, the search unit 623 sends a delegation signal 637 for commissioning the transmission of three-dimensional data to the surrounding vehicle 601 identified through the search. And after the search unit 623 receives an allow signal sent from the surrounding vehicle 601 indicating acceptance of the commission of the delegation signal 637, it sends the request range information 633 showing the request range to the surrounding vehicle 601 (S665).

[0451] Next, the receiving unit 624 detects the transmission notification of the transmission data 638 as the information related to the request range and accepts the transmission data 638 (S666).

[0452] In addition, the three-dimensional data production device 620A can also, instead of searching for the recipient of the request, issue requests to all vehicles existing within the determined range without discrimination, and receive the transmission data 638 from the party that responds as holding information related to the request range. And the search unit 623 is not limited to vehicles, and can also issue requests to objects such as traffic lights or signs, and receive the transmission data 638 from this object.

[0453] Moreover, the transmitted data 638 includes at least one of the encoded three-dimensional data 634 obtained by encoding the three-dimensional data of the requested range generated by the surrounding vehicle 601 and the surrounding condition detection result 639 of the requested range. The surrounding condition detection result 639 shows the positions, moving directions, moving speeds, etc. of the people and vehicles detected by the surrounding vehicle 601. Moreover, the transmitted data 638 may also include information such as the position and movement of the surrounding vehicle 601. For example, the transmitted data 638 may also include the braking information of the surrounding vehicle 601.

[0454] When the encoded three-dimensional data 634 is included in the received transmitted data 638 (Yes in S667), the encoded three-dimensional data 634 is decoded by the decoding unit 625 to obtain the second three-dimensional data 635 of the SWLD (S668). That is, the second three-dimensional data 635 is three-dimensional data (SWLD) generated by extracting data with a feature amount equal to or greater than the threshold value from the fourth three-dimensional data (WLD).

[0455] Next, the first three-dimensional data 632 and the second three-dimensional data 635 are synthesized by the synthesis unit 626 to generate the third three-dimensional data 636 (S669).

[0456] Next, the surrounding condition detection unit 628 uses the point cloud data of the spatial region required for surrounding condition detection, that is, the third three-dimensional data 636, to detect the surrounding conditions of the host vehicle 600 (S670). In addition, when the surrounding condition detection result 639 is included in the received transmitted data 638, the surrounding condition detection unit 628 uses not only the third three-dimensional data 636 but also the surrounding condition detection result 639 to detect the surrounding conditions of the host vehicle 600. Moreover, when the braking information of the surrounding vehicle 601 is included in the received transmitted data 638, the surrounding condition detection unit 628 uses not only the third three-dimensional data 636 but also the braking information to detect the surrounding conditions of the host vehicle 600.

[0457] Next, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the host vehicle 600 according to the surrounding condition detection result obtained by the surrounding condition detection unit 628 (S671). In addition, the surrounding condition detection result can be presented to the driver through a UI (user interface) or the like.

[0458] Also, when there is no request range in step S663 (No in S664), that is, when all the spatial region information required for surrounding condition detection is created based on the sensor information 631, the surrounding condition detection unit 628 uses the point cloud data of the spatial region required for surrounding condition detection, i.e., the first three-dimensional data 632, to detect the surrounding condition of the host vehicle 600 (S672). Then, the autonomous driving control unit 629 controls the autonomous driving (autonomous operation) of the host vehicle 600 based on the surrounding condition detection result by the surrounding condition detection unit 628 (S671).

[0459] Also, when the received transmission data 638 does not include the encoded three-dimensional data 634 (No in S667), that is, when the transmission data 638 only includes the surrounding condition detection result 639 or the braking information of the surrounding vehicle 601, the surrounding condition detection unit 628 uses the first three-dimensional data 632 and the surrounding condition detection result 639 or the braking information to detect the surrounding condition of the host vehicle 600 (S673). Then, the autonomous driving control unit 629 controls the autonomous driving (autonomous operation) of the host vehicle 600 based on the surrounding condition detection result by the surrounding condition detection unit 628 (S671).

[0460] Next, the three-dimensional data transmission device 640A that transmits the transmission data 638 to the above-described three-dimensional data creation device 620A will be described. Figure 32 It is a block diagram of the three-dimensional data transmission device 640A.

[0461] Figure 32 The shown three-dimensional data transmission device 640A, in addition to Figure 28 the configuration of the three-dimensional data transmission device 640 shown, further includes a transmission availability determination unit 646. And the three-dimensional data transmission device 640A is included in the surrounding vehicle 601.

[0462] Figure 33 It is a flowchart showing an operation example of the three-dimensional data transmission device 640A. First, the three-dimensional data creation unit 641 creates the fifth three-dimensional data 652 using the sensor information 651 detected by the sensors provided in the surrounding vehicle 601 (S681).

[0463] Next, the receiving unit 642 receives a delegation signal 637 for delegating a transmission request for three-dimensional data from its own vehicle 600 (S682). Next, the transmission permission determination unit 646 determines whether to respond to the delegation indicated by the delegation signal 637 (S683). For example, the transmission permission determination unit 646 determines whether to respond to the delegation according to the content preset by the user in advance. Alternatively, the receiving unit 642 may first receive a request from the other party such as a request range, and the transmission permission determination unit 646 determines whether to respond to the delegation according to this content. For example, the transmission permission determination unit 646 may determine to respond to the delegation when holding three-dimensional data within the request range, and determine not to respond to the delegation when not having three-dimensional data within the request range.

[0464] In the case of responding to the delegation (Yes in S683), the three-dimensional data transmission device 640A sends a permission signal to its own vehicle 600, and the receiving unit 642 receives request range information 633 indicating the request range (S684). Next, the extraction unit 643 extracts the point cloud data within the request range from the fifth three-dimensional data 652 which is point cloud data, and creates transmission data 638 of the SWLD (the sixth three-dimensional data 654) including the extracted point cloud data (S685).

[0465] That is, it may be that the three-dimensional data transmission device 640A creates the seventh three-dimensional data (WLD) according to the sensor information 651, and creates the fifth three-dimensional data 652 (SWLD) by extracting data with a feature amount equal to or greater than a threshold value from the seventh three-dimensional data (WLD). And it may be that the three-dimensional data creation unit 641 creates three-dimensional data of SWLD in advance, and the extraction unit 643 extracts three-dimensional data of SWLD within the request range from the three-dimensional data of SWLD. Alternatively, the extraction unit 643 may generate three-dimensional data of SWLD within the request range according to the three-dimensional data of WLD within the request range.

[0466] Moreover, the transmission data 638 may include the surrounding condition detection result 639 within the request range performed by the surrounding vehicle 601, and the braking information of the surrounding vehicle 601. Also, the transmission data 638 may not include the sixth three-dimensional data 654, but may include at least one of the surrounding condition detection result 639 within the request range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601.

[0467] In the case where the transmission data 638 includes the sixth three-dimensional data 654 (Yes in S686), the encoding unit 644 encodes the sixth three-dimensional data 654 to generate encoded three-dimensional data 634 (S687).

[0468] Then, the transmission unit 645 sends the transmission data 638 including the encoded three-dimensional data 634 to its own vehicle 600 (S688).

[0469] Also, when the transmission data 638 does not include the sixth three-dimensional data 654 (the "No" in S686), the transmission unit 645 transmits the transmission data 638 including at least one of the surrounding condition detection result 639 of the request range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601 to its own vehicle 600 (S688).

[0470] A modification example of the present embodiment will be described below.

[0471] For example, the information transmitted from the surrounding vehicle 601 may not be the three-dimensional data or the surrounding condition detection result created by the surrounding vehicle, but may be the correct feature point information of the surrounding vehicle 601 itself. The own vehicle 600 uses the feature point information of the surrounding vehicle 601 to correct the feature point information of the vehicle traveling ahead in the point cloud data obtained by the own vehicle 600. Accordingly, the own vehicle 600 can improve the matching accuracy at the time of estimating its own position.

[0472] And, the feature point information of the vehicle traveling ahead is, for example, three-dimensional point information composed of color information and coordinate information. Accordingly, even if the sensor of the own vehicle 600 is a laser sensor or a stereo camera, the feature point information of the vehicle traveling ahead can be used regardless of its type.

[0473] In addition, the own vehicle 600 is not limited by the time of transmission, and can use the point cloud data of the SWLD when calculating the accuracy at the time of estimating its own position. For example, when the sensor of the own vehicle 600 is an imaging device such as a stereo camera, two-dimensional points on the image captured by the camera of the own vehicle 600 are detected, and the two-dimensional points are used to estimate the own position. And, while the own vehicle 600 estimates its own position, it creates the point cloud data of the surrounding objects. The own vehicle 600 projects the three-dimensional points of the SWLD among them onto the two-dimensional image again, and evaluates the accuracy of the own position estimation based on the error between the detected points on the two-dimensional image and the projected points performed again.

[0474] And, when the sensor of the own vehicle 600 is a laser sensor such as LIDAR, the own vehicle 600 evaluates the accuracy of the own position estimation based on the error calculated by the Iterative Closest Point algorithm between the SWLD of the created point cloud data and the SWLD of the three-dimensional map.

[0475] And, when the communication state via a base station or a server such as 5G is poor, the own vehicle 600 can obtain the three-dimensional map from the surrounding vehicle 601.

[0476] Furthermore, for information in the distance that cannot be obtained from the vehicles around the host vehicle 600, it can be obtained through vehicle-to-vehicle communication. For example, regarding traffic accident information that just occurred several hundred meters or several kilometers ahead, the host vehicle 600 can obtain it through communication when passing by oncoming vehicles or in the way of successive transmission among surrounding vehicles. At this time, the data form of the transmitted data is that meta-information at the upper layer of the dynamic three-dimensional map is transmitted.

[0477] Moreover, the detection results of the surrounding conditions and the information detected by the host vehicle 600 can be presented to the user through a user interface. For example, these information are presented by overlapping them onto the navigation screen or the windshield.

[0478] In addition, for vehicles equipped with cruise control that do not support autonomous driving, when a surrounding vehicle traveling in the autonomous driving mode is detected, the host vehicle can track that surrounding vehicle.

[0479] In the case where a three-dimensional map cannot be obtained or the host vehicle's position cannot be estimated due to excessive occlusion areas or other reasons, the host vehicle 600 can switch the working mode from the autonomous driving mode to the surrounding vehicle tracking mode.

[0480] Also, the vehicle being tracked can issue a warning to the user about the being-tracked status, and a user interface that allows the user to specify whether to allow tracking can also be installed. At this time, advertising displays can be designed on the tracking vehicle, and payment rewards or the like can be designed on the being-tracked side.

[0481] Furthermore, although the transmitted information is basically the SWLD as three-dimensional data, it can also be information corresponding to the request settings set in the host vehicle 600 or the public settings of the vehicle traveling ahead. For example, the transmitted information can be the WLD of dense point cloud data, or the detection results of the surrounding conditions by the vehicle traveling ahead, or the braking information of the vehicle traveling ahead.

[0482] The host vehicle 600 receives the WLD, visualizes the three-dimensional data of the WLD, and presents the visualized three-dimensional data to the driver using the GUI. At this time, the host vehicle 600 can adopt a method that enables the user to distinguish the point cloud data made by the host vehicle 600 from the received point cloud data, and use different colors to represent the information for presentation.

[0483] When the host vehicle 600 presents the information detected by the host vehicle 600 and the detection results of the surrounding vehicle 601 to the driver using the GUI, the information is presented by color-coding or the like in a way that enables the user to distinguish the information detected by the host vehicle 600 from the received detection results.

[0484] As described above, in the three-dimensional data production device 620 according to this embodiment, the three-dimensional data production unit 621 produces the first three-dimensional data 632 based on the sensor information 631 detected by the sensor. The receiving unit 624 receives the encoded three-dimensional data 634 obtained by encoding the second three-dimensional data 635. The decoding unit 625 decodes the received encoded three-dimensional data 634 to obtain the second three-dimensional data 635. The synthesizing unit 626 produces the third three-dimensional data 636 by synthesizing the first three-dimensional data 632 and the second three-dimensional data 635.

[0485] Accordingly, the three-dimensional data production device 620 can produce the detailed third three-dimensional data 636 by using the produced first three-dimensional data 632 and the received second three-dimensional data 635.

[0486] Moreover, the synthesizing unit 626 can produce the third three-dimensional data 636 having a higher density than the first three-dimensional data 632 and the second three-dimensional data 635 by synthesizing the first three-dimensional data 632 and the second three-dimensional data 635.

[0487] Moreover, the second three-dimensional data 635 (e.g., SWLD) is three-dimensional data generated by extracting data having a feature amount equal to or greater than a threshold value from the fourth three-dimensional data (e.g., WLD).

[0488] Accordingly, the three-dimensional data production device 620 can reduce the data amount of the transmitted three-dimensional data.

[0489] Moreover, the three-dimensional data production device 620 further includes a search unit 623 that searches for the transmitting device that is the source of the encoded three-dimensional data 634. The receiving unit 624 receives the encoded three-dimensional data 634 from the searched transmitting device.

[0490] Accordingly, the three-dimensional data production device 620 can, for example, determine the transmitting device holding the required three-dimensional data by searching.

[0491] Moreover, the three-dimensional data production device further includes a request range determination unit 622 that determines the request range, which is the range of the three-dimensional space for which the three-dimensional data is requested. The search unit 623 sends the request range information 633 indicating the request range to the transmitting device. The second three-dimensional data 635 includes the three-dimensional data within the request range.

[0492] Accordingly, the three-dimensional data production device 620 can not only receive the required three-dimensional data, but also reduce the data amount of the transmitted three-dimensional data.

[0493] Moreover, the request range determination unit 622 determines the spatial range including the occlusion region 604 that cannot be detected by the sensor as the request range.

[0494] Further, in the three-dimensional data transmission device 640 according to the present embodiment, the three-dimensional data creation unit 641 creates the fifth three-dimensional data 652 based on the sensor information 651 detected by the sensor. The extraction unit 643 creates the sixth three-dimensional data 654 by extracting a part of the fifth three-dimensional data 652. The encoding unit 644 generates the encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654. The transmission unit 645 transmits the encoded three-dimensional data 634.

[0495] Accordingly, the three-dimensional data transmission device 640 can not only transmit the three-dimensional data created by itself to other devices, but also reduce the amount of data of the transmitted three-dimensional data.

[0496] Further, the three-dimensional data creation unit 641 creates the seventh three-dimensional data (e.g., WLD) based on the sensor information 651 detected by the sensor, and creates the fifth three-dimensional data 652 (e.g., SWLD) by extracting data with a feature amount equal to or greater than a threshold from the seventh three-dimensional data.

[0497] Accordingly, the three-dimensional data transmission device 640 can reduce the amount of data of the transmitted three-dimensional data.

[0498] Further, the three-dimensional data transmission device 640 further includes a reception unit 642 that receives request range information 633 indicating a request range from a receiving device, where the request range is the range of the three-dimensional space for which three-dimensional data is requested. The extraction unit 643 creates the sixth three-dimensional data 654 by extracting the three-dimensional data of the request range from the fifth three-dimensional data 652. The transmission unit 645 transmits the encoded three-dimensional data 634 to the receiving device.

[0499] Accordingly, the three-dimensional data transmission device 640 can reduce the amount of data of the transmitted three-dimensional data.

[0500] (Embodiment 4)

[0501] In the present embodiment, the operation related to an abnormal condition in the self-position estimation based on the three-dimensional map will be described.

[0502] The use of autonomous movement of moving bodies such as the autonomous driving of motor vehicles, robots, or flying objects such as drones will expand in the future. As an example of a method for realizing such autonomous movement, there is a method in which a moving body estimates its own position in a three-dimensional map (self-position estimation) and travels according to the map.

[0503] The self-position estimation is achieved by matching a three-dimensional map with three-dimensional information around the own vehicle (hereinafter referred to as own vehicle detection three-dimensional data) obtained by a distance measuring instrument (LiDAR, etc.) or a stereo camera mounted on the own vehicle, and estimating the position of the own vehicle in the three-dimensional map.

[0504] A 3D map, such as the HD map proposed by HERE Technologies, etc., is not only a 3D point cloud, but may also include 2D map data such as road and intersection shape information, or information such as traffic jams and accidents that changes in real time. The 3D map is composed of multiple levels such as 3D data, 2D data, and metadata that changes in real time. The device can obtain only the required data, or can also refer to the required data.

[0505] The data of the point cloud can be the above-mentioned SWLD, or can also include point group data that is not feature points. And the transmission and reception of the data of the point cloud are basically performed in one or more random access units.

[0506] As a method for matching a 3D map with the 3D data detected by the own vehicle, the following method can be adopted. For example, the device compares the shapes of the point groups in the respective point clouds, and determines the part with a high similarity between the feature points as the same position. And when the 3D map is composed of SWLD, the device compares the feature points constituting the SWLD with the 3D feature points extracted from the 3D data detected by the own vehicle and performs matching.

[0507] Here, in order to estimate the own position with high accuracy, the following (A) and (B) need to be satisfied. (A) The 3D map and the 3D data detected by the own vehicle can already be obtained. (B) Their accuracies satisfy a predetermined standard. However, in the following abnormal situations, (A) or (B) cannot be satisfied.

[0508] (1) The 3D map cannot be obtained through the communication path.

[0509] (2) There is no 3D map, or the obtained 3D map is damaged.

[0510] (3) The sensor of the own vehicle fails, or due to bad weather, the generation accuracy of the 3D data detected by the own vehicle is insufficient.

[0511] The following describes the operations for coping with these abnormal situations. Although the following describes the operations taking a vehicle as an example, the following methods can also be applied to all moving objects that perform autonomous movement such as robots and drones.

[0512] The following will describe the configuration and operations of the 3D information processing device according to the present embodiment for coping with abnormal situations in the 3D map or the 3D data detected by the own vehicle. Figure 34 It is a block diagram showing a configuration example of the 3D information processing device 700 according to the present embodiment. Figure 35 It is a flowchart of the 3D information processing method performed by the 3D information processing device 700.

[0513] The three-dimensional information processing device 700 is mounted on a moving object such as a motor vehicle, for example. As Figure 34 shown, the three-dimensional information processing device 700 includes: a three-dimensional map acquisition unit 701, a host vehicle detection data acquisition unit 702, an abnormal situation determination unit 703, a response operation determination unit 704, and an operation control unit 705.

[0514] In addition, the three-dimensional information processing device 700 may include a camera that acquires a two-dimensional image, or may include a two-dimensional or one-dimensional sensor (not shown) such as a sensor that uses ultrasonic waves or lasers to detect a structural object or a moving object around the host vehicle. Further, the three-dimensional information processing device 700 may include a communication unit (not shown) that acquires a three-dimensional map through a mobile communication network such as 4G or 5G, vehicle-to-vehicle communication, or road-to-vehicle communication.

[0515] As Figure 35 shown, the three-dimensional map acquisition unit 701 acquires a three-dimensional map 711 near the driving route (S701). For example, the three-dimensional map acquisition unit 701 acquires the three-dimensional map 711 through a mobile communication network, vehicle-to-vehicle communication, or road-to-vehicle communication.

[0516] Next, the host vehicle detection data acquisition unit 702 acquires host vehicle detection three-dimensional data 712 based on the sensor information (S702). For example, the host vehicle detection data acquisition unit 702 generates the host vehicle detection three-dimensional data 712 based on the sensor information acquired by the sensors provided in the host vehicle.

[0517] Next, the abnormal situation determination unit 703 detects an abnormal situation by performing a pre-determined check on at least one of the acquired three-dimensional map 711 and the host vehicle detection three-dimensional data 712 (S703). That is, the abnormal situation determination unit 703 determines whether at least one of the acquired three-dimensional map 711 and the host vehicle detection three-dimensional data 712 is abnormal.

[0518] In step S703, when an abnormal situation is detected (Yes in S704), the response operation determination unit 704 determines a response operation for the abnormal situation (S705). Next, the operation control unit 705 controls the operations of the various processing units required in the implementation of the response operation, such as the three-dimensional map acquisition unit 701 (S706).

[0519] In addition, in step S703, when no abnormal situation is detected (No in S704), the three-dimensional information processing device 700 ends the processing.

[0520] Furthermore, the three-dimensional information processing device 700 estimates the own position of the vehicle equipped with the three-dimensional information processing device 700 by using the three-dimensional map 711 and the own vehicle detection three-dimensional data 712. Then, the three-dimensional information processing device 700 causes the vehicle to perform autonomous driving by using the result of the own position estimation.

[0521] Accordingly, the three-dimensional information processing device 700 obtains map data (three-dimensional map 711) including the first three-dimensional position information via a channel. For example, the first three-dimensional position information is encoded in units of partial spaces having three-dimensional coordinate information, the first three-dimensional position information includes a plurality of random access units, each of the plurality of random access units is an aggregate of one or more partial spaces, and can be independently decoded. For example, the first three-dimensional position information is data (SWLD) in which feature points where the three-dimensional feature amount exceeds a specified threshold are encoded.

[0522] In addition, the three-dimensional information processing device 700 generates second three-dimensional position information (own vehicle detection three-dimensional data 712) based on the information detected by the sensor. Then, the three-dimensional information processing device 700 determines whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information.

[0523] When the three-dimensional information processing device 700 determines that the first three-dimensional position information or the second three-dimensional position information is abnormal, it determines a response operation for the abnormality. Then, the three-dimensional information processing device 700 executes the control required for the implementation of the response operation.

[0524] Accordingly, the three-dimensional information processing device 700 can detect an abnormality in the first three-dimensional position information or the second three-dimensional position information and can perform a response operation.

[0525] The response operation in the case of abnormal situation 1, that is, when the three-dimensional map 711 cannot be obtained via communication, will be described below.

[0526] The three-dimensional map 711 is required for estimating the own position. However, when the vehicle has not previously obtained the three-dimensional map 711 corresponding to the route to the destination, it is necessary to obtain the three-dimensional map 711 via communication. However, due to channel congestion or deterioration of the radio wave reception state, etc., there may be a situation where the vehicle cannot obtain the three-dimensional map 711 on the driving route.

[0527] The abnormality determination unit 703 confirms whether the three-dimensional map 711 has been obtained for all sections on the path to the destination or for sections within a specified range from the current position. If it cannot be obtained, it is determined as abnormal situation 1. That is, the abnormality determination unit 703 determines whether the three-dimensional map 711 (the first three-dimensional position information) can be obtained via the channel. If the three-dimensional map 711 cannot be obtained via the channel, the three-dimensional map 711 is determined to be abnormal.

[0528] In the case where it is determined as abnormal situation 1, the response operation decision unit 704 selects one of the following two response operations: (1) continue with the self-position estimation, and (2) stop the self-position estimation.

[0529] First, a specific example of the response operation in the case of (1) continuing with the self-position estimation will be described. When continuing with the self-position estimation, the three-dimensional map 711 on the path to the destination is required.

[0530] For example, the vehicle determines the places within the range where the three-dimensional map 711 has been obtained and where the channel can be used, moves to those places, and obtains the three-dimensional map 711. At this time, the vehicle can obtain all the three-dimensional maps 711 to the destination, or can obtain the three-dimensional map 711 for each random access unit within the upper limit size that can be stored in the memory or HDD of the vehicle itself.

[0531] Alternatively, the following operation can also be performed. The vehicle additionally obtains the communication state on the path. When it can be predicted that the communication state on the path will deteriorate, before reaching the section with a poor communication state, the three-dimensional map 711 of that section is obtained in advance, or the three-dimensional map 711 of the maximum range that can be obtained is obtained in advance. That is, the three-dimensional information processing device 700 predicts whether the vehicle will enter a region with a poor communication state. When the three-dimensional information processing device 700 predicts that the vehicle will enter a region with a poor communication state, before the vehicle enters that region, the three-dimensional map 711 is obtained.

[0532] Furthermore, it can also be that the vehicle determines the random access unit that constitutes the minimum three-dimensional map 711 required for self-position estimation on the path, which is narrower than the normal range, and receives the determined random access unit. That is, when the three-dimensional information processing device 700 cannot obtain the three-dimensional map 711 (the first three-dimensional position information) via the channel, it can obtain the third three-dimensional position information that is narrower than the range of the first three-dimensional position information via the channel.

[0533] Also, when the vehicle cannot access the distribution server of the 3D map 711, it can obtain the 3D map 711 from a moving body such as another vehicle traveling around its own vehicle. In this case, the other vehicle or the like is a moving body that has obtained the 3D map 711 on the path to the destination and can communicate with its own vehicle.

[0534] Next, a specific example of the response operation in the case of (2) stopping the self-position estimation will be described. In this case, the 3D map 711 on the path to the destination is not required.

[0535] For example, the vehicle notifies the driver of the situation where functions such as autonomous driving based on self-position estimation cannot continue, and switches the operation mode to a manual operation mode performed by the driver.

[0536] Normally, when performing self-position estimation, although there are cases where the level varies according to the presence of a person, autonomous driving is executed. In addition, the result of self-position estimation may also utilize navigation or the like when a person is driving. Therefore, the result of self-position estimation is not necessarily required for autonomous driving.

[0537] Also, when the vehicle cannot use a normally used channel such as a mobile communication network such as 4G or 5G, it is confirmed whether the 3D map 711 can be obtained via another communication path such as Wi-Fi (registered trademark) or millimeter-wave communication between the road and the vehicle, or vehicle-to-vehicle communication, and the used channel can be switched to a channel capable of obtaining the 3D map 711.

[0538] Also, when the vehicle cannot obtain the 3D map 711, it can also obtain a 2D map and continue autonomous driving by using the 2D map and the three-dimensional data 712 detected by its own vehicle. That is, when the 3D information processing device 700 cannot obtain the 3D map 711 via the channel, it can obtain map data (2D map) including 2D position information via the channel, and perform self-position estimation of the vehicle by using the 2D position information and the three-dimensional data 712 detected by its own vehicle.

[0539] Specifically, the vehicle uses the 2D map and the three-dimensional data 712 detected by its own vehicle in self-position estimation, and uses the three-dimensional data 712 detected by its own vehicle in the detection of surrounding vehicles, pedestrians, and obstacles.

[0540] Here, map data such as HD maps can include not only a three-dimensional map 711 composed of three-dimensional point clouds or the like, but also two-dimensional map data (two-dimensional map), a simplified version of map data obtained by extracting characteristic information such as road shapes or intersections from the two-dimensional map data, and metadata showing real-time information such as traffic jams, accidents, or construction. For example, the map data has a layer structure in which three-dimensional data (three-dimensional map 711), two-dimensional data (two-dimensional map), and metadata are arranged in sequence from the lower layer.

[0541] Here, compared with three-dimensional data, two-dimensional data is smaller in size. Therefore, even when the communication state is poor, the vehicle can obtain the two-dimensional map. Moreover, the vehicle can uniformly obtain a large-scale two-dimensional map in an area with good communication state. Thus, when the channel state is poor and it is difficult to obtain the three-dimensional map 711, the vehicle can receive the layer including the two-dimensional map without receiving the three-dimensional map 711. In addition, since the data size of the metadata is small, for example, the vehicle can always receive the metadata without being affected by the communication state.

[0542] In the method of estimating the vehicle's own position using the two-dimensional map and the vehicle's own detection three-dimensional data 712, for example, there are the following two methods.

[0543] The first method is a method of performing matching of two-dimensional feature quantities. Specifically, the vehicle extracts two-dimensional feature quantities from the vehicle's own detection three-dimensional data 712 and performs matching of the extracted two-dimensional feature quantities with the two-dimensional map.

[0544] For example, the vehicle projects the vehicle's own detection three-dimensional data 712 onto the same plane as the two-dimensional map and performs matching of the obtained two-dimensional data with the two-dimensional map. The matching is performed using two-dimensional image feature quantities extracted from both.

[0545] When the three-dimensional map 711 includes SWLD, the three-dimensional map 711 can store both three-dimensional feature quantities at feature points in the three-dimensional space and two-dimensional feature quantities in the same plane as the two-dimensional map. For example, identification information is given to the two-dimensional feature quantities. Alternatively, the two-dimensional feature quantities are stored in a layer different from the three-dimensional data and the two-dimensional map, and the vehicle obtains the data of the two-dimensional feature quantities while obtaining the two-dimensional map.

[0546] When the two-dimensional map represents information on positions at different heights (not in the same plane) from the ground, such as white lines, guardrails, and buildings inside the road, in the same map, the vehicle extracts feature quantities from multiple height data of the vehicle's own detection three-dimensional data 712.

[0547] Moreover, the information showing the correspondence between the feature points in the two-dimensional map and the feature points in the three-dimensional map 711 can be stored as meta-information of the map data.

[0548] The second method is a method for matching three-dimensional feature quantities. Specifically, the vehicle obtains three-dimensional feature quantities corresponding to the feature points of the two-dimensional map, and matches the obtained three-dimensional feature quantities with the three-dimensional feature quantities of its own vehicle detection three-dimensional data 712.

[0549] Specifically, the three-dimensional feature quantities corresponding to the feature points of the two-dimensional map are stored in the map data. When the vehicle obtains the two-dimensional map, it also obtains the three-dimensional feature quantities. In addition, when the three-dimensional map 711 includes SWLD, by giving the information of the feature points corresponding to the feature points of the two-dimensional map in the feature points for identifying SWLD, the vehicle can determine the three-dimensional feature quantities obtained together with the two-dimensional map according to the identification information. In addition, in this case, since it is only necessary to represent the two-dimensional position, the data amount can be reduced compared with the case of representing the three-dimensional position. Figure 1 In addition, in this case, since it is only necessary to represent the two-dimensional position, the data amount can be reduced compared with the case of representing the three-dimensional position.

[0550] Moreover, when estimating the vehicle's own position using the two-dimensional map, the accuracy of the vehicle's own position estimation is lower than that of the three-dimensional map 711. Therefore, the vehicle determines whether to continue autonomous driving even when the estimation accuracy is reduced, and only continues autonomous driving when it is determined that it can continue.

[0551] Whether autonomous driving can continue is affected by the following factors, that is, whether the road on which the vehicle is traveling is an urban area, or a road such as a highway where fewer other vehicles or pedestrians enter, or the driving environment such as the road width and the degree of chaos of the road (the density of vehicles or pedestrians). Moreover, markers for enabling sensors such as cameras to identify can be arranged in enterprise sites, streets, or buildings. In these specific areas, since the markers can be accurately identified by two-dimensional sensors, for example, by including the position information of the markers in the two-dimensional map, high-precision vehicle's own position estimation can be performed.

[0552] Furthermore, by including in the map the identification information indicating whether each area is a specific area, the vehicle can determine whether the vehicle exists in a specific area. When the vehicle exists in a specific area, it is determined to continue autonomous driving. In this way, the vehicle can determine whether it can continue autonomous driving according to the accuracy of the vehicle's own position estimation when using the two-dimensional map or the driving environment of the vehicle.

[0553] In this way, the three-dimensional information processing device 700 can determine whether to perform the autonomous driving of the vehicle based on the driving environment (the moving environment of the moving body) using the result of the vehicle's own position estimation obtained by using the two-dimensional map and the vehicle's own vehicle detection three-dimensional data 712.

[0554] Also, it is possible that the vehicle does not determine whether it can continue with autonomous driving, but instead switches the level (mode) of autonomous driving according to the accuracy of its own position estimation or the driving environment of the vehicle. Here, the switching of the level (mode) of autonomous driving refers to, for example, restricting the speed, increasing the amount of operation by the driver (reducing the automatic level of autonomous driving), obtaining the driving information of the vehicle ahead and switching the mode with reference to this information, obtaining the driving information of the vehicle set to the same destination and switching the mode of autonomous driving using this information, etc.

[0555] Also, the map may include information corresponding to the position information and showing the recommended level of autonomous driving in the case of estimating its own position using a two-dimensional map. The recommended level may be metadata that changes dynamically according to traffic volume, etc. Accordingly, the vehicle can determine the level only by obtaining the information in the map without having to sequentially judge the level according to the surrounding environment, etc. Also, by multiple vehicles referring to the same map, the level of autonomous driving of each vehicle can be kept stable. In addition, the recommended level may not be a recommended level but a level that must be complied with.

[0556] Also, the vehicle can switch the level of autonomous driving according to whether there is a driver (whether it is manned or unmanned). For example, the vehicle reduces the level of autonomous driving when there is a person and stops when there is no person. The vehicle judges the position where it can safely stop by recognizing the surrounding pedestrians, vehicles, and traffic signs. Or, the map may include position information showing the position where the vehicle can safely stop, and the vehicle can refer to this position information to judge the position where it can safely stop.

[0557] Next, the response work in the case of abnormal situation 2, that is, when the three-dimensional map 711 does not exist or the obtained three-dimensional map 711 is damaged, will be described.

[0558] The abnormal situation determination unit 703 confirms which of the following (1) and (2) the situation belongs to, and determines it as abnormal situation 2 when it belongs to one of them. (1) The three-dimensional map 711 in a part or all of the intervals on the path to the destination does not exist in the distribution server, etc. that is the access destination and cannot be obtained, or (2) a part or all of the obtained three-dimensional map 711 is damaged. That is, the abnormal situation determination unit 703 determines whether the data of the three-dimensional map 711 is complete, and determines the three-dimensional map 711 as abnormal when the data of the three-dimensional map 711 is incomplete.

[0559] In the case where it is determined as abnormal situation 2, the following response work is executed. First, an example of the response work in the case where the three-dimensional map 711 cannot be obtained will be described.

[0560] For example, the vehicle sets a route that does not pass through an area without the 3D map 711.

[0561] Moreover, when the vehicle cannot set an alternative route because there is no alternative route or even if there is an alternative route, the distance will be greatly increased, etc., the vehicle sets a route including the area without the 3D map 711. And the vehicle notifies the driver to switch the driving mode in this area to switch the driving mode to the manual operation mode.

[0562] (2) When part or all of the obtained 3D map 711 is damaged, the following countermeasures are taken.

[0563] The vehicle determines the damaged part in the 3D map 711, requests the data of the damaged part through communication, obtains the data of the damaged part, and updates the 3D map 711 using the obtained data. At this time, the vehicle can specify the damaged part by position information such as absolute coordinates or relative coordinates in the 3D map 711, or can also specify it by the index number of the random access unit constituting the damaged part. In this case, the vehicle replaces the random access unit including the damaged part with the obtained random access unit.

[0564] Next, the countermeasures in the case of abnormal situation 3, that is, when the sensors of the own vehicle cannot generate the own vehicle detection 3D data 712 due to a failure or bad weather, will be described.

[0565] The abnormal situation determination unit 703 confirms whether the generation error of the own vehicle detection 3D data 712 is within the allowable range. If it is not within the allowable range, it is determined as abnormal situation 3. That is, the abnormal situation determination unit 703 determines whether the generation accuracy of the data of the own vehicle detection 3D data 712 is above the reference value. When the generation accuracy of the data of the own vehicle detection 3D data 712 is not above the reference value, the own vehicle detection 3D data 712 is determined as abnormal.

[0566] As a method for confirming whether the generation error of the own vehicle detection 3D data 712 is within the allowable range, the following method can be adopted.

[0567] According to the resolution in the depth direction and the scanning direction of the 3D sensors of the own vehicle such as a rangefinder or a stereo camera, or the density of the point cloud that can be generated, etc., the spatial resolution of the own vehicle detection 3D data 712 during normal operation is determined in advance. And the vehicle obtains the spatial resolution of the 3D map 711 according to the meta-information included in the 3D map 711.

[0568] The vehicle uses the spatial resolutions of both to estimate a reference value of a matching error when matching the self-vehicle detected three-dimensional data 712 and the three-dimensional map 711 based on three-dimensional feature quantities and the like. As the matching error, statistics such as the error of the three-dimensional feature quantity of each feature point, the average value of the errors of the three-dimensional feature quantities between multiple feature points, or the error of the spatial distance between multiple feature points can be used. A permissible range of deviation from the reference value is set in advance.

[0569] When the matching error between the self-vehicle detected three-dimensional data 712 and the three-dimensional map 711 generated by the vehicle before starting to drive or during driving is not within the permissible range, it is determined as an abnormal situation 3.

[0570] Alternatively, the vehicle may use a test pattern having a known three-dimensional shape for accuracy inspection to obtain the self-vehicle detected three-dimensional data 712 for the test pattern such as before starting to drive, and determine whether it is an abnormal situation 3 based on whether the shape error is within the permissible range.

[0571] For example, the vehicle makes the above determination each time before starting to drive. Alternatively, the vehicle makes the above determination at regular time intervals or the like during driving to obtain the time-series change of the matching error. When the matching error has an increasing tendency, even if the error is within the permissible range, it may be determined as an abnormal situation 3. And, based on the time-series change, when it is possible to predict an abnormality, the vehicle can notify the user of the predicted abnormal situation by displaying a message urging inspection or repair. And, the vehicle can determine, through the time-series change, an abnormality caused by a temporary cause such as bad weather and an abnormality caused by a sensor failure, and notify only the user of the abnormality caused by the sensor failure.

[0572] And, when the vehicle determines that it is an abnormal situation 3, it selects any one of the following three response operations, or selectively executes them: (1) operating an emergency substitute sensor (rescue mode), (2) switching the operation mode, (3) performing work calibration of the three-dimensional sensor.

[0573] First, (1) the case of operating an emergency substitute sensor will be described. The vehicle operates an emergency substitute sensor different from the three-dimensional sensor used during normal operation. That is, when the data generation accuracy of the self-vehicle detected three-dimensional data 712 by the three-dimensional information processing device 700 is not above the reference value, the self-vehicle detected three-dimensional data 712 (the fourth three-dimensional position information) is generated based on information different from the normal sensor and substituting for the information detected by the sensor.

[0574] Specifically, when a vehicle uses multiple cameras or LiDARs to obtain three-dimensional data 712 for its own vehicle detection, the vehicle determines a malfunctioning sensor based on the direction in which the matching error of the three-dimensional data 712 for its own vehicle detection exceeds the allowable range. Then, the vehicle causes a substitute sensor corresponding to the malfunctioning sensor to operate.

[0575] The substitute sensor can be a three-dimensional sensor, a camera that obtains a two-dimensional image, or a one-dimensional sensor such as an ultrasonic sensor. When the substitute sensor is a sensor other than a three-dimensional sensor, since there may be a reduction in the accuracy of self-position estimation or self-position estimation may not be possible, the vehicle can switch the autonomous driving mode according to the type of the substitute sensor.

[0576] For example, when the substitute sensor is a three-dimensional sensor, the vehicle continues the autonomous driving mode. And when the substitute sensor is a two-dimensional sensor, the vehicle changes the operation mode from full autonomous driving to semi-autonomous driving premised on human operation. And when the substitute sensor is a one-dimensional sensor, the vehicle switches the operation mode to a manual operation mode in which automatic braking control cannot be performed.

[0577] Also, the vehicle can switch the autonomous driving mode according to the driving environment. For example, when the substitute sensor is a two-dimensional sensor, if the vehicle is driving on a highway, it continues the full autonomous driving mode, and if it is driving in an urban area, it switches the operation mode to semi-autonomous driving.

[0578] Also, when there is no substitute sensor, if a sufficient number of feature points can be obtained only by the normally operating sensors, the vehicle can continue self-position estimation. However, since detection in a specific direction cannot be performed, the vehicle switches the operation mode to semi-autonomous driving or manual operation mode.

[0579] Next, (2) the response work for switching the operation mode will be described. The vehicle switches the operation mode from the autonomous driving mode to the manual operation mode. Or, the vehicle can continue autonomous driving until it reaches a shoulder or other place where it can stop safely and then stop. And after stopping, the vehicle can switch the operation mode to the manual operation mode. In this way, when the generation accuracy of the three-dimensional data 712 for its own vehicle detection by the three-dimensional information processing device 700 is not above the reference value, the autonomous driving mode is switched.

[0580] Next, (3) will describe the countermeasures for performing the operation calibration of the three-dimensional sensor. The vehicle determines the malfunctioning three-dimensional sensor based on the direction in which the matching error occurs, etc., and calibrates the determined sensor. Specifically, as sensors, when using multiple LiDARs or cameras, a part of the three-dimensional space reconstructed by each sensor overlaps. That is, the data of the overlapping part is obtained by multiple sensors. For the overlapping part, the three-dimensional point cloud data obtained by the normal sensor and the malfunctioning sensor is different. Therefore, the vehicle adjusts the operations of the predetermined parts, such as performing the origin calibration of the LiDAR or adjusting the exposure and focus of the camera, so that the malfunctioning sensor can obtain three-dimensional point cloud data equivalent to that of the normal sensor.

[0581] After the adjustment, if the matching error can be within the allowable range, the vehicle continues the previous operation mode. Additionally, after the adjustment, if the matching accuracy cannot be within the allowable range, the vehicle performs the countermeasures in (1) above to operate the emergency replacement sensor or (2) perform the countermeasures to switch the operation mode.

[0582] In this way, when the generation accuracy of the three-dimensional data 712 detected by the own vehicle by the three-dimensional information processing device 700 is not above the reference value, the operation calibration of the sensor is performed.

[0583] The following describes the selection method of the countermeasures. The countermeasures can be selected by a user such as the driver, or can be automatically selected by the vehicle without passing through the user.

[0584] Moreover, the vehicle can also switch the control according to whether the driver is on board. For example, when the driver is on board, the vehicle gives priority to switching to the manual operation mode. Additionally, when the driver is not on board, the vehicle gives priority to the mode of moving to a safe place to stop.

[0585] The information indicating the stop place can be included in the three-dimensional map 711 as meta-information. Or, the vehicle can send a response request for the stop place to the service that manages the operation information of the autonomous driving, so as to obtain the information indicating the stop place.

[0586] Moreover, in the case where the vehicle is running on a specified route, etc., the operation mode of the vehicle can be shifted to the mode in which the operation of the vehicle is managed by an operator via a channel. In particular, in a vehicle traveling in the fully autonomous driving mode, the risk of abnormality of the own position estimation function is high. Therefore, when an abnormality is detected or when the detected abnormality cannot be corrected, the vehicle notifies the service that manages the operation information of the occurrence of the abnormality via the channel. This service can notify the presence of the abnormal vehicle to the vehicles traveling around the vehicle, etc., or issue an instruction to vacate the nearby stop place.

[0587] Also, when detecting an abnormal situation, the vehicle can travel at a slower speed than normal.

[0588] When the vehicle is an autonomous vehicle providing vehicle dispatching services such as taxis and an abnormal situation occurs to the vehicle, the vehicle contacts the operation management center to stop at a safe location. Also, a substitute vehicle can be dispatched for the vehicle dispatching service. Or it can be that the user of the vehicle dispatching service drives the vehicle. In these situations, discounts on fees or awarding special points, etc. can be used in combination.

[0589] Also, in the method for dealing with abnormal situation 1, although the method for estimating its own position based on a two-dimensional map has been described, even under normal circumstances, a two-dimensional map can be used for estimating its own position. Figure 36 It is a flowchart of the self-position estimation process in this case.

[0590] First, the vehicle obtains a three-dimensional map 711 near the driving path (S711). Next, the vehicle obtains its own vehicle detection three-dimensional data 712 based on the sensor information (S712).

[0591] Next, the vehicle determines whether a three-dimensional map 711 is required when estimating its own position (S713). Specifically, the vehicle determines whether a three-dimensional map 711 is required based on the accuracy of self-position estimation when using the two-dimensional map and the driving environment. For example, the same method as the method for dealing with abnormal situation 1 described above is adopted.

[0592] In the case where it is determined that the three-dimensional map 711 is not required (the "no" in S714), the vehicle obtains a two-dimensional map (S715). At this time, the vehicle can simultaneously obtain the additional information described in the method for dealing with abnormal situation 1. Also, the vehicle can generate a two-dimensional map based on the three-dimensional map 711. For example, the vehicle can extract an arbitrary plane from the three-dimensional map 711 to generate a two-dimensional map.

[0593] Next, the vehicle uses its own vehicle detection three-dimensional data 712 and the two-dimensional map to estimate its own position (S716). Also, the method for self-position estimation using a two-dimensional map is, for example, the same as the method described in the method for dealing with abnormal situation 1 above.

[0594] In addition, in the case where it is determined that the three-dimensional map 711 is required (the "yes" in S714), the vehicle obtains the three-dimensional map 711 (S717). Next, the vehicle uses its own vehicle detection three-dimensional data 712 and the three-dimensional map 711 to estimate its own position (S718).

[0595] In addition, the vehicle can switch between basically using a two-dimensional map and using a three-dimensional map 711 according to the corresponding speed of the communication device of its own vehicle or the condition of the channel. For example, while receiving the three-dimensional map 711, a communication speed required during driving is preset in advance. When the communication speed during driving is equal to or lower than the set value, the vehicle basically uses the two-dimensional map. When the communication speed during driving is greater than the set value, the vehicle basically uses the three-dimensional map 711. In addition, the vehicle can also basically use the two-dimensional map without making a switching judgment on whether to use the two-dimensional map or the three-dimensional map.

[0596] (Embodiment 5)

[0597] In this embodiment, a method for sending three-dimensional data to a following vehicle and the like will be described. Figure 37 An example of the object space of the three-dimensional data sent to a following vehicle or the like is shown.

[0598] The vehicle 801 sends three-dimensional data such as point cloud data (point group) included in a rectangular parallelepiped space 802 with a width W, a height H, and a depth D at a distance L in front of the vehicle 801 to the traffic cloud monitoring for monitoring the road condition or a following vehicle at a time interval of Δt.

[0599] When a vehicle or a person enters the space 802 or the like from the outside, and thus the three-dimensional data included in the previously transmitted space 802 has changed, the vehicle 801 also sends the three-dimensional data of the changed space.

[0600] In addition, although Figure 37 shows an example in which the shape of the space 802 is a rectangular parallelepiped, the space 802 only needs to include the space on the front road that is a blind spot from the perspective of the following vehicle, and it is not necessarily a rectangular parallelepiped.

[0601] The distance L is preferably set to a distance at which the following vehicle that has received the three-dimensional data can safely stop. For example, the distance L is set to the sum of the following distances: the distance that the following vehicle moves during the reception of the three-dimensional data, the distance that the following vehicle moves until it starts to decelerate according to the received data, and the distance required for the following vehicle to safely stop in principle. Since these distances change according to the speed, as shown by L = a×V + b (a and b are constants), the distance L can change according to the speed V of the vehicle.

[0602] The width W is set to a value that is at least larger than the width of the lane in which the vehicle 801 travels. More preferably, the width W is set to a size that includes adjacent spaces such as the left and right driving lanes or the curb strip.

[0603] Although the depth D can be a fixed value, it can also vary according to the vehicle speed V as shown by D = c×V + d (where c and d are constants). Also, by setting D such that D > V×Δt, the transmission space can be made to overlap with the space that has been transmitted in the past. Accordingly, the vehicle 801 can more reliably and without omission transmit the space on the driving route to the following vehicle or the like.

[0604] In this way, by limiting the three-dimensional data transmitted by the vehicle 801 to the space that is useful for the following vehicle, the capacity of the transmitted three-dimensional data can be effectively reduced, achieving low latency and low cost of communication.

[0605] Next, the configuration of the three-dimensional data production device 810 according to the present embodiment will be described. Figure 38 FIG. is a block diagram showing a configuration example of the three-dimensional data production device 810 according to the present embodiment. The three-dimensional data production device 810 is mounted on the vehicle 801, for example. The three-dimensional data production device 810 transmits and receives three-dimensional data to and from external traffic cloud monitoring, the preceding vehicle, or the following vehicle, and at the same time produces and stores the three-dimensional data.

[0606] The three-dimensional data production device 810 includes: a data reception unit 811, a communication unit 812, a reception control unit 813, a format conversion unit 814, a plurality of sensors 815, a three-dimensional data production unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0607] The data reception unit 811 receives three-dimensional data 831 from traffic cloud monitoring or the preceding vehicle. The three-dimensional data 831 includes, for example, information on areas that cannot be detected by the sensors 815 of its own vehicle, and this information is, for example, point cloud data, visible light images, depth information, sensor position information, or speed information, etc.

[0608] The communication unit 812 communicates with traffic cloud monitoring or the preceding vehicle, and transmits a data transmission request or the like to traffic cloud monitoring or the preceding vehicle.

[0609] The reception control unit 813 exchanges information such as the corresponding format with the communication partner via the communication unit 812, and establishes communication with the communication partner.

[0610] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion or the like on the three-dimensional data 831 received by the data reception unit 811. Also, when the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0611] A plurality of sensors 815 are a group of sensors such as LIDAR, visible light cameras, or infrared cameras that obtain information about the exterior of the vehicle 801, and generate sensor information 833. For example, when the sensor 815 is a laser sensor such as LIDAR, the sensor information 833 is three-dimensional data such as point cloud data (point group data). Additionally, the sensor 815 may not be plural.

[0612] The three-dimensional data creation unit 816 generates three-dimensional data 834 based on the sensor information 833. The three-dimensional data 834 includes, for example, information such as point cloud data, visible light images, depth information, sensor position information, or speed information.

[0613] The three-dimensional data synthesis unit 817 synthesizes the three-dimensional data 832 created by traffic cloud monitoring or a preceding vehicle, etc., into the three-dimensional data 834 created based on the sensor information 833 of its own vehicle, thereby enabling the construction of three-dimensional data 835 that also includes the space in front of the preceding vehicle that cannot be detected by the sensors 815 of its own vehicle.

[0614] The three-dimensional data storage unit 818 stores the generated three-dimensional data 835, etc.

[0615] The communication unit 819 communicates with traffic cloud monitoring or a following vehicle, and sends a data transmission request, etc., to traffic cloud monitoring or a following vehicle.

[0616] The transmission control unit 820 exchanges information such as the corresponding format, etc., with the communication partner via the communication unit 819, and establishes communication with the communication partner. And the transmission control unit 820 determines the transmission area of the space of the three-dimensional data to be transmitted based on the three-dimensional data construction information of the three-dimensional data 832 generated in the three-dimensional data synthesis unit 817 and the data transmission request from the communication partner.

[0617] Specifically, the transmission control unit 820 determines the transmission area that includes the space in front of its own vehicle that cannot be detected by the sensors of the following vehicle in accordance with the data transmission request from traffic cloud monitoring or a following vehicle. And the transmission control unit 820 determines the transmission area by judging, based on the three-dimensional data construction information, whether there is an update of the space that can be transmitted or the transmitted space, etc. For example, the transmission control unit 820 determines as the transmission area the area that is both specified by the data transmission request and where the corresponding three-dimensional data 835 exists. And the transmission control unit 820 notifies the format conversion unit 821 of the format corresponding to the communication partner and the transmission area.

[0618] The format conversion unit 821 generates the three-dimensional data 837 by converting the three-dimensional data 836 of the transmission area in the three-dimensional data 835 stored in the three-dimensional data storage unit 818 into a format corresponding to the receiving side. Additionally, the format conversion unit 821 can compress or encode the three-dimensional data 837 to reduce the data volume.

[0619] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic cloud monitoring or the following vehicle. This three-dimensional data 837 includes, for example, information on an area that is a blind spot for the following vehicle, such as point cloud data, visible light images, depth information, or sensor position information in front of the own vehicle.

[0620] In addition, although the format conversion units 814 and 821 are used as examples for format conversion and the like here, format conversion may not be performed.

[0621] With this configuration, the three-dimensional data production device 810 obtains the three-dimensional data 831 of an area that cannot be detected by the sensor 815 of the own vehicle from the outside, and generates the three-dimensional data 835 by synthesizing the three-dimensional data 831 and the three-dimensional data 834 based on the sensor information 833 detected by the sensor 815 of the own vehicle. Accordingly, the three-dimensional data production device 810 can generate the three-dimensional data of a range that cannot be detected by the sensor 815 of the own vehicle.

[0622] Moreover, the three-dimensional data production device 810 can transmit the three-dimensional data of the space in front of the own vehicle, which cannot be detected by the sensors of the following vehicle, to the traffic cloud monitoring or the following vehicle, etc., in accordance with a data transmission request from the traffic cloud monitoring or the following vehicle.

[0623] Next, the transmission order of the three-dimensional data to the following vehicle in the three-dimensional data production device 810 will be described. Figure 39 It is a flowchart showing an example of the order in which the three-dimensional data production device 810 transmits the three-dimensional data to the traffic cloud monitoring or the following vehicle.

[0624] First, the three-dimensional data production device 810 generates and updates the three-dimensional data 835 of the space 802 including the space on the road in front of the own vehicle 801 (S801). Specifically, the three-dimensional data production device 810 synthesizes the three-dimensional data 831 produced by the traffic cloud monitoring or the vehicle in front, etc., into the three-dimensional data 834 produced based on the sensor information 833 of the own vehicle 801, and constructs the three-dimensional data 835 that also includes the space in front of the vehicle in front that cannot be detected by the sensor 815 of the own vehicle through this synthesis.

[0625] Next, the three-dimensional data production device 810 determines whether the three-dimensional data 835 included in the already transmitted space has changed (S802).

[0626] When the three-dimensional data 835 included in the space changes due to a vehicle or a person entering the transmitted space from the outside, etc. (Yes in S802), the three-dimensional data production device 810 sends the three-dimensional data including the three-dimensional data 835 of the space where the change has occurred to the traffic cloud monitoring or the following vehicle (S803).

[0627] In addition, although the three-dimensional data production device 810 can send the three-dimensional data of the space where the change has occurred at the transmission timing of sending the three-dimensional data at a prescribed interval, it can also be sent immediately after the change is detected. That is, the three-dimensional data production device 810 can give priority to sending the three-dimensional data of the space where the change has occurred over the three-dimensional data sent at a prescribed interval.

[0628] Moreover, the three-dimensional data production device 810 can also send all the three-dimensional data of the space where the change has occurred as the three-dimensional data of the space where the change has occurred, or can also send only the difference of the three-dimensional data (for example, information of three-dimensional points that appear or disappear, or displacement information of three-dimensional points, etc.).

[0629] Also, it can be that the three-dimensional data production device 810 sends metadata related to the danger avoidance operation of its own vehicle, such as an emergency brake alarm, to the following vehicle before sending the three-dimensional data of the space where the change has occurred. Accordingly, the following vehicle can confirm the emergency brake, etc. of the vehicle ahead earlier, and thus can start the danger avoidance operation such as decelerating as soon as possible.

[0630] When the three-dimensional data 835 included in the transmitted space does not change (No in S802) or after step S803, the three-dimensional data production device 810 sends the three-dimensional data included in the space of a prescribed shape on the front distance L of its own vehicle 801 to the traffic cloud monitoring or the following vehicle (S804).

[0631] Moreover, for example, the processes of steps S801 to S804 are repeatedly executed at a prescribed time interval.

[0632] Also, when there is no difference between the three-dimensional data 835 of the currently transmitted target space 802 and the three-dimensional map, the three-dimensional data production device 810 may not send the three-dimensional data 837 of the space 802.

[0633] Figure 40 It is a flowchart showing the operation of the three-dimensional data production device 810 in this case.

[0634] First, the three-dimensional data production device 810 generates and updates the three-dimensional data 835 of the space including the space 802 on the front road of its own vehicle 801 (S811).

[0635] Next, the three-dimensional data production device 810 determines whether there is an update in the three-dimensional data 835 of the generated space 802 that is different from the three-dimensional map (S812). That is, the three-dimensional data production device 810 determines whether there is a difference between the three-dimensional data 835 of the generated space 802 and the three-dimensional map. Here, the three-dimensional map is three-dimensional map information managed by devices on the infrastructure side such as traffic cloud monitoring. For example, this three-dimensional map can be obtained as the three-dimensional data 831.

[0636] In the case of an update (Yes in S812), the three-dimensional data production device 810, as described above, sends the three-dimensional data included in the space 802 to traffic cloud monitoring or the following vehicle (S813).

[0637] In addition, in the case of no update (No in S812), the three-dimensional data production device 810 does not send the three-dimensional data included in the space 802 to traffic cloud monitoring and the following vehicle (S814). In addition, the three-dimensional data production device 810 can be controlled so that the three-dimensional data of the space 802 is not sent by setting the volume of the space 802 to zero. Also, the three-dimensional data production device 810 can send information indicating that there is no update in the space 802 to traffic cloud monitoring or the following vehicle.

[0638] As described above, for example, when there are no obstacles on the road, there is no difference between the generated three-dimensional data 835 and the three-dimensional map on the infrastructure side, so data is not sent. In this way, the transmission of unnecessary data can be suppressed.

[0639] In addition, although the three-dimensional data production device 810 is mounted on a vehicle as an example in the above description, the three-dimensional data production device 810 is not limited to being mounted on a vehicle and can be mounted on any moving body.

[0640] As shown above, the three-dimensional data production device 810 according to the present embodiment is mounted on a moving body equipped with a sensor 815 and a communication unit (such as a data receiving unit 811 or a data sending unit 822) that transmits and receives three-dimensional data to and from the outside. The three-dimensional data production device 810 produces three-dimensional data 835 (second three-dimensional data) based on the sensor information 833 detected by the sensor 815 and the three-dimensional data 831 (first three-dimensional data) received by the data receiving unit 811. The three-dimensional data production device 810 sends the three-dimensional data 837, which is part of the three-dimensional data 835, to the outside.

[0641] Accordingly, the three-dimensional data creation device 810 can generate three-dimensional data of a range that its own vehicle cannot detect. Also, the three-dimensional data creation device 810 can send the three-dimensional data of a range that other vehicles or the like cannot detect to such other vehicles.

[0642] Also, the three-dimensional data creation device 810 repeatedly performs the creation of the three-dimensional data 835 and the sending of the three-dimensional data 837 at a prescribed interval. The three-dimensional data 837 is the three-dimensional data of a small space 802 having a prescribed size located at a prescribed distance L in the forward direction of the moving direction of the vehicle 801 from the position of the current vehicle 801.

[0643] Accordingly, since the range of the sent three-dimensional data 837 is limited, the data amount of the sent three-dimensional data 837 can be reduced.

[0644] Also, the prescribed distance L changes according to the moving speed V of the vehicle 801. For example, the greater the moving speed V, the longer the prescribed distance L. Accordingly, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801, and can send the three-dimensional data 837 of the small space 802 to a following vehicle or the like.

[0645] Also, the prescribed size changes according to the moving speed V of the vehicle 801. For example, the greater the moving speed V, the greater the prescribed size. For example, the greater the moving speed V, the greater the depth D, which is the length in the moving direction of the vehicle of the small space 802. Accordingly, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801, and can send the three-dimensional data 837 of the small space 802 to a following vehicle or the like.

[0646] Also, the three-dimensional data creation device 810 determines whether there is a change in the three-dimensional data 835 of the small space 802 corresponding to the sent three-dimensional data 837. When the three-dimensional data creation device 810 determines that there is a change, it sends the three-dimensional data 837 (the fourth three-dimensional data), which is at least a part of the three-dimensional data 835 as the change, to an external following vehicle or the like.

[0647] Accordingly, the vehicle 801 can send the three-dimensional data 837 of the space where the change has occurred to a following vehicle or the like.

[0648] Furthermore, the 3D data production device 810 transmits the changed 3D data 837 (the fourth 3D data) preferentially over the normal 3D data 837 (the third 3D data) that is regularly transmitted. Specifically, the 3D data production device 810 transmits the changed 3D data 837 (the fourth 3D data) before transmitting the normal 3D data 837 (the third 3D data) that is regularly transmitted. That is, the 3D data production device 810 does not wait for the transmission of the normal 3D data 837 that is regularly transmitted, but transmits the changed 3D data 837 (the fourth 3D data) irregularly.

[0649] Accordingly, since the vehicle 801 can preferentially transmit the 3D data 837 of the changed space to the following vehicle or the like, the following vehicle or the like can make a quick judgment based on the 3D data.

[0650] Furthermore, the changed 3D data 837 (the fourth 3D data) shows the difference between the 3D data 835 of the small space 802 corresponding to the transmitted 3D data 837 and the changed 3D data 835. Accordingly, the data volume of the transmitted 3D data 837 can be reduced.

[0651] Moreover, when there is no difference between the 3D data 837 of the small space 802 and the 3D data 831 of the small space 802, the 3D data production device 810 does not transmit the 3D data 837 of the small space 802. Also, the 3D data production device 810 may transmit information indicating that there is no difference between the 3D data 837 of the small space 802 and the 3D data 831 of the small space 802 to the outside.

[0652] Accordingly, since the transmission of unnecessary 3D data 837 can be suppressed, the data volume of the transmitted 3D data 837 can be reduced.

[0653] The 3D data production device related to the embodiment of the present application has been described above, but the present application is not limited by these embodiments.

[0654] Moreover, each processing unit included in the 3D data production device related to the above embodiment is typically implemented as an LSI of an integrated circuit. These can be made into individual chips, or a part or all of them can be made into one chip.

[0655] Furthermore, the integration into an integrated circuit is not limited to the LSI, and can also be implemented by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconstruct the connection or setting of the circuit units inside the LSI can also be used.

[0656] Also, in each of the above-described embodiments, each component may be constituted by dedicated hardware, or may be implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0657] Also, the present application can be implemented as a three-dimensional data production method executed by a three-dimensional data production apparatus.

[0658] Also, the division of the functional blocks in the block diagram is an example. A plurality of functional blocks may be implemented as one functional block, one functional block may be divided into a plurality of blocks, and a part of the functions may be moved to other functional blocks. Also, the functions of a plurality of functional blocks having similar functions may be processed in parallel or time-divided by a single piece of hardware or software.

[0659] Also, regarding the execution order of each step in the flowchart, it is an example given for specifically explaining the present application, and it may also be an order other than the above. Also, a part of the above steps may be executed simultaneously (in parallel) with other steps.

[0660] The three-dimensional data production apparatus related to one or more aspects has been described above based on the embodiments. However, the present application is not limited to these embodiments. Within the scope not departing from the gist of the present application, forms obtained by performing various modifications conceivable by those skilled in the art on these embodiments, and forms obtained by combining components in different embodiments are all included within the scope of one or more aspects.

[0661] Industrial Applicability

[0662] The present application can be applied to a three-dimensional data production apparatus.

[0663] Symbol Description

[0664] 100, 400 Three-dimensional data encoding apparatus

[0665] 101, 201, 401, 501 Acquisition unit

[0666] 102, 402 Encoding area determination unit

[0667] 103 Division unit

[0668] 104, 644 Encoding unit

[0669] 111, 607 Three-dimensional data

[0670] 112, 211, 413, 414, 511, 634 Encoded three-dimensional data

[0671] 3D data decoding device for 200 and 500

[0672] GOS decision-making unit for decoding start in 202

[0673] SPC decision-making unit for decoding in 203

[0674] Decoding units for 204 and 625

[0675] Decoding 3D data in 212, 512, and 513

[0676] SWLD extraction unit in 403

[0677] WLD encoding unit in 404

[0678] SWLD encoding unit in 405

[0679] Input 3D data in 411

[0680] Extracting 3D data in 412

[0681] Header analysis unit in 502

[0682] WLD decoding unit in 503

[0683] SWLD decoding unit in 504

[0684] Own vehicle in 600

[0685] Surrounding vehicles in 601

[0686] Sensor detection ranges in 602 and 605

[0687] Regions in 603 and 606

[0688] Occlusion region in 604

[0689] 3D data production devices in 620 and 620A

[0690] 3D data production units in 621 and 641

[0691] Request range determination unit in 622

[0692] Search unit in 623

[0693] Receiving units in 624 and 642

[0694] Synthesis unit in 626

[0695] Detection region determination unit in 627

[0696] Surrounding condition detection unit in 628

[0697] 629 Self-regulating motion control unit

[0698] 631, 651 Sensor information

[0699] 632 First 3D data

[0700] 633 Request range information

[0701] 635 Second 3D data

[0702] 636 Third 3D data

[0703] 637 Commission signal

[0704] 638 Transmitted data

[0705] 639 Surrounding condition detection result

[0706] 640, 640A 3D data transmission device

[0707] 643 Extraction unit

[0708] 645 Transmission unit

[0709] 646 Transmission feasibility judgment unit

[0710] 652 Fifth 3D data

[0711] 654 Sixth 3D data

[0712] 700 3D information processing device

[0713] 701 3D map acquisition unit

[0714] 702 Own vehicle detection data acquisition unit

[0715] 703 Abnormal situation judgment unit

[0716] 704 Response operation decision unit

[0717] 705 Operation control unit

[0718] 711 3D map

[0719] 712 Own vehicle detection 3D data

[0720] 801 Vehicle

[0721] 802 Space

[0722] 810 3D data production device

[0723] 811 Data reception unit

[0724] 812, 819 Communication unit

[0725] 813 Reception Control Unit

[0726] 814, 821 Format Conversion Unit

[0727] 815 Sensor

[0728] 816 3D Data Creation Unit

[0729] 817 3D Data Composition Unit

[0730] 818 3D Data Storage Unit

[0731] 820 Transmission Control Unit

[0732] 822 Data Transmission Unit

[0733] 831, 832, 834, 835, 836, 837 3D Data

[0734] 833 Sensor Information

Claims

1. A three-dimensional data generation method, wherein, Including: Receiving first data wirelessly by a device mounted on a first moving body moving along a traveling direction, the first data representing a plurality of three-dimensional positions; Generating second data based on detections made by sensors mounted on the first moving body, the second data representing a plurality of three-dimensional positions in a space in front of the first moving body in the traveling direction; And Wirelessly transmitting third data to at least one of a second moving body following the first moving body in the traveling direction and a traffic monitoring system, the third data representing a plurality of three-dimensional positions generated based on the first data and the second data, The above three-dimensional data generation method further includes: When the plurality of three-dimensional positions represented by the third data change, updating the third data; And Wirelessly retransmitting the updated third data to the second moving body.

2. The three-dimensional data generation method according to claim 1, wherein The received first data is transmitted from a third moving body in front of the first moving body in the traveling direction, The first data represents a plurality of three-dimensional positions around the third moving body.

3. The three-dimensional data generation method according to claim 1, wherein The third data is a part of the combined data of the first data and the second data, The plurality of three-dimensional positions represented by the third data at least include a plurality of three-dimensional positions in an area that is a blind spot from the perspective of the second moving body.

4. The three-dimensional data generation method according to claim 1, wherein A first distance between the first moving body in the traveling direction and the space changes according to the moving speed of the first moving body.

5. A three-dimensional data generation device is mounted on a first moving body that moves along a traveling direction, wherein, Including: A communication circuit configured to wirelessly receive first data representing a plurality of three-dimensional positions; And A processor connected to the communication circuit, generating second data based on detections made by sensors mounted on the first moving body, the second data representing a plurality of three-dimensional positions in a space in front of the first moving body in the traveling direction, The communication circuit is configured to wirelessly transmit third data to at least one of a second moving body following the first moving body in the traveling direction and a traffic monitoring system, the third data representing a plurality of three-dimensional positions generated based on the first data and the second data, The communication circuit is further configured to When the plurality of three-dimensional positions represented by the third data change, update the third data, Wirelessly retransmit the updated third data to the second moving body.

Images