Mobile device and execution method thereof, server and execution method thereof

Abstract

The invention provides a mobile device, a method for executing the same, a server and a method for executing the same. The mobile device includes: a communication device; and a processor that acquires a three-dimensional map including first information indicating a position of an object and second information indicating a process to be performed by the mobile device at a position corresponding to the object, and performs the process at the position corresponding to the object on the basis of the second information.

Description

[0001] This application is a divisional of an invention patent application with an application date of January 31, 2019, application number 201980023497.X, and invention name “Information sending method and client device”. Technical Field

[0002] The present disclosure relates to an information sending method and a client device. Background Art

[0003] In the future, devices and services that make use of 3D data will become more common in large fields such as computer vision, map information, monitoring, infrastructure inspection, or image distribution, which are used for autonomous operation of cars or robots. 3D data is obtained by various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple single-lens reflex cameras.

[0004] As a method of expressing three-dimensional data, there is a method called point cloud data, which expresses the shape of a three-dimensional structure through a point group in a three-dimensional space (for example, refer to non-patent document 1). The position and color of the point group are stored in the point cloud data. Although point cloud data is expected to become the mainstream method of expressing three-dimensional data, the amount of point group data is very large. Therefore, in the accumulation or transmission of three-dimensional data, as with two-dimensional dynamic images (as an example, there are MPEG-4AVC or HEVC standardized by MPEG), it is necessary to compress the data volume through encoding.

[0005] Furthermore, compression of point cloud data is partially supported by a public library (Point Cloud Library) or the like that performs point cloud data association processing.

[0006] Furthermore, there is a known technique for searching for facilities around a vehicle using three-dimensional map data and displaying the facilities (for example, refer to Patent Document 1).

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1 International Publication No. 2014 / 020663 Summary of the invention

[0010] Problems to be solved by the invention

[0011] It is desirable to appropriately transmit such information for creating three-dimensional data.

[0012] An object of the present disclosure is to provide an information transmission method or a client device capable of appropriately transmitting information for creating three-dimensional data.

[0013] Means used to solve problems

[0014] A mobile device involved in one form of the present disclosure comprises: a communication device; and a processor, wherein the processor obtains a three-dimensional map including first information and second information, wherein the first information represents the position of an object, and the second information represents a process to be performed by the mobile device at the position corresponding to the object, and performs the process at the position corresponding to the object based on the second information.

[0015] Effects of the Invention

[0016] The present disclosure can provide an information transmission method or a client device capable of appropriately transmitting information for creating three-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The structure of the encoded three-dimensional data involved in Embodiment 1 is shown.

[0018] Figure 2 An example of a prediction structure between SPCs belonging to the lowest layer of the GOS involved in the first embodiment is shown.

[0019] Figure 3 An example of an inter-layer prediction structure according to the first embodiment is shown.

[0020] Figure 4 An example of the coding order of the GOS according to the first embodiment is shown.

[0021] Figure 5 An example of the coding order of the GOS according to the first embodiment is shown.

[0022] Figure 6 This is a block diagram of the three-dimensional data encoding device involved in embodiment 1.

[0023] Figure 7 This is a flowchart of the encoding process involved in Implementation 1.

[0024] Figure 8 This is a block diagram of the three-dimensional data decoding device involved in Embodiment 1.

[0025] Fig. 9 This is a flowchart of the decoding process involved in Implementation 1.

[0026] Fig.10 An example of meta-information according to the first embodiment is shown.

[0027] Fig.11 A configuration example of a SWLD according to the second embodiment is shown.

[0028] Fig.12 An operation example of the server and the client according to the second embodiment is shown.

[0029] Fig.13 An operation example of the server and the client according to the second embodiment is shown.

[0030] Fig.14 An operation example of the server and the client according to the second embodiment is shown.

[0031] Fig.15 An operation example of the server and the client according to the second embodiment is shown.

[0032] Fig.16 This is a block diagram of a three-dimensional data encoding device according to the second embodiment.

[0033] Fig.17 This is a flowchart of the encoding process involved in Implementation Method 2.

[0034] Fig.18 This is a block diagram of a three-dimensional data decoding device according to the second embodiment.

[0035] Fig.19 This is a flowchart of the decoding process involved in Implementation Method 2.

[0036] Fig. 20 A configuration example of a WLD according to the second embodiment is shown.

[0037] Fig.21 An example of the octree structure of the WLD according to the second embodiment is shown.

[0038] Fig. 22 A configuration example of a SWLD according to the second embodiment is shown.

[0039] Fig.23 An example of the octree structure of the SWLD according to the second embodiment is shown.

[0040] Fig.24 This is a schematic diagram showing a state of transmission and reception of three-dimensional data between vehicles according to the third embodiment.

[0041] Fig.25 An example of three-dimensional data transmitted between vehicles according to the third embodiment is shown.

[0042] Fig.26 This is a block diagram of a three-dimensional data creation device according to the third embodiment.

[0043] Fig. 27 This is a flowchart of the three-dimensional data creation process involved in the third embodiment.

[0044] Fig.28 This is a block diagram of a three-dimensional data transmitting device according to Embodiment 3.

[0045] Fig.29 This is a flowchart of the three-dimensional data transmission process involved in the third embodiment.

[0046] Fig.30 This is a block diagram of a three-dimensional data creation device according to the third embodiment.

[0047] Fig.31 This is a flowchart of the three-dimensional data creation process involved in the third embodiment.

[0048] Fig.32 This is a block diagram of a three-dimensional data transmitting device according to Embodiment 3.

[0049] Fig.33 This is a flowchart of the three-dimensional data transmission process involved in the third embodiment.

[0050] Fig.34 This is a block diagram of a three-dimensional information processing device according to a fourth embodiment.

[0051] Fig.35 This is a flowchart of the three-dimensional information processing method involved in the fourth embodiment.

[0052] Fig.36 This is a flowchart of the three-dimensional information processing method involved in the fourth embodiment.

[0053] Fig.37 This is a diagram for explaining the three-dimensional data transmission process involved in the fifth embodiment.

[0054] Fig.38 This is a block diagram of a three-dimensional data production device according to the fifth embodiment.

[0055] Fig.39 This is a flowchart of the three-dimensional data production method involved in implementation mode 5.

[0056] Fig.40 This is a flowchart of the three-dimensional data production method involved in implementation mode 5.

[0057] Fig.41 This is a flowchart of the display method involved in embodiment 6.

[0058] Fig.42 An example of the surrounding environment viewed through the windshield according to the sixth embodiment is shown.

[0059] Fig.43 A display example of the head-up display according to the sixth embodiment is shown.

[0060] Fig.44 A display example of the head-up display after adjustment according to the sixth embodiment is shown.

[0061] Fig.45 The configuration of the system according to the seventh embodiment is shown.

[0062] Fig.46 This is a block diagram of a client device according to the seventh embodiment.

[0063] Fig.47 This is a block diagram of the server involved in Implementation Example 7.

[0064] Fig.48 This is a flowchart of the three-dimensional data creation process performed by the client device involved in the seventh embodiment.

[0065] Fig.49 This is a flowchart of the sensor information transmission process performed by the client device according to the seventh embodiment.

[0066] Fig.50 This is a flowchart of the three-dimensional data creation process performed by the server involved in Implementation Example 7.

[0067] Fig.51 This is a flowchart of the three-dimensional map transmission processing performed by the server involved in Implementation Example 7.

[0068] Fig.52 The configuration of a modified example of the system according to the seventh embodiment is shown.

[0069] Fig.53 The configuration of the server and client device according to the seventh embodiment is shown.

[0070] Fig.54 This is a diagram showing the configuration of a server and a client device according to the eighth embodiment.

[0071] Fig.55 This is a flowchart of the processing performed by the client device involved in the eighth embodiment.

[0072] Fig.56 This is a diagram showing the configuration of a sensor information collection system according to the eighth embodiment. DETAILED DESCRIPTION

[0073] An information sending method involved in one form of the present disclosure is an information sending method for a client device mounted on a mobile body, which obtains sensor information representing the surrounding conditions of the mobile body obtained by a sensor mounted on the mobile body, and stores the sensor information in a storage unit; determines whether the mobile body exists in an environment where the sensor information can be sent to a server; and sends the sensor information to the server when it is determined that the mobile body exists in an environment where the sensor information can be sent to the server.

[0074] According to this, the information transmission method can appropriately transmit information for creating three-dimensional data.

[0075] For example, in the information transmission method, three-dimensional data of the surroundings of the moving object may be further generated based on the sensor information, and the position of the moving object itself may be estimated using the generated three-dimensional data.

[0076] For example, the information sending method may further include sending a request for sending a three-dimensional map to the server, receiving the three-dimensional map from the server, and estimating the own position by using the three-dimensional data and the three-dimensional map.

[0077] Accordingly, the information sending method can improve the accuracy of its own position estimation.

[0078] For example, the sensor information may include at least one of information obtained by a laser sensor, a brightness image, an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0079] For example, the sensor information may include acquisition location information indicating the position of the moving object or the sensor when the sensor information is acquired by the sensor.

[0080] For example, the sensor information may include acquisition time information indicating a time when the sensor information is acquired by the sensor.

[0081] For example, the information sending method may further include obtaining time information from the server; and generating the obtained time information using the obtained time information.

[0082] This makes it possible to synchronize the acquisition time information of the sensor information transmitted from a plurality of client devices.

[0083] For example, it may also be that in the information sending method, a sensor information sending request containing designated information is further received from the server, and the designated information specifies a place and a time; in the storage unit, sensor information obtained at the place and time represented by the designated information is stored, and when it is determined that the mobile body exists in an environment where the sensor information can be sent to the server, the sensor information obtained at the place and time represented by the designated information is sent to the server.

[0084] For example, in the information transmission method, the sensor information that has been transmitted to the server may be further deleted from the storage unit.

[0085] This can reduce the capacity of the storage unit.

[0086] For example, in the information sending method, further, when the difference between the position of the moving body or the sensor when the sensor information is obtained by the sensor and the current position of the moving body or the sensor exceeds a predetermined distance, the sensor information is deleted from the storage unit.

[0087] This can reduce the capacity of the storage unit.

[0088] For example, in the information transmission method, when the difference between the time when the sensor information is obtained by the sensor and the current time exceeds a preset time, the sensor information may be deleted from the storage unit.

[0089] This can reduce the capacity of the storage unit.

[0090] In addition, a client device involved in one form of the present disclosure is a client device mounted on a mobile body, comprising a processor and a memory; the processor uses the memory to perform the following processing: obtaining sensor information representing the surrounding conditions of the mobile body obtained by a sensor mounted on the mobile body, and storing the sensor information in a storage unit; determining whether the mobile body exists in an environment where the sensor information can be sent to a server; and sending the sensor information to the server when it is determined that the mobile body exists in an environment where the sensor information can be sent to the server.

[0091] This allows the client device to appropriately transmit information for creating three-dimensional data.

[0092] In addition, these general or specific forms can be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, and can be implemented by any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0093] The following detailed description of the implementation mode is given with reference to the accompanying drawings. In addition, the implementation modes to be described below are all specific examples of the present disclosure. The numerical values, shapes, materials, constituent elements, configuration positions of constituent elements, connection forms, steps, order of steps, etc. shown in the following implementation modes are all examples, and the main purpose is not to limit the present disclosure. Furthermore, the constituent elements of the following implementation modes that are not recorded in the technical solution showing the highest concept are described as arbitrary constituent elements.

[0094] (Implementation Method 1)

[0095] First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to the present embodiment will be described. Figure 1 The structure of the encoded three-dimensional data involved in this embodiment is shown.

[0096] In this embodiment, the three-dimensional space is divided into a space (SPC) equivalent to a picture in the encoding of a dynamic image, and the three-dimensional data is encoded in units of space. The space is further divided into volumes (VLM) equivalent to macroblocks in dynamic image encoding, and prediction and conversion are performed in units of VLM. The volume includes a plurality of voxels (VXL), which are the smallest units corresponding to position coordinates. In addition, prediction means that, similar to the prediction performed in a two-dimensional image, predicted three-dimensional data similar to the processing unit of the processing object is generated with reference to other processing units, and the difference between the predicted three-dimensional data and the processing unit of the processing object is encoded. Furthermore, the prediction includes not only spatial prediction with reference to other prediction units at the same time, but also temporal prediction with reference to prediction units at different times.

[0097] For example, when encoding a three-dimensional space represented by point group data such as point cloud data, a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes each point of the point group or a plurality of points contained in a voxel according to the size of the voxel. If the voxel is subdivided, the three-dimensional shape of the point group can be expressed with high accuracy, and if the size of the voxel is increased, the three-dimensional shape of the point group can be roughly expressed.

[0098] In addition, although the following description takes the case where the three-dimensional data is point cloud data as an example, the three-dimensional data is not limited to point cloud data, and may also be three-dimensional data in any form.

[0099] Furthermore, voxels of a hierarchical structure may be used. In this case, in the n-order hierarchy, whether or not a sampling point exists in the n-1-order or lower hierarchy (the lower hierarchy of the n-order hierarchy) may be sequentially indicated. For example, when decoding only the n-order hierarchy, if a sampling point exists in the n-1-order or lower hierarchy, decoding may be performed by assuming that a sampling point exists at the center of a voxel in the n-order hierarchy.

[0100] Furthermore, the encoding device obtains point group data through a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor.

[0101] As with the encoding of moving images, the space is classified into at least one of the following three prediction structures: an intra-frame space (I-SPC) that can be decoded independently, a prediction space (P-SPC) that can only be referenced unidirectionally, and a bidirectional space (B-SPC) that can be referenced bidirectionally. In addition, the space has two types of time information: decoding time and display time.

[0102] And, if Figure 1 As shown, as a processing unit including a plurality of spaces, there is a GOS (Group Of Space) which is a random access unit. Also, as a processing unit including a plurality of GOS, there is a world space (WLD).

[0103] The spatial area occupied by the world space is associated with an absolute position on the earth through GPS or latitude and longitude information. This position information is stored as meta information. In addition, the meta information can be included in the encoded data or transmitted separately from the encoded data.

[0104] Furthermore, within the GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0105] In addition, the encoding, decoding or referencing processing corresponding to the three-dimensional data included in the processing unit such as GOS, SPC or VLM is also simply referred to as encoding, decoding or referencing the processing unit. Furthermore, the three-dimensional data included in the processing unit includes at least one set of spatial positions such as three-dimensional coordinates and characteristic values ​​such as color information.

[0106] Next, the prediction structure of the SPC in the GOS will be described. Although multiple SPCs in the same GOS or multiple VLMs in the same SPC occupy different spaces, they have the same time information (decoding time and display time).

[0107] Furthermore, in the GOS, the first SPC in the decoding order is the I-SPC. Furthermore, there are two types of GOS, the closed GOS and the open GOS. The closed GOS is a GOS that can decode all the SPCs in the GOS when decoding starts from the first I-SPC. In the open GOS, in the GOS, some SPCs earlier than the display time of the first I-SPC refer to different GOSs and can only be decoded in the GOS.

[0108] In addition, in the case of coded data such as map information, the WLD may be decoded in the reverse direction of the coding order. If there is a dependency between GOS, it is difficult to reproduce the data in the reverse direction. Therefore, in this case, a closed GOS is basically used.

[0109] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the bottom layer.

[0110] Figure 2 An example of a prediction structure between SPCs belonging to the lowest layer of the GOS is shown. Figure 3 An example of an inter-layer prediction structure is shown.

[0111] There are more than one I-SPC in the GOS. Although there are objects such as people, animals, cars, bicycles, traffic lights, or buildings that serve as land landmarks in the three-dimensional space, it is particularly effective to encode small-sized objects as I-SPCs. For example, when a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes the GOS at a low processing amount or high speed, it only decodes the I-SPC in the GOS.

[0112] Furthermore, the encoding device may switch the encoding interval or the occurrence frequency of the I-SPC according to the density of the objects in the WLD.

[0113] And, in Figure 3 In the configuration shown, the encoding device or decoding device encodes or decodes the plurality of layers sequentially from the lower layer (layer 1). This allows, for example, autonomous vehicles to prioritize data near the ground with a large amount of information.

[0114] In addition, in the coded data used by a drone or the like, coding or decoding may be performed sequentially starting from the SPC of the upper layer in the height direction within the GOS.

[0115] Furthermore, the encoding device or decoding device may encode or decode multiple layers in such a way that the decoding device roughly grasps the GOS and can gradually increase the resolution. For example, the encoding device or decoding device may encode or decode in the order of layers 3, 8, 1, 9, ...

[0116] Next, the corresponding method of the static object and the dynamic object is described.

[0117] In three-dimensional space, there are static objects or scenes such as buildings and roads (hereinafter collectively referred to as static objects), and dynamic objects such as vehicles and people (hereinafter referred to as dynamic objects). Object detection can be performed by extracting feature points from point cloud data or images captured by stereo cameras. Here, an example of a method for encoding dynamic objects is described.

[0118] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects by using identification information.

[0119] For example, GOS is used as the identification unit. In this case, GOS including SPCs constituting static objects and GOS including SPCs constituting dynamic objects are distinguished within the coded data or by identification information stored separately from the coded data.

[0120] Alternatively, SPC is used as the identification unit. In this case, the SPC including only the VLM constituting the static object and the SPC including the VLM constituting the dynamic object are distinguished by the above-mentioned identification information.

[0121] Alternatively, VLM or VXL may be used as the identification unit. In this case, the VLM or VXL including the static object and the VLM or VXL including the dynamic object are distinguished by the above-mentioned identification information.

[0122] Furthermore, the encoding device may encode the dynamic object as one or more VLMs or SPCs, and encode the VLM or SPC including the static object and the SPC including the dynamic object as different GOSs. Furthermore, when the size of the GOS becomes variable according to the size of the dynamic object, the encoding device may store the size of the GOS separately as meta-information.

[0123] Furthermore, the encoding device encodes the static object and the dynamic object independently of each other, and the dynamic object can be overlapped with respect to the world space composed of the static object. In this case, the dynamic object is composed of one or more SPCs, and each SPC corresponds to one or more SPCs constituting the static object overlapped with the SPC. In addition, the dynamic object may not be represented by an SPC, but may be represented by one or more VLMs or VXLs.

[0124] Furthermore, the encoding device may encode static objects and dynamic objects as different streams.

[0125] Furthermore, the encoding device may generate a GOS including one or more SPCs constituting a dynamic object. Furthermore, the encoding device may set the GOS (GOS_M) including the dynamic object and the GOS of the static object corresponding to the spatial region of the GOS_M to be of the same size (occupying the same spatial region). In this way, overlapping processing can be performed in units of GOS.

[0126] The P-SPC or B-SPC constituting the dynamic object may also refer to the SPC included in the encoded different GOS. When the position of the dynamic object changes over time and the same dynamic object is encoded as the GOS at different times, cross-GOS reference is effective from the perspective of compression rate.

[0127] Furthermore, the first method and the second method may be switched according to the purpose of the encoded data. For example, when the encoded three-dimensional data is used as a map, it is desired to separate the three-dimensional data from the dynamic objects, so the encoding device uses the second method. In addition, when the encoding device encodes the three-dimensional data of an event such as a concert or sports, if it is not necessary to separate the dynamic objects, the first method is used.

[0128] Furthermore, the decoding time and display time of GOS or SPC can be stored in the encoded data or as meta-information. Furthermore, the time information of static objects can all be the same. At this time, the actual decoding time and display time can be determined by the decoding device. Alternatively, different values ​​can be assigned to each GOS or SPC as the decoding time, and the same value can be assigned to all the display times. Moreover, as shown in the decoder mode in dynamic image encoding such as HEVC's HRD (Hypothetical Reference Decoder), the decoder has a buffer of a specified size. As long as the bit stream is read at a specified bit rate according to the decoding time, a model that will not be destroyed and is guaranteed to be decodable can be imported.

[0129] Next, the configuration of the GOS in the world space is described. The coordinates of the three-dimensional space in the world space are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, and z-axis). By setting a prescribed rule in the encoding order of the GOS, spatially adjacent GOS can be encoded continuously in the encoded data. For example, Figure 4 In the example shown, the GOS in the xz plane is continuously encoded. After the encoding of all GOS in an xz plane is completed, the value of the y axis is updated. That is, as the encoding continues, the world space extends in the y axis direction. And the index number of the GOS is set as the encoding order.

[0130] Here, the three-dimensional space of the world space corresponds one-to-one to absolute geographical coordinates such as GPS, latitude and longitude. Alternatively, the three-dimensional space can be represented by a relative position relative to a preset reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, etc., and the direction vectors are stored together with the encoded data as meta information.

[0131] Furthermore, the size of the GOS is set to be fixed, and the encoding device stores the size as meta-information. Furthermore, the size of the GOS can be switched, for example, depending on whether it is in the city, indoors, or outdoors. That is, the size of the GOS can be switched according to the amount or nature of objects that have value as information. Alternatively, the encoding device can appropriately switch the size of the GOS or the interval of the I-SPC in the GOS according to the density of the object, etc. in the same world space. For example, the encoding device sets the size of the GOS to be smaller and the interval of the I-SPC in the GOS to be shorter when the density of the object is higher.

[0132] exist Figure 5 In the example, in the area from the 3rd to the 10th GOS, since the density of objects is high, the GOS is subdivided to achieve random access of fine granularity. And, the 7th to the 10th GOS exist on the back of the 3rd to the 6th GOS, respectively.

[0133] Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be described. Figure 6 It is a block diagram of the three-dimensional data encoding device 100 involved in this embodiment. Figure 7 : is a flowchart showing an operation example of the three-dimensional data encoding device 100 .

[0134] Figure 6 The three-dimensional data encoding device 100 shown generates encoded three-dimensional data 112 by encoding three-dimensional data 111. The three-dimensional data encoding device 100 includes an acquisition unit 101, an encoding region determination unit 102, a division unit 103, and an encoding unit 104.

[0135] like Figure 7 As shown, first, the acquisition unit 101 acquires three-dimensional data 111 as point group data ( S101 ).

[0136] Next, the coding region determination unit 102 determines a coding target region from the spatial region corresponding to the obtained point cloud data (S102). For example, the coding region determination unit 102 determines a spatial region around the position as the coding target region according to the position of the user or vehicle.

[0137] Next, the division unit 103 divides the point group data contained in the area of ​​the encoding object into each processing unit. Here, the processing unit is the above-mentioned GOS and SPC, etc. And, the area of ​​the encoding object corresponds to the above-mentioned world space, for example. Specifically, the division unit 103 divides the point group data into processing units according to the size of the pre-set GOS, the presence or size of the dynamic object (S103). And, the division unit 103 determines the starting position of the SPC that becomes the beginning in the encoding order in each GOS.

[0138] Next, the encoding unit 104 generates the encoded three-dimensional data 112 by sequentially encoding a plurality of SPCs in each GOS ( S104 ).

[0139] In addition, here, after the area to be coded is divided into GOS and SPC, although an example of coding each GOS is shown, the order of processing is not limited to the above. For example, after the composition of a GOS is determined, the GOS may be coded, and then the composition of the GOS may be determined.

[0140] In this way, the three-dimensional data encoding device 100 generates the encoded three-dimensional data 112 by encoding the three-dimensional data 111. Specifically, the three-dimensional data encoding device 100 divides the three-dimensional data into random access units, that is, into first processing units (GOS) corresponding to the three-dimensional coordinates, divides the first processing unit (GOS) into a plurality of second processing units (SPC), and divides the second processing unit (SPC) into a plurality of third processing units (VLM). In addition, the third processing unit (VLM) includes more than one voxel (VXL), and the voxel (VXL) is the smallest unit corresponding to the position information.

[0141] Next, the three-dimensional data encoding device 100 generates the encoded three-dimensional data 112 by encoding each of the plurality of first processing units (GOS). Specifically, the three-dimensional data encoding device 100 encodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the three-dimensional data encoding device 100 encodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0142] For example, when the first processing unit (GOS) of the processing object is a closed GOS, the three-dimensional data encoding device 100 performs encoding with reference to other second processing units (SPC) included in the first processing unit (GOS) of the processing object for the second processing unit (SPC) of the processing object included in the first processing unit (GOS) of the processing object. That is, the three-dimensional data encoding device 100 does not refer to the second processing unit (SPC) included in the first processing unit (GOS) different from the first processing unit (GOS) of the processing object.

[0143] Furthermore, when the first processing unit (GOS) of the processing object is an open GOS, the second processing unit (SPC) of the processing object contained in the first processing unit (GOS) of the processing object is encoded with reference to other second processing units (SPC) contained in the first processing unit (GOS) of the processing object, or a second processing unit (SPC) contained in a first processing unit (GOS) different from the first processing unit (GOS) of the processing object.

[0144] Furthermore, the three-dimensional data encoding device 100 selects one as the type of the second processing unit (SPC) of the processing object from among a first type (I-SPC) that does not refer to other second processing units (SPC), a second type (P-SPC) that refers to one other second processing unit (SPC), and a third type that refers to two other second processing units (SPC), and encodes the second processing unit (SPC) of the processing object according to the selected type.

[0145] Next, the configuration and operation flow of the three-dimensional data decoding device according to the present embodiment will be described. Figure 8 It is a block diagram of the three-dimensional data decoding device 200 according to this embodiment. Fig. 9 : is a flowchart showing an example of the operation of the three-dimensional data decoding device 200 .

[0146] Figure 8 The three-dimensional data decoding device 200 shown generates decoded three-dimensional data 212 by decoding the encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. The three-dimensional data decoding device 200 includes an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0147] First, the acquisition unit 201 acquires the encoded three-dimensional data 211 (S201). Next, the decoding start GOS determination unit 202 determines the GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to the meta information stored in the encoded three-dimensional data 211 or separately from the encoded three-dimensional data, and determines the GOS including the spatial position, object, or SPC corresponding to the time at which decoding is started as the GOS to be decoded.

[0148] Next, the decoded SPC determination unit 203 determines the type (I, P, B) of the SPC to be decoded in the GOS (S203). For example, the decoded SPC determination unit 203 determines (1) whether to decode only the I-SPC, (2) whether to decode the I-SPC and the P-SPC, or (3) whether to decode all types. In addition, when the type of the SPC to be decoded is predetermined, such as when all SPCs are decoded, this step may not be performed.

[0149] Next, the decoding unit 204 obtains the SPC that is the first in the decoding order (the same as the encoding order) in the GOS, and the address position that starts in the encoded three-dimensional data 211, obtains the encoded data of the first SPC from the address position, and decodes each SPC in sequence from the first SPC (S204). And, the above-mentioned address position is stored in meta information, etc.

[0150] In this way, the three-dimensional data decoding device 200 decodes the decoded three-dimensional data 212. Specifically, the three-dimensional data decoding device 200 generates the decoded three-dimensional data 212 of the first processing unit (GOS) as a random access unit by decoding each of the encoded three-dimensional data 211 of the first processing unit (GOS) corresponding to the three-dimensional coordinates. More specifically, the three-dimensional data decoding device 200 decodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). And, the three-dimensional data decoding device 200 decodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0151] The meta information for random access is described below. This meta information is generated by the three-dimensional data encoding device 100 and included in the encoded three-dimensional data 112 (211).

[0152] In conventional random access of two-dimensional moving images, decoding is started from the head frame of a random access unit near a designated time. However, in the world space, random access to (coordinates or objects, etc.) is also envisioned in addition to time.

[0153] Therefore, in order to realize random access to at least the three elements of coordinates, objects, and time, a table is prepared that associates each element with the index number of the GOS. Furthermore, the index number of the GOS is associated with the address of the I-SPC that is the beginning of the GOS. Fig.10 An example of a table included in the meta information is shown. Fig.10 Of all the tables shown, at least one table may be used.

[0154] As an example, random access starting from coordinates is described below. When accessing the coordinates (x2, y2, z2), the coordinate-GOS table is first referenced, and it can be known that the location with coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is referenced, and since it can be known that the address of the first I-SPC in the second GOS is addr(2), the decoding unit 204 obtains data from this address and starts decoding.

[0155] In addition, the address may be an address in a logical format or a physical address of a HDD or a memory. Furthermore, information for identifying a file segment may be used instead of an address. For example, a file segment is a unit after segmenting one or more GOSs.

[0156] Furthermore, when the object spans across multiple GOSs, the GOSs to which the multiple objects belong may also be shown in the object GOS table. If the multiple GOSs are closed GOSs, the encoding device and the decoding device can perform encoding or decoding in parallel. In addition, if the multiple GOSs are open GOSs, the compression efficiency can be further improved by having the multiple GOSs refer to each other.

[0157] Examples of objects include people, animals, cars, bicycles, traffic lights, or buildings that serve as landmarks on land, etc. For example, when encoding in world space, the three-dimensional data encoding device 100 extracts feature points unique to the object from three-dimensional point cloud data, etc., detects the object based on the feature points, and can set the detected object as a random access point.

[0158] In this way, the three-dimensional data encoding device 100 generates first information, which shows a plurality of first processing units (GOS) and a three-dimensional coordinate corresponding to each of the plurality of first processing units (GOS). In addition, the encoded three-dimensional data 112 (211) includes the first information. In addition, the first information further shows at least one of an object, a time, and a data storage destination corresponding to each of the plurality of first processing units (GOS).

[0159] The three-dimensional data decoding device 200 obtains first information from the encoded three-dimensional data 211 , uses the first information to determine the first processing unit of the encoded three-dimensional data 211 corresponding to the specified three-dimensional coordinates, object or time, and decodes the encoded three-dimensional data 211 .

[0160] Other examples of meta information are described below. In addition to the meta information for random access, the three-dimensional data encoding device 100 can also generate and store the following meta information. Furthermore, the three-dimensional data decoding device 200 can also use this meta information during decoding.

[0161] In the case where three-dimensional data is used as map information, a profile is specified according to the purpose, and information indicating the profile may be included in the meta-information. For example, a profile for urban areas or suburbs is specified, or a profile for flying objects is specified, and the maximum or minimum size of the world space, SPC or VLM is defined respectively. For example, in the profile for urban areas, more detailed information is required than in the suburbs, so the minimum size of VLM is set smaller.

[0162] The meta information may also include a tag value indicating the type of the object. The tag value corresponds to the VLM, SPC, or GOS that constitutes the object. The tag value may be set according to the type of the object, for example, a tag value of "0" indicates a "person", a tag value of "1" indicates a "car", and a tag value of "2" indicates a "traffic light". Alternatively, when the type of the object is difficult to determine or does not need to be determined, a tag value indicating the size, or whether it is a dynamic object or a static object, etc. may be used.

[0163] Furthermore, the meta-information may include information indicating the range of the spatial region occupied by the world space.

[0164] Furthermore, the meta-information may be used as header information shared by the entire stream of coded data or a plurality of SPCs such as an SPC in a GOS, and may store the size of the SPC or VXL.

[0165] Furthermore, the meta-information may include identification information of a distance sensor or a camera used to generate the point cloud data, or information indicating the positional accuracy of a point group in the point cloud data.

[0166] Also, the meta information may include information showing whether the world space is composed of only static objects or contains dynamic objects.

[0167] Modifications of this embodiment will be described below.

[0168] The encoding device or decoding device may encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel can be determined based on meta information indicating the spatial position of the GOSs.

[0169] In the case where three-dimensional data is used as a spatial map for the movement of a vehicle or a flying object, or such a spatial map is generated, the encoding device or decoding device can encode or decode the GOS or SPC contained in the space determined based on GPS, path information, or a zoom factor, etc.

[0170] Furthermore, the decoding device may also start decoding from the space close to its own position or walking path. The encoding device or decoding device may also make the priority of the space far from its own position or walking path lower than the priority of the space close to it, and perform encoding or decoding. Here, lowering the priority means lowering the processing order, lowering the resolution (screening post-processing), or lowering the image quality (improving the coding efficiency. For example, increasing the quantization step size), etc.

[0171] Furthermore, when decoding the coded data which is hierarchically coded in space, the decoding device may decode only the lower layer.

[0172] Furthermore, the decoding device may start decoding from a lower layer according to the zoom ratio or purpose of the map.

[0173] Furthermore, in applications such as estimating the self-position of a car or robot during automatic driving or identifying an object, the encoding device or decoding device may also reduce the resolution of an area outside the area within a specified height from the road surface (the area to be identified) for encoding or decoding.

[0174] Furthermore, the encoding device may also independently encode the point cloud data representing the indoor and outdoor spatial shapes, for example, by separating the GOS representing the indoor space (indoor GOS) from the GOS representing the outdoor space (outdoor GOS), so that when the decoding device uses the encoded data, it can select the GOS to be decoded according to the viewpoint position.

[0175] Furthermore, the encoding device may encode the indoor GOS and the outdoor GOS whose coordinates are close to each other by placing them adjacent to each other in the coded stream. For example, the encoding device may correspond the identifiers of the two and store information indicating that the corresponding identifiers are established in the coded stream or in meta-information stored separately. Accordingly, the decoding device can refer to the information in the meta-information to identify the indoor GOS and the outdoor GOS whose coordinates are close to each other.

[0176] Furthermore, the encoding device may switch the size of the GOS or SPC between the indoor GOS and the outdoor GOS. For example, the encoding device sets the size of the GOS to be smaller indoors than outdoors. Furthermore, the encoding device may change the accuracy of extracting feature points from point cloud data or the accuracy of object detection between the indoor GOS and the outdoor GOS.

[0177] Furthermore, the encoding device may add information for the decoding device to distinguish and display dynamic objects from static objects to the encoded data. Accordingly, the decoding device can represent the dynamic objects by combining them with red frames or explanatory texts. In addition, the decoding device may replace the dynamic objects and represent them with only red frames or explanatory texts. Furthermore, the decoding device may represent more detailed object categories. For example, a car may be represented by a red frame, and a person may be represented by a yellow frame.

[0178] Furthermore, the encoding device or decoding device may determine whether to perform encoding or decoding by treating the dynamic object and the static object as different SPCs or GOSs according to the frequency of occurrence of the dynamic object or the ratio of the static object to the dynamic object. For example, when the frequency of occurrence or the ratio of the dynamic object exceeds a threshold, the SPC or GOS in which the dynamic object and the static object are mixed is allowed, and when the frequency of occurrence or the ratio of the dynamic object does not exceed the threshold, the SPC or GOS in which the dynamic object and the static object are mixed is not allowed.

[0179] When the dynamic object is detected from the two-dimensional image information of the camera instead of the point cloud data, the encoding device can obtain the information (frame or text, etc.) for identifying the detection result and the object position separately, and encode the information as part of the three-dimensional encoded data. In this case, the decoding device overlaps the auxiliary information (frame or text) representing the dynamic object with the decoding result of the static object.

[0180] Furthermore, the coding device may change the density of VXL or VLM according to the complexity of the shape of the static object, etc. For example, the coding device sets VXL or VLM to be denser when the shape of the static object is more complex. Furthermore, the coding device may determine the quantization step size when quantizing the spatial position or color information according to the density of VXL or VLM. For example, the coding device sets the quantization step size to be smaller when the VXL or VLM is denser.

[0181] As described above, the encoding device or decoding device according to the present embodiment performs spatial encoding or decoding in a spatial unit having coordinate information.

[0182] Furthermore, the encoding device and the decoding device perform encoding or decoding in volume units in space. The volume includes voxels which are the minimum units corresponding to the position information.

[0183] Furthermore, the encoding device and the decoding device establish a table of correspondence between each element of spatial information including coordinates, objects, and time and GOP, or a table of correspondence between each element, so as to establish correspondence between arbitrary elements to perform encoding or decoding. Furthermore, the decoding device determines the coordinates using the value of the selected element, determines the volume, voxel, or space based on the coordinates, and decodes the space including the volume or voxel, or the determined space.

[0184] Then, the encoding device determines a volume, voxel, or space that can be selected by an element by extracting feature points or recognizing an object, and encodes it as a volume, voxel, or space that can be randomly accessed.

[0185] Spaces are divided into three types: I-SPC which can be encoded or decoded as a single space, P-SPC which can be encoded or decoded with reference to any one of the processed spaces, and B-SPC which can be encoded or decoded with reference to any two of the processed spaces.

[0186] One or more volumes correspond to static objects or dynamic objects. Spaces containing static objects and spaces containing dynamic objects are encoded or decoded as different GOSs. That is, SPCs containing static objects and SPCs containing dynamic objects are assigned to different GOSs.

[0187] Dynamic objects are encoded or decoded for each object and correspond to one or more spaces containing only static objects. That is, multiple dynamic objects are encoded separately, and the encoded data of multiple dynamic objects obtained correspond to SPCs containing only static objects.

[0188] The encoding device and the decoding device increase the priority of the I-SPC in the GOS to perform encoding or decoding. For example, the encoding device performs encoding in a manner that reduces the degradation of the I-SPC (after decoding, the original three-dimensional data can be reproduced more faithfully). And, the decoding device, for example, only decodes the I-SPC.

[0189] The encoding device may change the frequency of using the I-SPC according to the density or value (number) of objects in the world space to perform encoding. That is, the encoding device changes the frequency of selecting the I-SPC according to the number or density of objects included in the three-dimensional data. For example, the encoding device increases the frequency of using the I space when the density of objects in the world space is greater.

[0190] Furthermore, the encoding device sets the random access point in units of GOS, and stores information indicating the spatial area corresponding to the GOS in the header information.

[0191] The coding device, for example, adopts a default value as the spatial size of the GOS. In addition, the coding device may also change the size of the GOS according to the value (number) or density of the objects or dynamic objects. For example, the coding device sets the spatial size of the GOS to be smaller when the objects or dynamic objects are denser or more in number.

[0192] Furthermore, the space or volume includes a group of feature points derived from information obtained by sensors such as a depth sensor, a gyroscope, or a camera. The coordinates of the feature points are set to the center position of the voxel. Furthermore, by subdividing the voxels, the position information can be highly accurate.

[0193] The feature point group is derived using a plurality of pictures. The plurality of pictures have at least the following two types of time information: actual time information and the same time information in the plurality of pictures corresponding to the space (for example, encoding time for rate control, etc.).

[0194] Furthermore, encoding or decoding is performed in units of GOS including one or more spaces.

[0195] The encoding device and the decoding device refer to the space in the processed GOS to predict the P space or the B space in the GOS to be processed.

[0196] Alternatively, the encoding device and the decoding device predict the P space or B space in the GOS to be processed by using the processed space in the GOS to be processed without referring to different GOSs.

[0197] Furthermore, the encoding device and the decoding device transmit or receive the encoded stream in units of a world space including one or more GOSs.

[0198] Furthermore, the GOS has a layer structure in at least one direction in the world space, and the encoding device and the decoding device perform encoding or decoding from the lower layer. For example, the GOS that can be randomly accessed belongs to the lowest layer. The GOS belonging to the upper layer only refers to the GOS belonging to the layer below the same layer. That is, the GOS is spatially divided in a predetermined direction, including a plurality of layers each having one or more SPCs. The encoding device and the decoding device perform encoding or decoding for each SPC by referring to the SPC included in the layer that is the same layer as the SPC or the layer below the SPC.

[0199] Furthermore, the encoding device and the decoding device continuously encode or decode the GOS in the world space unit including the plurality of GOS. The encoding device and the decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. That is, the encoded data includes information indicating the encoding order of the plurality of GOS.

[0200] Furthermore, the encoding device and the decoding device encode or decode two or more different spaces or GOS in parallel.

[0201] Furthermore, the encoding device and the decoding device encode or decode the space information (coordinates, size, etc.) of the space or the GOS.

[0202] Furthermore, the encoding device and the decoding device encode or decode the space or GOS included in the specific space determined based on external information related to the own position and / or area size, such as GPS, path information, or magnification.

[0203] The encoding device or decoding device performs encoding or decoding by giving a lower priority to a space far from the own position than to a space close to the own position.

[0204] The encoding device sets a direction in the world space according to the magnification or purpose, and encodes the GOS having a layer structure in the direction. And the decoding device preferentially decodes the GOS having a layer structure in the direction of the world space set according to the magnification or purpose from the lower layer.

[0205] The encoding device changes the feature point extraction, object recognition accuracy, or space area size contained in the indoor and outdoor spaces. However, the encoding device and the decoding device encode or decode the indoor GOS and the outdoor GOS with close coordinates adjacent to each other in the world space, and also encode or decode these identifiers in correspondence.

[0206] (Implementation Method 2)

[0207] When the coded data of point cloud data is used in actual devices or services, it is desirable to transmit and receive the required information according to the purpose in order to reduce the network bandwidth. However, such a function does not exist in the coding structure of three-dimensional data so far, and therefore there is no coding method for it.

[0208] What will be described in this embodiment is a three-dimensional data encoding method and a three-dimensional data encoding device for providing the function of sending and receiving required information according to the purpose in the encoded data of three-dimensional point cloud data, as well as a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data.

[0209] A voxel (VXL) having a certain feature value or more is defined as a feature voxel (FVXL), and a world space (WLD) formed by the FVXL is defined as a sparse world space (SWLD). Fig.11The sparse world space and the configuration example of the world space are shown. SWLD includes: FGOS, which is a GOS composed of FVXL; FSPC, which is an SPC composed of FVXL; and FVLM, which is a VLM composed of FVXL. The data structure and prediction structure of FGOS, FSPC and FVLM can be the same as those of GOS, SPC and VLM.

[0210] The feature quantity refers to a feature quantity that expresses the three-dimensional position information of VXL or the visible light information of the position of VXL, and in particular, a feature quantity that can detect more corners and edges of a three-dimensional object. Specifically, the feature quantity is a three-dimensional feature quantity or a visible light feature quantity described below, but other than this, any feature quantity can be used as long as it expresses the position, brightness, or color information of VXL.

[0211] As the three-dimensional feature, a SHOT feature (Signature of Histograms of Orientations), a PFH feature (Point Feature Histograms), or a PPF feature (Point Pair Feature) is used.

[0212] The SHOT feature is obtained by dividing the area around the VXL, calculating the inner product between the reference point and the normal vector of the divided area, and converting it into a histogram. The SHOT feature has a high dimension and high feature expression.

[0213] The PFH feature is obtained by selecting a plurality of two-point groups near VXL, calculating the normal vector etc. from the two points, and histogramming them. Since the PFH feature is a histogram feature, it is robust to small amounts of interference and has a high feature expression.

[0214] The PPF feature is a feature calculated according to the VXL of two points using a normal vector, etc. Since all VXLs are used in this PPF feature, it is robust to occlusion.

[0215] Furthermore, as feature quantities of visible light, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients) that utilize information such as brightness gradient information of an image can be used.

[0216] SWLD is generated by calculating the above-mentioned feature amount from each VXL of WLD and extracting FVXL. Here, SWLD may be updated every time WLD is updated, or may be updated regularly after a certain period of time has passed regardless of the update timing of WLD.

[0217] SWLD can be generated for each feature quantity. For example, as shown in SWLD1 based on SHOT feature quantity and SWLD2 based on SIFT feature quantity, SWLD can be generated for each feature quantity and used according to the purpose. In addition, the feature quantity of each FVXL calculated can be stored as feature quantity information in each FVXL.

[0218] Next, a method of using the sparse world space (SWLD) will be described. Since the SWLD only includes feature voxels (FVXL), the data size is generally smaller than that of the WLD including all VXLs.

[0219] In applications that use feature quantities to achieve a certain purpose, by using SWLD information instead of WLD, it is possible to reduce the time required to read from the hard disk, and to reduce the bandwidth and transmission time during network transmission. For example, as map information, WLD and SWLD are stored in the server in advance, and the map information to be sent is switched to WLD or SWLD according to the needs of the client, thereby reducing the network bandwidth and transmission time. A specific example is shown below.

[0220] Fig.12 as well as Fig.13 The following shows the use cases of SWLD and WLD. Fig.12 As shown, when the client 1 as a vehicle-mounted device needs map information for the purpose of determining its own position, the client 1 sends a request for obtaining map data for estimating its own position to the server (S301). The server sends SWLD to the client 1 according to the acquisition request (S302). The client 1 uses the received SWLD to determine its own position (S303). At this time, the client 1 obtains the VXL information of the surrounding area of ​​the client 1 through various methods such as a distance sensor such as a rangefinder, a stereo camera, or a combination of multiple monocular cameras, and estimates its own position information based on the obtained VXL information and SWLD. Here, the own position information includes the three-dimensional position information and orientation of the client 1.

[0221] like Fig.13As shown, when the client 2 as a vehicle-mounted device needs map information for the purpose of drawing a three-dimensional map or the like, the client 2 sends a request for obtaining map data for drawing a map to the server (S311). The server sends a WLD to the client 2 in accordance with the acquisition request (S312). The client 2 uses the received WLD to draw a map (S313). At this time, the client 2 uses, for example, an image taken by itself with a visible light camera and the WLD obtained from the server to create a conceptual image, and draws the created image on a screen such as a car navigation system.

[0222] As described above, the server sends SWLD to the client when the feature amount of each VXL is mainly needed, such as estimating the own position, and sends WLD to the client when detailed VXL information is needed, such as drawing a map. This makes it possible to efficiently send and receive map data.

[0223] In addition, the client can determine which one of SWLD and WLD it needs and request the server to send SWLD or WLD. And the server can determine which one of SWLD and WLD should be sent according to the status of the client or the network.

[0224] Next, a method of switching the transmission and reception between the sparse world space (SWLD) and the world space (WLD) will be described.

[0225] The reception of WLD or SWLD can be switched according to the network bandwidth. Fig.14 An example of operation in this case is shown. For example, when a low-speed network that can use the network bandwidth in an LTE (Long Term Evolution) environment is used, when the client accesses the server via the low-speed network (S321), the SWLD as map information is obtained from the server (S322). In addition, when a high-speed network with sufficient network bandwidth is used in a WiFi environment, the client accesses the server via the high-speed network (S323) and obtains the WLD from the server (S324). Accordingly, the client can obtain appropriate map information according to the network bandwidth of the client.

[0226] Specifically, the client receives SWLD via LTE outdoors, and acquires WLD via WiFi when entering a facility or the like indoors. This allows the client to acquire more detailed map information indoors.

[0227] In this way, the client can request WLD or SWLD from the server according to the frequency band of the network it uses. Alternatively, the client can send information showing the frequency band of the network it uses to the server, and the server can send appropriate data (WLD or SWLD) to the client according to the information. Alternatively, the server can determine the network bandwidth of the client and send appropriate data (WLD or SWLD) to the client.

[0228] Furthermore, the reception of WLD or SWLD can be switched according to the moving speed. Fig.15 An example of operation in this case is shown. For example, when the client moves at high speed (S331), the client receives SWLD from the server (S332). In addition, when the client moves at low speed (S333), the client receives WLD from the server (S334). Accordingly, the client can suppress the network bandwidth and obtain map information according to the speed. Specifically, when the client is driving on a highway, by receiving SWLD with a small amount of data, the map information can be updated at a roughly appropriate speed. In addition, when the client is driving on a general road, by receiving WLD, more detailed map information can be obtained.

[0229] In this way, the client can request WLD or SWLD from the server according to its own moving speed. Alternatively, the client can send information showing its own moving speed to the server, and the server can send appropriate data (WLD or SWLD) to the client according to the information. Alternatively, the server can determine the moving speed of the client and send appropriate data (WLD or SWLD) to the client.

[0230] Furthermore, the client may first obtain the SWLD from the server, and then obtain the WLD of the important area. For example, when the client obtains map data, it first obtains the general map information with the SWLD, selects the area where the features such as buildings, signs, or people appear more, and then obtains the WLD of the selected area. In this way, the client can suppress the amount of data received from the server and obtain the detailed information of the required area.

[0231] Furthermore, the server may create SWLDs for each object based on the WLD, and the client may receive them separately according to the purpose. In this way, the network bandwidth can be suppressed. For example, the server pre-identifies a person or a car from the WLD, and creates a SWLD for the person and a SWLD for the car. When the client wants to obtain information about the people around it, it receives the SWLD for the person, and when it wants to obtain information about the car, it receives the SWLD for the car. Furthermore, the type of such SWLD can be distinguished based on information (flag or type, etc.) attached to the header, etc.

[0232] Next, the configuration and operation flow of the three-dimensional data encoding device (for example, a server) according to the present embodiment will be described. Fig.16 It is a block diagram of the three-dimensional data encoding device 400 involved in this embodiment. Fig.17 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device 400.

[0233] Fig.16 The three-dimensional data encoding device 400 shown in the figure generates encoded three-dimensional data 413 and 414 as encoded streams by encoding input three-dimensional data 411. Here, the encoded three-dimensional data 413 is encoded three-dimensional data corresponding to WLD, and the encoded three-dimensional data 414 is encoded three-dimensional data corresponding to SWLD. The three-dimensional data encoding device 400 includes: an acquisition unit 401, a coding area determination unit 402, a SWLD extraction unit 403, a WLD encoding unit 404, and a SWLD encoding unit 405.

[0234] like Fig.17 As shown, first, the acquisition unit 401 acquires input three-dimensional data 411 which is point group data in a three-dimensional space (S401).

[0235] Next, the encoding region determination unit 402 determines the encoding target spatial region based on the spatial region where the point cloud data exists ( S402 ).

[0236] Next, the SWLD extraction unit 403 defines the spatial region of the encoding object as WLD, and calculates the feature value from each VXL included in the WLD. In addition, the SWLD extraction unit 403 extracts the VXL whose feature value is greater than a predetermined threshold value, defines the extracted VXL as FVXL, and adds the FVXL to the SWLD to generate the extracted three-dimensional data 412 (S403). That is, the extracted three-dimensional data 412 whose feature value is greater than the threshold value is extracted from the input three-dimensional data 411.

[0237] Next, the WLD encoding unit 404 generates encoded three-dimensional data 413 corresponding to the WLD by encoding the input three-dimensional data 411 corresponding to the WLD (S404). At this time, the WLD encoding unit 404 adds information for distinguishing that the encoded three-dimensional data 413 is a stream containing the WLD to the header of the encoded three-dimensional data 413.

[0238] Then, the SWLD encoding unit 405 generates encoded three-dimensional data 414 corresponding to the SWLD by encoding the extracted three-dimensional data 412 corresponding to the SWLD (S405). At this time, the SWLD encoding unit 405 adds information for distinguishing that the encoded three-dimensional data 414 is a stream containing the SWLD to the header of the encoded three-dimensional data 414.

[0239] Furthermore, the processing order of the process of generating the encoded three-dimensional data 413 and the process of generating the encoded three-dimensional data 414 may be reversed from the above. Furthermore, part or all of the above-mentioned processes may be executed in parallel.

[0240] Information assigned to the header of the encoded three-dimensional data 413 and 414 is defined as a parameter such as "world_type". When world_type = 0, it means that the stream contains WLD, and when world_type = 1, it means that the stream contains SWLD. In the case of defining more other categories, the assigned value can be increased, such as world_type = 2. In addition, a specific flag can be included in one of the encoded three-dimensional data 413 and 414. For example, the encoded three-dimensional data 414 can be assigned a flag indicating that the stream contains SWLD. In this case, the decoding device can determine whether it is a stream containing WLD or a stream containing SWLD based on the presence or absence of the flag.

[0241] Furthermore, the encoding method used by the WLD encoding unit 404 when encoding the WLD may be different from the encoding method used by the SWLD encoding unit 405 when encoding the SWLD.

[0242] For example, since data is decimated in SWLD, the correlation with surrounding data may be lower than that in WLD. Therefore, in the encoding method for SWLD, inter-frame prediction is prioritized over intra-frame prediction and inter-frame prediction in comparison with the encoding method for WLD.

[0243] Furthermore, the encoding method used for SWLD and the encoding method used for WLD may have different methods of expressing the three-dimensional position. For example, the three-dimensional position of FVXL may be expressed by three-dimensional coordinates in FWLD, and the three-dimensional position may be expressed by an octree described later in WLD, or vice versa.

[0244] Furthermore, the SWLD coding unit 405 performs coding in such a manner that the data size of the coded three-dimensional data 414 of SWLD is smaller than the data size of the coded three-dimensional data 413 of WLD. For example, as described above, the correlation between data in SWLD may be reduced compared to that in WLD. Accordingly, the coding efficiency is reduced, and the data size of the coded three-dimensional data 414 may be larger than the data size of the coded three-dimensional data 413 of WLD. Therefore, when the data size of the obtained coded three-dimensional data 414 is larger than the data size of the coded three-dimensional data 413 of WLD, the SWLD coding unit 405 performs re-coding to generate the coded three-dimensional data 414 with a reduced data size again.

[0245] For example, the SWLD extraction unit 403 generates the extracted three-dimensional data 412 again with the number of extracted feature points reduced, and the SWLD encoding unit 405 encodes the extracted three-dimensional data 412. Alternatively, the degree of quantization in the SWLD encoding unit 405 may be made coarser. For example, in the octree structure described later, the degree of quantization may be made coarser by rounding the data of the lowest layer.

[0246] Furthermore, when the data size of the SWLD encoded three-dimensional data 414 cannot be made smaller than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 may not generate the SWLD encoded three-dimensional data 414. Alternatively, the WLD encoded three-dimensional data 413 may be copied to the SWLD encoded three-dimensional data 414. That is, the WLD encoded three-dimensional data 413 may be used directly as the SWLD encoded three-dimensional data 414.

[0247] Next, the configuration and operation flow of the three-dimensional data decoding device (eg, client) according to the present embodiment will be described. Fig.18 It is a block diagram of a three-dimensional data decoding device 500 according to this embodiment. Fig.19 3D data decoding processing performed by the 3D data decoding apparatus 500 is shown in FIG.

[0248] Fig.18 The three-dimensional data decoding device 500 shown generates decoded three-dimensional data 512 or 513 by decoding the encoded three-dimensional data 511. Here, the encoded three-dimensional data 511 is, for example, the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400.

[0249] The three-dimensional data decoding device 500 includes an acquisition unit 501 , a header analysis unit 502 , a WLD decoding unit 503 , and a SWLD decoding unit 504 .

[0250] like Fig.19 As shown, first, the acquisition unit 501 acquires the encoded three-dimensional data 511 (S501). Then, the header analysis unit 502 analyzes the header of the encoded three-dimensional data 511 to determine whether the encoded three-dimensional data 511 is a stream containing WLD or a stream containing SWLD (S502). For example, the determination is made by referring to the above-mentioned world_type parameter.

[0251] When the coded three-dimensional data 511 is a stream including WLD ("Yes" in S503), the WLD decoding unit 503 decodes the coded three-dimensional data 511 to generate decoded three-dimensional data 512 of WLD (S504). In addition, when the coded three-dimensional data 511 is a stream including SWLD ("No" in S503), the SWLD decoding unit 504 decodes the coded three-dimensional data 511 to generate decoded three-dimensional data 513 of SWLD (S505).

[0252] Also, similar to the encoding device, the decoding method used by the WLD decoding unit 503 when decoding the WLD may be different from the decoding method used by the SWLD decoding unit 504 when decoding the SWLD. For example, in the decoding method for SWLD, priority may be given to the inter-frame prediction of intra-frame prediction and inter-frame prediction over the decoding method for WLD.

[0253] Furthermore, the decoding method for SWLD and the decoding method for WLD may have different representation methods for the three-dimensional position. For example, the three-dimensional position of FVXL may be represented by three-dimensional coordinates in SWLD and by an octree described later in WLD, or vice versa.

[0254] Next, octree expression as a method of expressing three-dimensional positions will be described. VXL data included in three-dimensional data is converted into an octree structure and encoded. Fig. 20 An example of a VXL of a WLD is shown. Fig.21 Shows Fig. 20 The octree structure of WLD is shown in Figure 1. Fig. 20 In the example shown, there are three VXLs 1 to 3 that are VXLs (hereinafter, effective VXLs) that include a point group. Fig.21 As shown, the octree structure consists of nodes and leaves. Each node has a maximum of 8 nodes or leaves. Each leaf has VXL information. Here, Fig.21 Among the leaves shown, leaves 1, 2, and 3 represent Fig. 20 VXL1, VXL2, VXL3 shown.

[0255] Specifically, each node and leaf corresponds to a three-dimensional position. Fig. 20 The block corresponding to node 1 is divided into 8 blocks, and among the 8 blocks, the block including the valid VXL is set as a node, and the other blocks are set as leaves. The block corresponding to the node is further divided into 8 nodes or leaves, and this process is repeated the same number of times as the number of levels in the tree structure. And all the blocks at the bottom are set as leaves.

[0256] and, Fig. 22Shows from Fig. 20 An example of SWLD generated by WLD is shown. Fig. 20 The feature extraction results of VXL1 and VXL2 shown are determined as FVXL1 and FVXL2 and are included in SWLD. VXL3 is not determined as FVXL and is therefore not included in SWLD. Fig.23 Shows Fig. 22 The octree structure of SWLD is shown in Figure 1. Fig.23 In the octree structure shown, Fig.21 The leaf 3 shown, which is equivalent to VXL3, is deleted. Fig.21 The node 3 shown has no valid VXL and is changed to a leaf. In this way, generally speaking, the number of leaves of SWLD is smaller than the number of leaves of WLD, and the encoded three-dimensional data of SWLD is also smaller than the encoded three-dimensional data of WLD.

[0257] Modifications of this embodiment will be described below.

[0258] For example, when a client such as a vehicle-mounted device estimates its own position, it receives SWLD from a server, uses SWLD to estimate its own position, and performs obstacle detection. It then uses various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras to perform obstacle detection based on the three-dimensional information of the surrounding area obtained by itself.

[0259] In general, it is difficult to include VXL data of a flat area in SWLD. For this reason, the server maintains a subsampled world space (SubWLD) that is a downsampled version of WLD for detecting stationary obstacles, and can send SWLD and SubWLD to the client. This can suppress network bandwidth while enabling the client to estimate its own position and detect obstacles.

[0260] Furthermore, when the client quickly depicts three-dimensional map data, it is convenient if the map information is in a grid structure. Therefore, the server can generate a grid based on the WLD and store it in advance as a grid world space (MWLD). For example, when the client needs to perform a rough three-dimensional depiction, the MWLD is received, and when a detailed three-dimensional depiction is required, the WLD is received. In this way, the network bandwidth can be suppressed.

[0261] Furthermore, although the server sets the VXL with a feature value above the threshold value as FVXL from each VXL, FVXL can also be calculated by different methods. For example, if the server determines that VXL, VLM, SPC, or GOS constituting a signal or intersection is required for self-position estimation, driving assistance, or automatic driving, it can be included in SWLD as FVXL, FVLM, FSPC, FGOS. Furthermore, the above judgment can be performed manually. In addition, FVXL obtained by the above method can be added to FVXL set based on the feature value. That is, the SWLD extraction unit 403 can further extract data corresponding to an object with predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412.

[0262] Furthermore, labels different from feature quantities may be assigned to situations that require these purposes. The server may separately maintain FVXL required for self-position estimation such as signals or intersections, driving assistance, or autonomous driving as an upper layer of SWLD (e.g., lane world space).

[0263] Furthermore, the server may also attach attributes to the VXL in the WLD in random access units or specified units. Attributes include, for example, information indicating whether the location is required or not required for estimating the location, or information indicating whether traffic information such as signals or intersections is important. Furthermore, attributes may also include the correspondence between lane information (GDF: Geographic Data Files, etc.) and features (intersections or roads, etc.).

[0264] Furthermore, as a method of updating WLD or SWLD, the following method can be adopted.

[0265] Update information showing changes in people, construction, or street trees (trajectory orientation) is uploaded to the server as a point group or metadata. The server updates the WLD based on the upload, and then updates the SWLD using the updated WLD.

[0266] Furthermore, when the client detects a mismatch between the three-dimensional information generated by itself and the three-dimensional information received from the server when estimating its own position, the client can send the three-dimensional information generated by itself to the server together with the update notification. In this case, the server updates the SWLD using the WLD. If the SWLD is not updated, the server determines that the WLD itself is old.

[0267] Furthermore, although information for distinguishing between WLD and SWLD is added as the header information of the coded stream, when there are multiple world spaces such as a grid world space or a lane world space, information for distinguishing them may be added to the header information. Furthermore, when there are multiple SWLDs with different feature quantities, information for distinguishing them may also be added to the header information.

[0268] Furthermore, although SWLD is composed of FVXL, it may also include VXL that is not determined to be FVXL. For example, SWLD may include adjacent VXL used when calculating the feature quantity of FVXL. Accordingly, even if each FVXL of SWLD does not have feature quantity information attached, the client can calculate the feature quantity of FVXL when receiving SWLD. In addition, at this time, SWLD may include information for distinguishing whether each VXL is FVXL or VXL.

[0269] As described above, the three-dimensional data encoding device 400 extracts extracted three-dimensional data 412 (second three-dimensional data) whose feature value is above a threshold value from the input three-dimensional data 411 (first three-dimensional data), and generates encoded three-dimensional data 414 (first encoded three-dimensional data) by encoding the extracted three-dimensional data 412.

[0270] According to this, the three-dimensional data encoding device 400 generates the encoded three-dimensional data 414 obtained by encoding the data whose feature quantity is greater than the threshold value. In this way, the amount of data can be reduced compared to the case where the input three-dimensional data 411 is directly encoded. Therefore, the three-dimensional data encoding device 400 can reduce the amount of data during transmission.

[0271] Furthermore, the three-dimensional data encoding device 400 further encodes the input three-dimensional data 411 to generate encoded three-dimensional data 413 (second encoded three-dimensional data).

[0272] According to this, the three-dimensional data encoding device 400 can selectively transmit the encoded three-dimensional data 413 and the encoded three-dimensional data 414 according to the purpose of use, for example.

[0273] Furthermore, the extracted three-dimensional data 412 is encoded by a first encoding method, and the input three-dimensional data 411 is encoded by a second encoding method different from the first encoding method.

[0274] Accordingly, the three-dimensional data encoding device 400 can adopt appropriate encoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0275] Furthermore, in the first encoding method, compared with the second encoding method, priority is given to the inter-frame prediction between the intra-frame prediction and the inter-frame prediction.

[0276] According to this, the three-dimensional data encoding device 400 can increase the priority of inter-frame prediction for the extracted three-dimensional data 412 where the correlation between adjacent data is likely to be low.

[0277] Furthermore, the first encoding method and the second encoding method have different methods of expressing the three-dimensional position. For example, in the second encoding method, the three-dimensional position is expressed by an octree, while in the first encoding method, the three-dimensional position is expressed by three-dimensional coordinates.

[0278] According to this, the three-dimensional data encoding device 400 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data numbers (number of VXLs or FVXLs).

[0279] Furthermore, at least one of the coded three-dimensional data 413 and 414 includes an identifier indicating whether the coded three-dimensional data is coded three-dimensional data obtained by encoding the input three-dimensional data 411 or coded three-dimensional data obtained by encoding a part of the input three-dimensional data 411. That is, the identifier indicates whether the coded three-dimensional data is the coded three-dimensional data 413 of the WLD or the coded three-dimensional data 414 of the SWLD.

[0280] Based on this, the decoding device can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

[0281] Furthermore, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 in such a manner that the data amount of the encoded three-dimensional data 414 is smaller than the data amount of the encoded three-dimensional data 413 .

[0282] According to this, the three-dimensional data encoding device 400 can make the data amount of the encoded three-dimensional data 414 smaller than the data amount of the encoded three-dimensional data 413 .

[0283] Furthermore, the three-dimensional data encoding device 400 further extracts data corresponding to an object having a predetermined attribute from the input three-dimensional data 411 as extracted three-dimensional data 412. For example, the object having a predetermined attribute is an object required for self-position estimation, driving assistance, or automatic driving, such as a signal or an intersection.

[0284] Accordingly, the three-dimensional data encoding device 400 can generate the encoded three-dimensional data 414 including the data required by the decoding device.

[0285] Furthermore, the three-dimensional data encoding device 400 (server) further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the status of the client.

[0286] Accordingly, the three-dimensional data encoding device 400 can send appropriate data according to the status of the client.

[0287] Furthermore, the status of the client includes the communication status of the client (eg, network bandwidth) or the moving speed of the client.

[0288] Furthermore, the three-dimensional data encoding device 400 further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the request of the client.

[0289] Thereby, the three-dimensional data encoding device 400 can send appropriate data according to the request of the client.

[0290] Furthermore, the three-dimensional data decoding device 500 according to the present embodiment decodes the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400 .

[0291] That is, the three-dimensional data decoding device 500 decodes the encoded three-dimensional data 414 obtained by encoding the extracted three-dimensional data 412 whose feature amount extracted from the input three-dimensional data 411 is greater than the threshold value by the first decoding method. Furthermore, the three-dimensional data decoding device 500 decodes the encoded three-dimensional data 413 obtained by encoding the input three-dimensional data 411 by using a second decoding method different from the first decoding method.

[0292] According to this, the three-dimensional data decoding device 500 can selectively receive the encoded three-dimensional data 414 and the encoded three-dimensional data 413 obtained by encoding data having a feature value greater than a threshold value, for example, according to the purpose of use. According to this, the three-dimensional data decoding device 500 can reduce the amount of data during transmission. Furthermore, the three-dimensional data decoding device 500 can adopt appropriate decoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412, respectively.

[0293] Furthermore, in the first decoding method, compared with the second decoding method, priority is given to the inter prediction between the intra prediction and the inter prediction.

[0294] According to this, the three-dimensional data decoding apparatus 500 can increase the priority of inter-frame prediction for extracting three-dimensional data in which the correlation between adjacent data is likely to be low.

[0295] Furthermore, the first decoding method and the second decoding method use different methods for expressing the three-dimensional position. For example, the second decoding method expresses the three-dimensional position by an octree, while the first decoding method expresses the three-dimensional position by three-dimensional coordinates.

[0296] According to this, the three-dimensional data decoding apparatus 500 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data numbers (number of VXLs or FVXLs).

[0297] Furthermore, at least one of the encoded three-dimensional data 413 and 414 includes an identifier indicating whether the encoded three-dimensional data is encoded three-dimensional data obtained by encoding the input three-dimensional data 411 or encoded three-dimensional data obtained by encoding a part of the input three-dimensional data 411. The three-dimensional data decoding device 500 identifies the encoded three-dimensional data 413 and 414 by referring to the identifier.

[0298] Based on this, the three-dimensional data decoding device 500 can easily determine whether the obtained encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414 .

[0299] Furthermore, the 3D data decoding device 500 notifies the server of the state of the client (the 3D data decoding device 500). The 3D data decoding device 500 receives one of the encoded 3D data 413 and 414 transmitted from the server according to the state of the client.

[0300] Accordingly, the three-dimensional data decoding apparatus 500 can receive appropriate data according to the status of the client.

[0301] Furthermore, the status of the client includes the communication status of the client (eg, network bandwidth) or the moving speed of the client.

[0302] Furthermore, the three-dimensional data decoding apparatus 500 further requests one of the encoded three-dimensional data 413 and 414 from the server, and receives one of the encoded three-dimensional data 413 and 414 transmitted from the server in accordance with the request.

[0303] Thereby, the three-dimensional data decoding device 500 can receive appropriate data according to the application.

[0304] (Implementation method 3)

[0305] In the present embodiment, a method of transmitting and receiving three-dimensional data between vehicles will be described.

[0306] Fig.24 It is a schematic diagram showing the state of transmission and reception of three-dimensional data 607 between the own vehicle 600 and the surrounding vehicles 601 .

[0307] When three-dimensional data is acquired by a sensor (such as a distance sensor such as a rangefinder, a stereo camera, or a combination of multiple single-lens cameras, etc.) mounted on the own vehicle 600, due to the influence of obstacles such as surrounding vehicles 601, there will be an area where three-dimensional data cannot be generated (hereinafter referred to as an occlusion area 604) although it is within the sensor detection range 602 of the own vehicle 600. In addition, although the larger the space for obtaining three-dimensional data, the higher the accuracy of autonomous action, the sensor detection range of the own vehicle 600 alone is limited.

[0308] The sensor detection range 602 of the own vehicle 600 includes an area 603 where three-dimensional data can be obtained and an occlusion area 604. The range in which the own vehicle 600 wants to obtain three-dimensional data includes the sensor detection range 602 of the own vehicle 600 and other areas. In addition, the sensor detection range 605 of the surrounding vehicle 601 includes the occlusion area 604 and an area 606 not included in the sensor detection range 602 of the own vehicle 600.

[0309] The surrounding vehicles 601 transmit information detected by the surrounding vehicles 601 to the own vehicle 600. The own vehicle 600 can obtain the three-dimensional data 607 of the blocked area 604 and the area 606 outside the sensor detection range 602 of the own vehicle 600 by obtaining the information detected by the surrounding vehicles 601 such as the vehicle traveling in front. The own vehicle 600 supplements the three-dimensional data of the blocked area 604 and the area 606 outside the sensor detection range using the information obtained from the surrounding vehicles 601.

[0310] The use of three-dimensional data in the autonomous operation of a vehicle or robot is to estimate the own position, detect surrounding conditions, or both. For example, in estimating the own position, three-dimensional data generated by the own vehicle 600 is used based on sensor information of the own vehicle 600. In detecting surrounding conditions, in addition to the three-dimensional data generated by the own vehicle 600, three-dimensional data obtained from surrounding vehicles 601 is also used.

[0311] The surrounding vehicles 601 for transmitting the three-dimensional data 607 to the own vehicle 600 may be determined according to the state of the own vehicle 600. For example, the surrounding vehicles 601 are the vehicles in front when the own vehicle 600 is traveling straight, the vehicles on the opposite direction when the own vehicle 600 is turning right, and the vehicles behind when the own vehicle 600 is backing up. In addition, the driver of the own vehicle 600 may directly designate the surrounding vehicles 601 for transmitting the three-dimensional data 607 to the own vehicle 600.

[0312] Furthermore, the vehicle 600 may search for surrounding vehicles 601 that have 3D data of an area that the vehicle 600 cannot obtain, which is contained in the space where the 3D data 607 is desired. The area that the vehicle 600 cannot obtain is, for example, an occlusion area 604 or an area 606 outside the sensor detection range 602.

[0313] Furthermore, the vehicle 600 may determine the blocked area 604 based on the sensor information of the vehicle 600. For example, the vehicle 600 determines the blocked area 604 as an area included in the sensor detection range 602 of the vehicle 600 where three-dimensional data cannot be generated.

[0314] The following describes an operation example in which the vehicle ahead is transmitting the three-dimensional data 607 . Fig.25 An example of three-dimensional data transmitted in this case is shown.

[0315] like Fig.25 As shown, the three-dimensional data 607 transmitted from the front vehicle is, for example, a sparse world space of point cloud data (SWLD). That is, the front vehicle creates three-dimensional data (point cloud data) of WLD based on information detected by the sensor of the front vehicle, and creates three-dimensional data (point cloud data) of SWLD by extracting data with feature quantities above a threshold from the three-dimensional data of WLD. Furthermore, the front vehicle transmits the created three-dimensional data of SWLD to the own vehicle 600.

[0316] The own vehicle 600 receives the SWLD and merges the received SWLD into the point cloud data created by the own vehicle 600 .

[0317] The transmitted SWLD has information of absolute coordinates (position of the SWLD in the coordinate system of the three-dimensional map), and the own vehicle 600 can perform a merging process by superimposing the point cloud data generated by the own vehicle 600 based on the absolute coordinates.

[0318] The SWLD transmitted from the surrounding vehicle 601 may be the SWLD of the area 606 outside the sensor detection range 602 of the own vehicle 600 and within the sensor detection range 605 of the surrounding vehicle 601, or may be the SWLD of the shielding area 604 relative to the own vehicle 600, or may be the SWLD of both. Furthermore, the transmitted SWLD may also be the SWLD of the area used by the surrounding vehicle 601 when detecting the surrounding conditions among the above-mentioned SWLDs.

[0319] Furthermore, the surrounding vehicle 601 may change the density of the transmitted point cloud data according to the communicable time based on the speed difference between the own vehicle 600 and the surrounding vehicle 601. For example, when the speed difference is large and the communicable time is short, the surrounding vehicle 601 may reduce the density (data volume) of the point cloud data by extracting three-dimensional points with large feature amounts from the SWLD.

[0320] Furthermore, the detection of the surrounding conditions refers to determining the presence of people, vehicles, road construction equipment, etc., identifying their types, and detecting their positions, moving directions, moving speeds, etc.

[0321] Furthermore, the own vehicle 600 may obtain the braking information of the surrounding vehicles 601 instead of the three-dimensional data 607 generated by the surrounding vehicles 601, or obtain the braking information of the surrounding vehicles 601 in addition to the three-dimensional data 607. Here, the braking information of the surrounding vehicles 601 is, for example, information indicating that the accelerator or brake of the surrounding vehicles 601 is depressed, or the degree of depression.

[0322] In addition, the point cloud data generated by each vehicle is divided into random access units in consideration of low-latency communication between vehicles. In addition, in the map data downloaded from the server, the three-dimensional map and the like are divided into larger random access units than in the case of communication between vehicles.

[0323] Data in areas that are prone to being blocked, such as an area in front of a leading vehicle or an area behind a trailing vehicle, are divided into small random access units as data for low latency.

[0324] When traveling at high speed, the importance of the front side increases, so each vehicle generates SWLD in a narrow viewing angle range in small random access units when traveling at high speed.

[0325] When the SWLD created for transmission by the preceding vehicle includes an area where point cloud data can be obtained by the own vehicle 600, the preceding vehicle can reduce the transmission amount by removing the point cloud data of the area.

[0326] Next, the configuration and operation of the three-dimensional data creating device 620 as the three-dimensional data receiving device according to the present embodiment will be described.

[0327] Fig.26 This is a block diagram of a three-dimensional data production device 620 according to the present embodiment. The three-dimensional data production device 620 is included in the above-mentioned vehicle 600, for example, and produces a denser third three-dimensional data 636 by synthesizing the received second three-dimensional data 635 with the first three-dimensional data 632 produced by the three-dimensional data production device 620.

[0328] The three-dimensional data creation device 620 includes a three-dimensional data creation unit 621 , a request range determination unit 622 , a search unit 623 , a reception unit 624 , a decoding unit 625 , and a synthesis unit 626 . Fig. 27 It is a flowchart showing the operation of the three-dimensional data creating device 620.

[0329] First, the three-dimensional data generating unit 621 generates the first three-dimensional data 632 (S621) using the sensor information 631 detected by the sensor of the own vehicle 600. Next, the request range determining unit 622 determines the request range, which refers to the three-dimensional space range where the data in the generated first three-dimensional data 632 is insufficient (S622).

[0330] Next, the search unit 623 searches for the surrounding vehicles 601 that have three-dimensional data of the requested range, and sends the request range information 633 showing the requested range to the surrounding vehicles 601 determined by the search (S623). Next, the receiving unit 624 receives the encoded three-dimensional data 634 as the encoded stream of the requested range from the surrounding vehicles 601 (S624). In addition, the search unit 623 can indiscriminately issue requests to all vehicles existing in the determined range, and receive the encoded three-dimensional data 634 from the responding party. In addition, the search unit 623 is not limited to vehicles, and can also issue requests to objects such as traffic lights or signs, and receive the encoded three-dimensional data 634 from the object.

[0331] Next, the decoder 625 decodes the received encoded three-dimensional data 634 to obtain the second three-dimensional data 635 (S625). Next, the synthesizer 626 synthesizes the first three-dimensional data 632 and the second three-dimensional data 635 to create a denser third three-dimensional data 636 (S626).

[0332] Next, the configuration and operation of the three-dimensional data transmitting device 640 according to this embodiment will be described. Fig.28 It is a block diagram of the three-dimensional data transmitting device 640.

[0333] The three-dimensional data sending device 640 is included in the above-mentioned surrounding vehicle 601, for example, and processes the fifth three-dimensional data 652 produced by the surrounding vehicle 601 into the sixth three-dimensional data 654 requested by the own vehicle 600, and generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654, and sends the encoded three-dimensional data 634 to the own vehicle 600.

[0334] The three-dimensional data transmitting device 640 includes a three-dimensional data creating unit 641 , a receiving unit 642 , an extracting unit 643 , a coding unit 644 , and a transmitting unit 645 . Fig.29 2 is a flowchart showing the operation of the three-dimensional data transmitting device 640 .

[0335] First, the three-dimensional data creation unit 641 creates fifth three-dimensional data 652 using sensor information 651 detected by sensors provided in the surrounding vehicles 601 (S641). Next, the receiving unit 642 receives the requested range information 633 transmitted from the own vehicle 600 (S642).

[0336] Next, the extraction unit 643 extracts the three-dimensional data of the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652, and processes the fifth three-dimensional data 652 into the sixth three-dimensional data 654 (S643). Next, the encoding unit 644 encodes the sixth three-dimensional data 654, thereby generating the encoded three-dimensional data 634 as an encoded stream (S644). Then, the transmission unit 645 transmits the encoded three-dimensional data 634 to the own vehicle 600 (S645).

[0337] In addition, although an example is described here in which the own vehicle 600 has the three-dimensional data creation device 620 and the surrounding vehicle 601 has the three-dimensional data transmission device 640, each vehicle may also have the functions of the three-dimensional data creation device 620 and the three-dimensional data transmission device 640.

[0338] Hereinafter, the configuration and operation of the three-dimensional data creation device 620 in the case of being a surrounding condition detection device that realizes detection processing of the surrounding condition of the own vehicle 600 will be described. Fig.30 This is a block diagram showing the structure of the three-dimensional data creation device 620A in this case. Fig.30 The three-dimensional data production device 620A shown in FIG. Fig.26 In addition to the configuration of the three-dimensional data creation device 620 shown, a detection area determination unit 627, a surrounding condition detection unit 628, and an autonomous operation control unit 629 are further provided. The three-dimensional data creation device 620A is included in the own vehicle 600.

[0339] Fig.31 This is a flowchart of the peripheral condition detection process of the own vehicle 600 performed by the three-dimensional data creation device 620A.

[0340] First, the three-dimensional data creation unit 621 creates the first three-dimensional data 632 as point cloud data using the sensor information 631 of the detection range of the own vehicle 600 detected by the sensor of the own vehicle 600 (S661). In addition, the three-dimensional data creation device 620A can also use the sensor information 631 to estimate its own position.

[0341] Next, the detection area determination unit 627 determines the detection target range as the spatial area in which the surrounding conditions are to be detected (S662). For example, the detection area determination unit 627 calculates the area required for the surrounding condition detection in order to perform safe autonomous operation according to the autonomous operation (autonomous driving) conditions such as the driving direction and speed of the own vehicle 600, and determines the area as the detection target range.

[0342] Next, the request range determination unit 622 determines the blocked area 604 and a spatial area which is outside the detection range of the sensor of the own vehicle 600 but is necessary for the peripheral condition detection as the request range ( S663 ).

[0343] When the request range determined by step S663 exists ("Yes" in S664), the search unit 623 searches for surrounding vehicles that have information related to the request range. For example, the search unit 623 can ask the surrounding vehicles whether they have information related to the request range, and determine whether the surrounding vehicles have information related to the request range based on the request range and the positions of the surrounding vehicles. Next, the search unit 623 sends a delegation signal 637 for entrusting the transmission of three-dimensional data to the surrounding vehicles 601 determined by the search. In addition, after receiving the permission signal sent from the surrounding vehicle 601 indicating acceptance of the delegation signal 637, the search unit 623 sends the request range information 633 indicating the request range to the surrounding vehicles 601 (S665).

[0344] Next, the receiving unit 624 detects a transmission notification of the transmission data 638 as information on the request range, and receives the transmission data 638 (S666).

[0345] In addition, the three-dimensional data production device 620A may not search for the other party to send the request, but may indiscriminately send the request to all vehicles existing in a certain range, and receive the transmission data 638 from the other party that responds that it has information related to the requested range. In addition, the search unit 623 is not limited to vehicles, and may send the request to objects such as traffic lights or signs, and receive the transmission data 638 from the objects.

[0346] Furthermore, the transmission data 638 includes at least one of the encoded three-dimensional data 634 generated by the surrounding vehicle 601 and obtained by encoding the three-dimensional data of the requested range, and the surrounding condition detection result 639 of the requested range. The surrounding condition detection result 639 shows the position, moving direction, and moving speed of the person and vehicle detected by the surrounding vehicle 601. Furthermore, the transmission data 638 may also include information showing the position and movement of the surrounding vehicle 601. For example, the transmission data 638 may also include braking information of the surrounding vehicle 601.

[0347] When the received transmission data 638 includes the encoded three-dimensional data 634 (Yes in S667), the decoder 625 decodes the encoded three-dimensional data 634 to obtain the second three-dimensional data 635 of SWLD (S668). That is, the second three-dimensional data 635 is three-dimensional data (SWLD) generated by extracting data having a feature quantity greater than a threshold value from the fourth three-dimensional data (WLD).

[0348] Next, the first three-dimensional data 632 and the second three-dimensional data 635 are synthesized by the synthesizing unit 626 to generate third three-dimensional data 636 ( S669 ).

[0349] Next, the surrounding condition detection unit 628 detects the surrounding condition of the own vehicle 600 using the third three-dimensional data 636, which is the point cloud data of the spatial region required for the surrounding condition detection (S670). In addition, when the received transmission data 638 includes the surrounding condition detection result 639, the surrounding condition detection unit 628 uses the surrounding condition detection result 639 in addition to the third three-dimensional data 636 to detect the surrounding condition of the own vehicle 600. In addition, when the received transmission data 638 includes the braking information of the surrounding vehicle 601, the surrounding condition detection unit 628 uses the braking information in addition to the third three-dimensional data 636 to detect the surrounding condition of the own vehicle 600.

[0350] Next, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the own vehicle 600 (S671) based on the peripheral condition detection result performed by the peripheral condition detection unit 628. In addition, the peripheral condition detection result can be presented to the driver through a UI (user interface) or the like.

[0351] Furthermore, when the requested range does not exist in step S663 (No in S664), that is, when all the information of the spatial region required for the peripheral condition detection is generated based on the sensor information 631, the peripheral condition detection unit 628 detects the peripheral condition of the own vehicle 600 using the point cloud data of the spatial region required for the peripheral condition detection, that is, the first three-dimensional data 632 (S672). Then, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the own vehicle 600 according to the peripheral condition detection result performed by the peripheral condition detection unit 628 (S671).

[0352] Furthermore, when the received transmission data 638 does not include the encoded three-dimensional data 634 (No in S667), that is, when the transmission data 638 includes only the surrounding condition detection result 639 or the braking information of the surrounding vehicle 601, the surrounding condition detection unit 628 detects the surrounding condition of the own vehicle 600 using the first three-dimensional data 632 and the surrounding condition detection result 639 or the braking information (S673). Then, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the own vehicle 600 according to the surrounding condition detection result performed by the surrounding condition detection unit 628 (S671).

[0353] Next, the three-dimensional data transmitting device 640A that transmits the transmission data 638 to the three-dimensional data creating device 620A described above will be described. Fig.32 It is a block diagram of the three-dimensional data transmitting device 640A.

[0354] Fig.32 The three-dimensional data transmitting device 640A shown in FIG. Fig.28 In addition to the configuration of the three-dimensional data transmitting device 640 shown, a transmission availability determination unit 646 is further provided. Furthermore, the three-dimensional data transmitting device 640A is included in the surrounding vehicle 601 .

[0355] Fig.33 1 is a flowchart showing an operation example of the three-dimensional data transmitting device 640A. First, the three-dimensional data creating unit 641 creates fifth three-dimensional data 652 using sensor information 651 detected by a sensor included in the surrounding vehicle 601 (S681).

[0356] Next, the receiving unit 642 receives a delegation signal 637 (S682) for requesting the transmission of three-dimensional data from the own vehicle 600. Next, the transmission feasibility determination unit 646 decides whether to respond to the delegation indicated by the delegation signal 637 (S683). For example, the transmission feasibility determination unit 646 decides whether to respond to the delegation based on the content pre-set by the user. Alternatively, the receiving unit 642 may first accept a request from the other party such as a request range, and the transmission feasibility determination unit 646 may decide whether to respond to the delegation based on the content. For example, the transmission feasibility determination unit 646 may decide to respond to the delegation when the three-dimensional data within the requested range is held, and decide not to respond to the delegation when the three-dimensional data within the requested range is not held.

[0357] In the case of a response request ("Yes" in S683), the three-dimensional data transmission device 640A transmits a permission signal to the own vehicle 600, and the receiving unit 642 receives the request range information 633 indicating the request range (S684). Next, the extraction unit 643 extracts the point cloud data of the request range from the fifth three-dimensional data 652 as point cloud data, and creates the transmission data 638 including the SWLD of the extracted point cloud data, that is, the sixth three-dimensional data 654 (S685).

[0358] That is, the three-dimensional data transmitting device 640A may generate the seventh three-dimensional data (WLD) based on the sensor information 651, and generate the fifth three-dimensional data 652 (SWLD) by extracting data having a feature quantity greater than a threshold value from the seventh three-dimensional data (WLD). Also, the three-dimensional data generating unit 641 may generate the three-dimensional data of SWLD in advance, and the extracting unit 643 may extract the three-dimensional data of SWLD in the requested range from the three-dimensional data of SWLD, or the extracting unit 643 may generate the three-dimensional data of SWLD in the requested range based on the three-dimensional data of WLD in the requested range.

[0359] Furthermore, the transmission data 638 may include the surrounding condition detection result 639 of the requested range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601. Furthermore, the transmission data 638 may not include the sixth three-dimensional data 654, but may include only at least one of the surrounding condition detection result 639 of the requested range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601.

[0360] When the transmission data 638 includes the sixth three-dimensional data 654 (Yes in S686), the encoding unit 644 encodes the sixth three-dimensional data 654 to generate the encoded three-dimensional data 634 (S687).

[0361] Then, the transmitting unit 645 transmits the transmission data 638 including the encoded three-dimensional data 634 to the own vehicle 600 (S688).

[0362] Furthermore, in the case where the sending data 638 does not include the sixth three-dimensional data 654 ("No" in S686), the sending unit 645 sends the sending data 638 including at least one of the surrounding condition detection result 639 of the requested range performed by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601 to the own vehicle 600 (S688).

[0363] Modifications of this embodiment will be described below.

[0364] For example, the information transmitted from the surrounding vehicle 601 may not be the three-dimensional data or the surrounding condition detection result produced by the surrounding vehicle, but may be the correct feature point information of the surrounding vehicle 601 itself. The own vehicle 600 uses the feature point information of the surrounding vehicle 601 to correct the feature point information of the vehicle ahead in the point cloud data obtained by the own vehicle 600. Accordingly, the own vehicle 600 can improve the matching accuracy when estimating its own position.

[0365] Furthermore, the feature point information of the vehicle ahead is three-dimensional point information composed of color information and coordinate information, for example. Thus, even if the sensor of the vehicle 600 is a laser sensor or a stereo camera, the feature point information of the vehicle ahead can be used regardless of its type.

[0366] In addition, the vehicle 600 is not limited by the transmission time, and can use the point cloud data of SWLD when calculating the accuracy of the estimated position of the vehicle. For example, when the sensor of the vehicle 600 is a stereo camera or other imaging device, the two-dimensional points on the image captured by the camera of the vehicle 600 are detected, and the vehicle's position is estimated using the two-dimensional points. In addition, while estimating its own position, the vehicle 600 creates point cloud data of surrounding objects. The vehicle 600 re-projects the three-dimensional points of SWLD onto the two-dimensional image, and evaluates the accuracy of the estimated position of the vehicle based on the error between the detected points on the two-dimensional image and the re-projected points.

[0367] Furthermore, when the sensor of the own vehicle 600 is a laser sensor such as LIDAR, the own vehicle 600 evaluates the accuracy of its own position estimation based on the error calculated by the SWLD of the produced point cloud data and the SWLD of the three-dimensional map through the iterative closest point algorithm (Iterative Closest Point).

[0368] Furthermore, when the communication status via a base station or server such as 5G is poor, the own vehicle 600 can obtain a three-dimensional map from the surrounding vehicles 601.

[0369] Furthermore, information about distant places that cannot be obtained from vehicles around the own vehicle 600 can be obtained through communication between vehicles. For example, information about a traffic accident that just occurred a few hundred meters or kilometers ahead can be obtained by the own vehicle 600 through communication with oncoming vehicles or by sequential transmission from surrounding vehicles. At this time, the data format of the transmitted data is that it is transmitted as meta-information of the upper layer of the dynamic three-dimensional map.

[0370] Furthermore, the detection results of the surrounding conditions and the information detected by the own vehicle 600 can be presented to the user through the user interface. For example, the presentation of such information can be realized by overlaying it on the navigation screen or the windshield.

[0371] Furthermore, it is also possible not to support autonomous driving, but when a vehicle with cruise control detects surrounding vehicles traveling in autonomous driving mode, it is possible to track the surrounding vehicles.

[0372] Furthermore, when a three-dimensional map cannot be obtained or the own position cannot be estimated due to too many obstructed areas, the own vehicle 600 can switch the working mode from the automatic driving mode to the tracking mode of surrounding vehicles.

[0373] And, the vehicle being tracked can warn the user of the situation being tracked, and a user interface that the user can specify whether to allow tracking can also be installed. At this time, advertising display can be designed on the tracking vehicle, and payment rewards can be designed on the tracked side.

[0374] Furthermore, although the information transmitted is basically SWLD as three-dimensional data, it may be information corresponding to the request setting set in the own vehicle 600 or the public setting of the vehicle traveling ahead. For example, the information transmitted may be SWLD of dense point cloud data, or the detection result of the surrounding conditions by the vehicle traveling ahead, or the braking information of the vehicle traveling ahead.

[0375] Furthermore, the own vehicle 600 receives the WLD, visualizes the three-dimensional data of the WLD, and prompts the driver with the visualized three-dimensional data using the GUI. At this time, the own vehicle 600 can use different colors to represent information in a manner that the point cloud data generated by the own vehicle 600 and the received point cloud data can be distinguished by the user for prompting.

[0376] Furthermore, when the own vehicle 600 uses the GUI to prompt the driver with information detected by the own vehicle 600 and detection results of surrounding vehicles 601, the information is prompted by color differentiation, etc. in a manner that the user can distinguish between the information detected by the own vehicle 600 and the received detection results.

[0377] As described above, in the three-dimensional data creation device 620 according to the present embodiment, the three-dimensional data creation unit 621 creates the first three-dimensional data 632 based on the sensor information 631 detected by the sensor. The receiving unit 624 receives the encoded three-dimensional data 634 obtained by encoding the second three-dimensional data 635. The decoding unit 625 obtains the second three-dimensional data 635 by decoding the received encoded three-dimensional data 634. The synthesis unit 626 synthesizes the first three-dimensional data 632 and the second three-dimensional data 635 to create the third three-dimensional data 636.

[0378] According to this, the three-dimensional data creation device 620 can create detailed third three-dimensional data 636 using the created first three-dimensional data 632 and the received second three-dimensional data 635 .

[0379] Furthermore, the synthesis unit 626 can synthesize the first three-dimensional data 632 and the second three-dimensional data 635 to create third three-dimensional data 636 having a higher density than the first three-dimensional data 632 and the second three-dimensional data 635 .

[0380] Furthermore, the second three-dimensional data 635 (for example, SWLD) is three-dimensional data generated by extracting data having a feature amount equal to or greater than a threshold value from the fourth three-dimensional data (for example, WLD).

[0381] According to this, the three-dimensional data creation device 620 can reduce the amount of three-dimensional data to be transmitted.

[0382] Furthermore, the three-dimensional data creating device 620 further includes a search unit 623 that searches for a transmission device that is a transmission source of the encoded three-dimensional data 634. The receiving unit 624 receives the encoded three-dimensional data 634 from the searched transmission device.

[0383] According to this, the three-dimensional data creation device 620 can identify the transmission device holding the required three-dimensional data by searching, for example.

[0384] The three-dimensional data production device further includes a request range determination unit 622 that determines a request range that is a range of a three-dimensional space for requesting three-dimensional data. The search unit 623 sends request range information 633 indicating the request range to the transmission device. The second three-dimensional data 635 includes three-dimensional data of the request range.

[0385] Accordingly, the three-dimensional data creation device 620 can not only receive the required three-dimensional data but also reduce the amount of three-dimensional data to be transmitted.

[0386] Furthermore, the request range determination unit 622 determines the spatial range including the blocked area 604 that cannot be detected by the sensor as the request range.

[0387] Furthermore, in the three-dimensional data transmitting device 640 according to the present embodiment, the three-dimensional data generating unit 641 generates fifth three-dimensional data 652 based on the sensor information 651 detected by the sensor. The extracting unit 643 generates sixth three-dimensional data 654 by extracting a part of the fifth three-dimensional data 652. The encoding unit 644 generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654. The transmitting unit 645 transmits the encoded three-dimensional data 634.

[0388] According to this, the three-dimensional data transmitting device 640 can not only transmit the three-dimensional data produced by itself to other devices, but also reduce the data volume of the three-dimensional data to be transmitted.

[0389] Furthermore, the three-dimensional data generating unit 641 generates seventh three-dimensional data (eg, WLD) based on sensor information 651 detected by the sensor, and generates fifth three-dimensional data 652 (eg, SWLD) by extracting data having a feature value greater than a threshold value from the seventh three-dimensional data.

[0390] Accordingly, the three-dimensional data transmitting device 640 can reduce the amount of three-dimensional data to be transmitted.

[0391] In addition, the three-dimensional data transmitting device 640 further includes a receiving unit 642, which receives request range information 633 indicating a request range, which is a range of a three-dimensional space for which three-dimensional data is requested, from the receiving device. The extracting unit 643 extracts the three-dimensional data of the request range from the fifth three-dimensional data 652, thereby generating the sixth three-dimensional data 654. The transmitting unit 645 transmits the encoded three-dimensional data 634 to the receiving device.

[0392] Accordingly, the three-dimensional data transmitting device 640 can reduce the amount of three-dimensional data to be transmitted.

[0393] (Implementation 4)

[0394] In this embodiment, the operation related to abnormal situations in the estimation of the own position based on the three-dimensional map will be described.

[0395] The use of autonomous movement of moving objects such as automatic driving of automobiles, robots, or flying objects such as drones will expand in the future. An example of a method for achieving such autonomous movement is a method in which a moving object estimates its position in a three-dimensional map (self-position estimation) and drives according to the map.

[0396] The self-position estimation is achieved by matching the three-dimensional map with the three-dimensional information around the own vehicle obtained by sensors such as the rangefinder (LIDAR, etc.) or stereo camera mounted on the own vehicle (hereinafter referred to as the own vehicle detection three-dimensional data), and estimating the own vehicle position within the three-dimensional map.

[0397] As shown in the HD map proposed by HERE, a 3D map is not only a 3D point cloud, but also includes 2D map data such as road and intersection shape information, and information such as congestion and accidents that change in real time. The 3D map is composed of multiple layers such as 3D data, 2D data, and metadata that changes in real time, and the device can obtain only the required data or refer to the required data.

[0398] The point cloud data may be the above-mentioned SWLD, or may include point group data that is not a feature point. Furthermore, the transmission and reception of the point cloud data is basically performed in one or more random access units.

[0399] As a matching method for the three-dimensional map and the three-dimensional data detected by the own vehicle, the following method can be adopted. For example, the device compares the shapes of the point groups in each other's point clouds and determines the parts with high similarity between the feature points as the same position. In addition, when the three-dimensional map is composed of SWLD, the device compares and matches the feature points constituting the SWLD with the three-dimensional feature points extracted from the three-dimensional data detected by the own vehicle.

[0400] Here, in order to estimate the own position with high accuracy, the following (A) and (B) need to be met: (A) a three-dimensional map and three-dimensional data for detecting the own vehicle are available, and (B) their accuracy meets a predetermined benchmark. However, in the following abnormal situation, (A) or (B) cannot be met.

[0401] (1) A three-dimensional map cannot be obtained through the communication path.

[0402] (2) There is no three-dimensional map or the obtained three-dimensional map is damaged.

[0403] (3) The sensor of the own vehicle fails, or due to bad weather, the accuracy of the three-dimensional data generated by the own vehicle is insufficient.

[0404] The following describes the operation for dealing with these abnormal situations. Although the operation is described below using a vehicle as an example, the following method can also be applied to all moving objects that move autonomously, such as robots and drones.

[0405] The following describes the configuration and operation of the three-dimensional information processing device according to the present embodiment for detecting abnormalities in three-dimensional data corresponding to a three-dimensional map or a vehicle. Fig.34 It is a block diagram showing a configuration example of a three-dimensional information processing device 700 according to this embodiment. Fig.35 3D information processing method performed by the 3D information processing apparatus 700 is a flowchart.

[0406] The three-dimensional information processing device 700 is mounted on a mobile object such as a motor vehicle. Fig.34 As shown, the three-dimensional information processing device 700 includes a three-dimensional map acquisition unit 701 , a vehicle detection data acquisition unit 702 , an abnormality determination unit 703 , a countermeasure operation determination unit 704 , and an operation control unit 705 .

[0407] In addition, the three-dimensional information processing device 700 may also include a camera for obtaining a two-dimensional image, or may include a sensor for one-dimensional data using ultrasonic waves or lasers, which is used to detect structural objects or moving objects around the vehicle, which is not shown in the figure. In addition, the three-dimensional information processing device 700 may also include a communication unit (not shown) for obtaining a three-dimensional map through a mobile communication network such as 4G or 5G, or communication between vehicles, or communication between roads and vehicles.

[0408] like Fig.35 As shown, the three-dimensional map acquisition unit 701 acquires a three-dimensional map 711 near the driving route (S701). For example, the three-dimensional map acquisition unit 701 acquires the three-dimensional map 711 through a mobile communication network, or communication between vehicles, or communication between roads and vehicles.

[0409] Next, the vehicle detection data acquisition unit 702 acquires the vehicle detection three-dimensional data 712 based on the sensor information (S702). For example, the vehicle detection data acquisition unit 702 generates the vehicle detection three-dimensional data 712 based on the sensor information acquired by the sensor of the vehicle.

[0410] Next, the abnormality determination unit 703 detects an abnormality (S703) by performing a predetermined check on at least one of the obtained three-dimensional map 711 and the vehicle detection three-dimensional data 712. That is, the abnormality determination unit 703 determines whether at least one of the obtained three-dimensional map 711 and the vehicle detection three-dimensional data 712 is abnormal.

[0411] In step S703, when an abnormal situation is detected ("Yes" in S704), the response work determination unit 704 determines the response work for the abnormal situation (S705). Next, the work control unit 705 controls the work of each processing unit required for the implementation of the response work, such as the three-dimensional map acquisition unit 701 (S706).

[0412] In addition, when no abnormality is detected in step S703 (No in S704), the three-dimensional information processing device 700 ends the processing.

[0413] The three-dimensional information processing device 700 estimates the own position of the vehicle having the three-dimensional information processing device 700 using the three-dimensional map 711 and the own vehicle detection three-dimensional data 712. Then, the three-dimensional information processing device 700 uses the result of the own position estimation to make the vehicle automatically drive.

[0414] Accordingly, the three-dimensional information processing device 700 obtains map data (three-dimensional map 711) including the first three-dimensional position information via the channel. For example, the first three-dimensional position information is encoded in units of partial spaces having three-dimensional coordinate information, and the first three-dimensional position information includes a plurality of random access units, each of which is a collection of more than one partial space and can be independently decoded. For example, the first three-dimensional position information is data (SWLD) encoded with a feature point whose three-dimensional feature quantity is above a predetermined threshold.

[0415] Furthermore, the three-dimensional information processing device 700 generates the second three-dimensional position information (the vehicle detection three-dimensional data 712) based on the information detected by the sensor. Next, the three-dimensional information processing device 700 performs an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information to determine whether the first three-dimensional position information or the second three-dimensional position information is abnormal.

[0416] When determining that the first three-dimensional position information or the second three-dimensional position information is abnormal, the three-dimensional information processing device 700 determines a countermeasure for the abnormality. Next, the three-dimensional information processing device 700 executes control required for executing the countermeasure.

[0417] According to this, the three-dimensional information processing device 700 can detect abnormality in the first three-dimensional position information or the second three-dimensional position information and perform a corresponding operation.

[0418] The following describes the response to abnormal situation 1, that is, the situation where the three-dimensional map 711 cannot be obtained via communication.

[0419] The three-dimensional map 711 is required for estimating the vehicle's own position, but if the vehicle has not obtained the three-dimensional map 711 corresponding to the path to the destination in advance, it is necessary to obtain the three-dimensional map 711 through communication. However, due to channel congestion or deterioration of radio wave reception, the vehicle may not be able to obtain the three-dimensional map 711 on the driving path.

[0420] The abnormal situation determination unit 703 confirms whether the three-dimensional map 711 of all sections on the path to the destination or the sections within a specified range from the current position has been obtained, and if it cannot be obtained, it is determined as abnormal situation 1. That is, the abnormal situation determination unit 703 determines whether the three-dimensional map 711 (first three-dimensional position information) can be obtained via the channel, and if the three-dimensional map 711 cannot be obtained via the channel, the three-dimensional map 711 is determined to be abnormal.

[0421] When it is determined that the abnormal situation is 1, the countermeasure decision unit 704 selects one of the following two countermeasures: (1) continuing the self-position estimation, and (2) stopping the self-position estimation.

[0422] First, (1) a specific example of the operation to be performed when the self-position estimation is continued will be described. When the self-position estimation is continued, a three-dimensional map 711 on the route to the destination is required.

[0423] For example, the vehicle determines a place where a channel can be used within the range of the three-dimensional map 711 that has been obtained, moves to the place, and obtains the three-dimensional map 711. At this time, the vehicle can obtain all three-dimensional maps 711 to the destination, or obtain the three-dimensional map 711 per random access unit within the upper limit size that can be maintained in the storage unit or HDD of the vehicle itself.

[0424] In addition, the following operation can be performed: the vehicle obtains the communication status on the path separately, and when it is predicted that the communication status on the path will deteriorate, the three-dimensional map 711 of the section is obtained in advance before reaching the section with poor communication status, or the three-dimensional map 711 of the maximum range that can be obtained is obtained in advance. That is, the three-dimensional information processing device 700 predicts whether the vehicle will enter an area with poor communication status. When the three-dimensional information processing device 700 predicts that the vehicle will enter an area with poor communication status, the three-dimensional map 711 is obtained before the vehicle enters the area.

[0425] Furthermore, the vehicle may determine a random access unit of the three-dimensional map 711 that is narrower than the normal range and is the minimum required for estimating the vehicle's position on the path, and receive the determined random access unit. That is, when the three-dimensional information processing device 700 cannot obtain the three-dimensional map 711 (first three-dimensional position information) via the channel, it can obtain the third three-dimensional position information that is narrower than the first three-dimensional position information via the channel.

[0426] Furthermore, when the vehicle cannot access the distribution server of the three-dimensional map 711, it can obtain the three-dimensional map 711 from other vehicles and other mobile bodies traveling around its own vehicle. At this time, the other vehicles and the like are mobile bodies that have already obtained the three-dimensional map 711 on the path to the destination and can communicate with its own vehicle.

[0427] Next, a specific example of the operation in the case of (2) stopping the estimation of the own position will be described. In this case, the three-dimensional map 711 on the route to the destination is not necessary.

[0428] For example, the vehicle will notify the driver that functions such as autonomous driving based on its own position estimation cannot continue to be executed, and the operating mode will be transferred to a manual operation mode performed by the driver.

[0429] Usually, when estimating the own position, the level may be different depending on the presence of a person, but autonomous driving is performed. In addition, the result of the self-position estimation may also be used for navigation when a person is driving. Therefore, the result of the self-position estimation does not necessarily need to be used for autonomous driving.

[0430] Furthermore, when the vehicle cannot utilize commonly used channels such as 4G or 5G mobile communication networks, it is possible to confirm whether the three-dimensional map 711 can be obtained via other channel paths such as Wi-Fi (registered trademark) or millimeter wave communication between the road and the vehicle, or communication between vehicles, and the channel in use can be switched to a channel that can obtain the three-dimensional map 711.

[0431] Furthermore, when the vehicle cannot obtain the three-dimensional map 711, it can also obtain a two-dimensional map, and continue the automatic driving by using the two-dimensional map and the self-vehicle detection three-dimensional data 712. That is, when the three-dimensional information processing device 700 cannot obtain the three-dimensional map 711 via the channel, it can obtain map data (two-dimensional map) including two-dimensional position information via the channel, and estimate the vehicle's own position by using the two-dimensional position information and the self-vehicle detection three-dimensional data 712.

[0432] Specifically, the vehicle uses the two-dimensional map and the vehicle's own detection three-dimensional data 712 to estimate its own position, and uses the vehicle's own detection three-dimensional data 712 to detect surrounding vehicles, pedestrians, and obstacles.

[0433] Here, map data such as HD maps may include not only the three-dimensional map 711 composed of three-dimensional point clouds, but also two-dimensional map data (two-dimensional map), simplified map data in which characteristic information such as road shapes and intersections are extracted from the two-dimensional map data, and metadata showing actual time information such as congestion, accidents, or construction. For example, the map data has a layer structure of three-dimensional data (three-dimensional map 711), two-dimensional data (two-dimensional map), and metadata arranged in order from the lower layer.

[0434] Here, the two-dimensional data is smaller than the three-dimensional data. Therefore, even if the communication status is not good, the vehicle can obtain the two-dimensional map. In addition, the vehicle can uniformly obtain a large-scale two-dimensional map in a section with good communication status. Therefore, when the vehicle has difficulty in obtaining the three-dimensional map 711 due to the poor channel status, it is possible to receive the layer including the two-dimensional map instead of the three-dimensional map 711. In addition, since the data size of the metadata is small, for example, the vehicle can always receive the metadata without being affected by the communication status.

[0435] Among the methods of estimating the own position using the two-dimensional map and the own vehicle detection three-dimensional data 712, there are, for example, the following two methods.

[0436] The first method is a method of matching two-dimensional feature quantities. Specifically, the vehicle extracts two-dimensional feature quantities from the vehicle detection three-dimensional data 712, and matches the extracted two-dimensional feature quantities with the two-dimensional map.

[0437] For example, the vehicle projects the vehicle detection three-dimensional data 712 onto the same plane as the two-dimensional map, and matches the obtained two-dimensional data with the two-dimensional map. The matching is performed using two-dimensional image feature quantities extracted from both.

[0438] When the three-dimensional map 711 includes SWLD, the three-dimensional map 711 can simultaneously store the three-dimensional feature quantity at the feature point in the three-dimensional space and the two-dimensional feature quantity in the same plane as the two-dimensional map. For example, identification information is given to the two-dimensional feature quantity. Alternatively, the two-dimensional feature quantity is stored in a layer different from the three-dimensional data and the two-dimensional map, and the vehicle obtains the data of the two-dimensional feature quantity at the same time as obtaining the two-dimensional map.

[0439] When a two-dimensional map displays information on the positions of white lines, guardrails, buildings, etc. on the road at different heights from the ground (not in the same plane) on the same map, the vehicle extracts feature values ​​from multiple height data of the three-dimensional data 712 detected by its own vehicle.

[0440] Furthermore, information showing the correspondence between feature points in the two-dimensional map and feature points in the three-dimensional map 711 may be stored as meta-information of the map data.

[0441] The second method is a method of matching three-dimensional feature quantities. Specifically, the vehicle obtains three-dimensional feature quantities corresponding to feature points in the two-dimensional map and matches the obtained three-dimensional feature quantities with the three-dimensional feature quantities of the vehicle detection three-dimensional data 712 .

[0442] Specifically, the three-dimensional feature quantity corresponding to the feature point of the two-dimensional map is stored in the map data. When the vehicle obtains the two-dimensional map, it also obtains the three-dimensional feature quantity. In addition, when the three-dimensional map 711 includes SWLD, by assigning information for identifying the feature point corresponding to the feature point of the two-dimensional map among the feature points of the SWLD, the vehicle can determine the vehicle corresponding to the two-dimensional map based on the identification information. Figure 1 In addition, in this case, since it is sufficient to express the two-dimensional position, the amount of data can be reduced compared to the case of expressing the three-dimensional position.

[0443] Furthermore, when estimating its own position using a two-dimensional map, the accuracy of its own position estimation is lower than that of the three-dimensional map 711. Therefore, the vehicle determines whether to continue autonomous driving even when the estimation accuracy is reduced, and continues autonomous driving only when it is determined that it can continue.

[0444] Whether the autonomous driving can continue is affected by whether the road the vehicle is traveling on is an urban area, a highway or other road with few vehicles or pedestrians, or the driving environment such as the road width and the road clutter (the density of vehicles or pedestrians). In addition, markers for sensors such as cameras to identify can be arranged on corporate land, streets, or inside buildings. In these specific areas, since the markers can be identified with high accuracy by a two-dimensional sensor, for example, by including the location information of the markers in a two-dimensional map, it is possible to perform a high-precision self-position estimation.

[0445] Furthermore, by including identification information indicating whether each area is a specific area in the map, the vehicle can determine whether the vehicle is in a specific area. If the vehicle is in a specific area, the vehicle determines to continue the automatic driving. In this way, the vehicle can determine whether to continue the automatic driving based on the accuracy of its own position estimation when using the two-dimensional map or the driving environment of the vehicle.

[0446] In this way, the three-dimensional information processing device 700 can determine whether to perform automatic driving of the vehicle based on the vehicle's driving environment (the moving environment of the moving body), which is the result of estimating the vehicle's own position using a two-dimensional map and the vehicle's own detection three-dimensional data 712.

[0447] Furthermore, the vehicle may switch the level (mode) of autonomous driving according to the accuracy of its own position estimation or the driving environment of the vehicle without determining whether it can continue autonomous driving. Here, switching the level (mode) of autonomous driving may include, for example, limiting the speed, increasing the amount of operation by the driver (lowering the level of autonomous driving), obtaining driving information of the vehicle ahead and switching the mode with reference to the information, obtaining driving information of the vehicle set to the same destination and switching the mode of autonomous driving using the information, etc.

[0448] Furthermore, the map may include information corresponding to the position information, indicating the recommended level of autonomous driving when estimating the vehicle's own position using a two-dimensional map. The recommended level may be metadata that changes dynamically according to traffic volume, etc. Accordingly, the vehicle does not need to determine the level in sequence according to the surrounding environment, etc., but can determine the level only by obtaining information in the map. Furthermore, by having multiple vehicles refer to the same map, the level of autonomous driving of each vehicle can be kept stable. In addition, the recommended level may not be a recommended level, but a level that must be followed.

[0449] Furthermore, the vehicle can switch the level of autonomous driving according to whether there is a driver (manned or unmanned). For example, the vehicle reduces the level of autonomous driving when there is a driver, and stops when there is no one. The vehicle determines the position where it can stop safely by identifying the pedestrians, vehicles and traffic signs around it. Alternatively, the map may include location information showing the position where the vehicle can stop safely, and the vehicle can refer to the location information to determine the position where it can stop safely.

[0450] Next, the operation to be performed in abnormal situation 2, that is, when the three-dimensional map 711 does not exist or the obtained three-dimensional map 711 is damaged, is described.

[0451] The abnormal situation determination unit 703 determines which of the following situations (1) and (2) it belongs to, and determines that it belongs to abnormal situation 2 in any of the situations, (1) the three-dimensional map 711 in part or all of the sections on the path to the destination does not exist in the distribution server that becomes the access destination and cannot be obtained, or (2) part or all of the obtained three-dimensional map 711 is damaged. In other words, the abnormal situation determination unit 703 determines whether the data of the three-dimensional map 711 is complete, and in the case where the data of the three-dimensional map 711 is incomplete, the three-dimensional map 711 is determined to be abnormal.

[0452] When it is determined to be abnormal situation 2, the following countermeasures are performed. First, an example of the countermeasures in the case where (1) the three-dimensional map 711 cannot be obtained is described.

[0453] For example, the vehicle sets a route that does not pass through a section without the three-dimensional map 711 .

[0454] Furthermore, when the vehicle cannot set an alternative route because there is no alternative route or even if there is an alternative route, the distance will be greatly increased, etc., the vehicle sets a route including a section without the three-dimensional map 711. Furthermore, the vehicle notifies the driver to switch the driving mode in the section to switch the driving mode to the manual operation mode.

[0455] (2) When part or all of the acquired three-dimensional map 711 is destroyed, the following countermeasures are performed.

[0456] The vehicle determines the damaged part in the three-dimensional map 711, requests the data of the damaged part through communication, obtains the data of the damaged part, and updates the three-dimensional map 711 with the obtained data. At this time, the vehicle can specify the damaged part by the position information such as the absolute coordinates or relative coordinates in the three-dimensional map 711, or by the index number of the random access unit constituting the damaged part. In this case, the vehicle replaces the random access unit including the damaged part with the obtained random access unit.

[0457] Next, the operation to be performed in abnormal situation 3, that is, when the sensor of the own vehicle cannot generate the own vehicle detection three-dimensional data 712 due to a malfunction or bad weather, will be described.

[0458] The abnormality determination unit 703 confirms whether the generation error of the own vehicle detection three-dimensional data 712 is within the allowable range, and if not, determines it as abnormality 3. That is, the abnormality determination unit 703 determines whether the generation accuracy of the data of the own vehicle detection three-dimensional data 712 is above the reference value, and if the generation accuracy of the data of the own vehicle detection three-dimensional data 712 is not above the reference value, the own vehicle detection three-dimensional data 712 is determined to be abnormal.

[0459] As a method of confirming whether the generation error of the own vehicle detection three-dimensional data 712 is within the allowable range, the following method can be adopted.

[0460] The spatial resolution of the self-vehicle detection three-dimensional data 712 during normal operation is predetermined based on the resolution in the depth direction and scanning direction of the self-vehicle's three-dimensional sensor such as a rangefinder or a stereo camera, or the density of the point group that can be generated. In addition, the vehicle obtains the spatial resolution of the three-dimensional map 711 based on the meta information included in the three-dimensional map 711.

[0461] The vehicle uses the spatial resolution of both to estimate the reference value of the matching error when matching the self-vehicle detection three-dimensional data 712 with the three-dimensional map 711 based on the three-dimensional feature quantity, etc. As the matching error, it is possible to use statistics such as the error of the three-dimensional feature quantity of each feature point, the average value of the error of the three-dimensional feature quantity between multiple feature points, or the error of the spatial distance between multiple feature points, etc. The allowable range of deviation from the reference value is set in advance.

[0462] When the matching error between the vehicle detection three-dimensional data 712 generated before or during the vehicle starts traveling and the three-dimensional map 711 is not within the allowable range, it is determined to be abnormal situation 3.

[0463] Alternatively, the vehicle may utilize a test pattern with a known three-dimensional shape for accuracy checking to obtain its own vehicle detection three-dimensional data 712 for test patterns such as before driving begins, and determine whether it is an abnormal situation 3 based on whether the shape error is within an allowable range.

[0464] For example, the vehicle performs the above judgment every time before starting to drive. Alternatively, the vehicle performs the above judgment at certain time intervals during driving, thereby obtaining a time series change of the matching error. When the matching error of the vehicle tends to increase, even if the error is within the allowable range, it can be judged as an abnormal situation 3. In addition, when the vehicle can predict that it will become abnormal based on the time series change, it can notify the user of the predicted abnormal situation by displaying a message urging inspection or repair. In addition, the vehicle can judge the abnormality caused by temporary reasons such as bad weather and the abnormality caused by sensor failure based on the time series change, and only notify the user of the abnormality caused by sensor failure.

[0465] Furthermore, when the vehicle is judged to be in abnormal situation 3, any one of the following three response actions is selected or selectively executed: (1) operating an emergency replacement sensor (rescue mode), (2) switching the operating mode, and (3) calibrating the operation of the three-dimensional sensor.

[0466] First, (1) the case where an alternative sensor for emergency use is operated is described. The vehicle operates an alternative sensor for emergency use that is different from the three-dimensional sensor used during normal operation. That is, when the generation accuracy of the data of the own vehicle detection three-dimensional data 712 is not above the reference value, the three-dimensional information processing device 700 generates the own vehicle detection three-dimensional data 712 (the fourth three-dimensional position information) based on the information detected by the alternative sensor that is different from the normal sensor.

[0467] Specifically, when the vehicle uses multiple cameras or LIDARs to obtain the vehicle detection three-dimensional data 712, the vehicle determines the malfunctioning sensor based on the direction in which the matching error of the vehicle detection three-dimensional data 712 exceeds the allowable range, etc. Then, the vehicle activates the replacement sensor corresponding to the malfunctioning sensor.

[0468] The substitute sensor may be a three-dimensional sensor, a camera that obtains a two-dimensional image, or a one-dimensional sensor such as ultrasonic wave, etc. When the substitute sensor is a sensor other than a three-dimensional sensor, the accuracy of the self-position estimation may be reduced or the self-position estimation may not be performed. Therefore, the vehicle may switch the autonomous driving mode according to the type of substitute sensor.

[0469] For example, when the replacement sensor is a three-dimensional sensor, the vehicle continues the automatic driving mode. Also, when the replacement sensor is a two-dimensional sensor, the vehicle changes the operation mode from fully automatic driving to a semi-automatic driving mode that requires human operation. Also, when the replacement sensor is a one-dimensional sensor, the vehicle switches the operation mode to a manual operation mode that cannot perform automatic brake control.

[0470] Furthermore, the vehicle can also switch the autonomous driving mode according to the driving environment. For example, when the vehicle uses a two-dimensional sensor instead of a sensor, it can continue to operate in the fully autonomous driving mode when driving on a highway, and switch the operating mode to the semi-autonomous driving mode when driving in an urban area.

[0471] Furthermore, if the vehicle does not have a replacement sensor, it can continue to estimate its own position if the sensor that is operating normally can obtain a sufficient number of feature points. However, since it cannot detect a specific direction, the vehicle switches the operating mode to a quasi-autonomous driving mode or a manual operation mode.

[0472] Next, (2) the response work for switching the operation mode is explained. The vehicle switches the operation mode from the automatic driving mode to the manual operation mode. Alternatively, the vehicle can continue to drive automatically until it reaches a place such as a road shoulder where it can stop safely, and then stop. In addition, the vehicle can switch the operation mode to the manual operation mode after stopping. In this way, the three-dimensional information processing device 700 switches the automatic driving mode when the generation accuracy of the three-dimensional data 712 detected by the vehicle itself is not above the reference value.

[0473] Next, (3) will explain the response work for correcting the operation of the three-dimensional sensor. The vehicle determines the malfunctioning three-dimensional sensor based on the direction in which the matching error occurs, and calibrates the determined sensor. Specifically, when multiple LIDARs or cameras are used as sensors, a portion of the three-dimensional space reconstructed by each sensor overlaps. That is, the data of the overlapping portion is obtained by multiple sensors. The three-dimensional point group data obtained for the overlapping portion is different between the normal sensor and the malfunctioning sensor. Therefore, the vehicle performs origin correction of the LIDAR, or exposure and focus of the camera, etc., in a manner that enables the malfunctioning sensor to obtain the same three-dimensional point group data as the normal sensor, and adjusts the operation of the predetermined part.

[0474] After adjustment, if the matching error is within the allowable range, the vehicle continues the previous operation mode. In addition, after adjustment, if the matching accuracy is not within the allowable range, the vehicle performs the above-mentioned (1) emergency response work of the replacement sensor, or (2) switching the operation mode.

[0475] In this way, the three-dimensional information processing device 700 performs sensor operation calibration when the generation accuracy of the data of the own vehicle detection three-dimensional data 712 is not equal to or higher than the reference value.

[0476] The following describes a method for selecting a countermeasure operation. The countermeasure operation may be selected by a user such as a driver, or may be automatically selected by the vehicle without the user.

[0477] Furthermore, the vehicle may switch control depending on whether or not there is a driver on board. For example, if there is a driver on board, the vehicle prioritizes switching to a manual operation mode. If there is no driver on board, the vehicle prioritizes stopping by moving to a safe location.

[0478] The information indicating the stopping place may be included as meta information in the three-dimensional map 711. Alternatively, the vehicle may send a response request for the stopping place to a service that manages the operation information of the autonomous driving, thereby obtaining the information indicating the stopping place.

[0479] Furthermore, when the vehicle is running on a prescribed route, the operation mode of the vehicle can be shifted to a mode in which the operation of the vehicle is managed by an operator via a channel. In particular, in a vehicle running in a fully autonomous driving mode, the risk of abnormality in the self-position estimation function is high. Therefore, when an abnormal situation is detected, or when the detected abnormality cannot be corrected, the vehicle notifies the occurrence of the abnormality to a service that manages the operation information via a channel. The service can notify vehicles running around the vehicle of the presence of an abnormal vehicle, or issue an instruction to vacate a nearby parking space.

[0480] Also, when an abnormal situation is detected, the vehicle can travel slower than normal.

[0481] If the vehicle is an autonomous vehicle that provides a taxi or other car-sharing service, and an abnormality occurs in the vehicle, the vehicle will contact the operation management center to stop at a safe place. In addition, the car-sharing service may distribute a replacement vehicle. Alternatively, the user of the car-sharing service may drive the vehicle. In these situations, a discount on the fare or special points may be given.

[0482] Furthermore, in the method for dealing with abnormal situation 1, although the method of estimating the own position based on the two-dimensional map is described, the own position can also be estimated using the two-dimensional map in normal times. Fig.36 This is a flowchart of the self-position estimation process in this case.

[0483] First, the vehicle obtains a three-dimensional map 711 near the driving path (S711). Next, the vehicle obtains its own vehicle detection three-dimensional data 712 based on sensor information (S712).

[0484] Next, the vehicle determines whether a three-dimensional map 711 is needed when estimating its own position (S713). Specifically, the vehicle determines whether a three-dimensional map 711 is needed based on the accuracy of its own position estimation when using a two-dimensional map and the driving environment. For example, the same method as the above-mentioned method for dealing with abnormal situation 1 is adopted.

[0485] When it is determined that the three-dimensional map 711 is not needed ("No" in S714), the vehicle obtains a two-dimensional map (S715). At this time, the vehicle can also obtain the additional information described in the method for dealing with abnormal situation 1. In addition, the vehicle can generate a two-dimensional map based on the three-dimensional map 711. For example, the vehicle can extract an arbitrary plane from the three-dimensional map 711 to generate a two-dimensional map.

[0486] Next, the vehicle estimates its own position using the vehicle detection three-dimensional data 712 and the two-dimensional map (S716). The method of estimating the own position using the two-dimensional map is the same as the method described in the above-mentioned method for dealing with abnormal situation 1.

[0487] If the three-dimensional map 711 is determined to be necessary (Yes in S714), the vehicle obtains the three-dimensional map 711 (S717). Next, the vehicle estimates its own position using the vehicle detection three-dimensional data 712 and the three-dimensional map 711 (S718).

[0488] In addition, the vehicle can switch whether to basically use the two-dimensional map or the three-dimensional map 711 according to the corresponding speed of the communication device of the vehicle itself or the channel status. For example, while receiving the three-dimensional map 711, the communication speed required for driving is preset. When the communication speed during driving is below the set value, the vehicle basically uses the two-dimensional map, and when the communication speed during driving is greater than the set value, the three-dimensional map 711 is basically used. In addition, the vehicle may not switch and judge whether to use the two-dimensional map or the three-dimensional map, but basically use the two-dimensional map.

[0489] (Implementation method 5)

[0490] In this embodiment, a method of transmitting three-dimensional data to a rear vehicle and the like will be described. Fig.37 An example of the object space of three-dimensional data transmitted to the following vehicle or the like is shown.

[0491] Vehicle 801 sends three-dimensional data such as point cloud data (point group) included in the rectangular space 802 with width W, height H and depth D in front of vehicle 801 at a distance L from vehicle 801 to the traffic cloud monitoring or the rear vehicle that monitors the road conditions at a time interval of Δt.

[0492] When the three-dimensional data included in the space 802 that has been transmitted in the past changes due to a vehicle or a person entering the space 802 from outside, the vehicle 801 also transmits the three-dimensional data of the changed space.

[0493] In addition, Fig.37 Although an example in which the shape of the space 802 is a rectangular parallelepiped is shown, the space 802 does not necessarily need to be a rectangular parallelepiped as long as it includes a space on the road ahead that becomes a blind spot when viewed from the rear vehicle.

[0494] The distance L is preferably set to a distance at which the rear vehicle that has received the three-dimensional data can stop safely. For example, the distance L is set to the sum of the following distances: the distance the rear vehicle moves during the reception of the three-dimensional data, the distance the rear vehicle moves until it starts to decelerate according to the received data, and the distance the rear vehicle needs to stop safely in principle. Since these distances vary according to the speed, the distance L can vary according to the speed V of the vehicle as shown by L=a×V+b (a, b are constants).

[0495] The width W is set to a value greater than at least the width of the lane in which the vehicle 801 travels. More preferably, the width W is set to a size that includes adjacent spaces such as left and right lanes or curbs.

[0496] The depth D may be a fixed value, or may vary according to the vehicle speed V as shown in D=c×V+d (c, d are constants). Furthermore, by setting D in a manner such that D>V×Δt, the transmission space can overlap with the space that has been transmitted in the past. Thus, the vehicle 801 can more reliably transmit the space on the road to the following vehicle without omission.

[0497] In this way, by limiting the three-dimensional data transmitted by the vehicle 801 to a space useful to the rear vehicle, the capacity of the transmitted three-dimensional data can be effectively reduced, thereby achieving low latency and low cost of communication.

[0498] Next, the configuration of the three-dimensional data creation device 810 according to this embodiment will be described. Fig.381 is a block diagram showing a configuration example of a three-dimensional data creation device 810 according to the present embodiment. The three-dimensional data creation device 810 is, for example, mounted on a vehicle 801. The three-dimensional data creation device 810 transmits and receives three-dimensional data with external traffic cloud monitoring, a vehicle ahead, or a vehicle behind, and creates and accumulates the three-dimensional data.

[0499] The three-dimensional data production device 810 includes: a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data production unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a sending control unit 820, a format conversion unit 821, and a data sending unit 822.

[0500] The data receiving unit 811 receives three-dimensional data 831 from traffic cloud monitoring or the vehicle ahead. The three-dimensional data 831 includes information about areas that cannot be detected by the vehicle's sensor 815, such as point cloud data, visible light images, depth information, sensor position information, or speed information.

[0501] The communication unit 812 communicates with the traffic cloud monitoring or the vehicle ahead, and sends a data transmission request and the like to the traffic cloud monitoring or the vehicle ahead.

[0502] The reception control unit 813 exchanges information such as a corresponding format with the communication partner via the communication unit 812, and establishes communication with the communication partner.

[0503] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion or the like on the three-dimensional data 831 received by the data reception unit 811. Furthermore, when the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0504] The plurality of sensors 815 are a group of sensors such as LIDAR, visible light camera, or infrared camera that obtain information outside the vehicle 801 and generate sensor information 833. For example, when the sensor 815 is a laser sensor such as LIDAR, the sensor information 833 is three-dimensional data such as point cloud data (point group data). In addition, the number of sensors 815 may not be multiple.

[0505] The three-dimensional data generating unit 816 generates three-dimensional data 834 based on the sensor information 833. The three-dimensional data 834 includes, for example, point cloud data, visible light images, depth information, sensor position information, or speed information.

[0506] The three-dimensional data synthesis unit 817 synthesizes the three-dimensional data 832 produced by traffic cloud monitoring or the front vehicle, etc., into the three-dimensional data 834 produced based on the sensor information 833 of the own vehicle, thereby constructing three-dimensional data 835 that also includes the space in front of the front vehicle that cannot be detected by the sensor 815 of the own vehicle.

[0507] The three-dimensional data accumulation unit 818 accumulates the generated three-dimensional data 835 and the like.

[0508] The communication unit 819 communicates with the traffic cloud monitoring or the vehicle behind, and sends a data transmission request and the like to the traffic cloud monitoring or the vehicle behind.

[0509] The transmission control unit 820 exchanges information such as the corresponding format with the communication partner via the communication unit 819 to establish communication with the communication partner. In addition, the transmission control unit 820 determines the transmission area of ​​the space of the three-dimensional data to be transmitted based on the three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and the data transmission request from the communication partner.

[0510] Specifically, the transmission control unit 820 determines the transmission area including the space in front of the vehicle that cannot be detected by the sensor of the rear vehicle according to the data transmission request from the traffic cloud monitoring or the rear vehicle. In addition, the transmission control unit 820 determines the transmission area by judging whether the space that can be transmitted or the transmitted space has been updated based on the three-dimensional data construction information. For example, the transmission control unit 820 determines the area that is both specified by the data transmission request and the area where the corresponding three-dimensional data 835 exists as the transmission area. In addition, the transmission control unit 820 notifies the format corresponding to the communication partner and the transmission area to the format conversion unit 821.

[0511] The format conversion unit 821 generates three-dimensional data 837 by converting the three-dimensional data 836 of the transmission area in the three-dimensional data 835 accumulated in the three-dimensional data accumulation unit 818 into a format corresponding to the receiving side. In addition, the format conversion unit 821 may compress or encode the three-dimensional data 837 to reduce the data amount.

[0512] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic cloud monitoring or the rear vehicle. The three-dimensional data 837 includes, for example, information about the area that becomes a blind spot for the rear vehicle, such as point cloud data, visible light images, depth information, or sensor position information in front of the vehicle.

[0513] In addition, although the example in which the format conversion units 814 and 821 perform format conversion is described here, format conversion may not be performed.

[0514] With this configuration, the three-dimensional data production device 810 obtains three-dimensional data 831 of an area that cannot be detected by the sensor 815 of the own vehicle from the outside, and generates three-dimensional data 835 by synthesizing the three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the sensor 815 of the own vehicle. In this way, the three-dimensional data production device 810 can generate three-dimensional data of a range that cannot be detected by the sensor 815 of the own vehicle.

[0515] In addition, the three-dimensional data production device 810 can send three-dimensional data of the space in front of its own vehicle that cannot be detected by the sensors of the rear vehicle to the traffic cloud monitoring or the rear vehicle, etc. according to the data sending request from the traffic cloud monitoring or the rear vehicle.

[0516] Next, the procedure for transmitting three-dimensional data to the following vehicle in the three-dimensional data creation device 810 will be described. Fig.39 This is a flowchart showing an example of the procedure of sending three-dimensional data to traffic cloud monitoring or a following vehicle by the three-dimensional data production device 810.

[0517] First, the three-dimensional data production device 810 generates and updates three-dimensional data 835 of a space including a space 802 on the road ahead of the own vehicle 801 (S801). Specifically, the three-dimensional data production device 810 synthesizes the three-dimensional data 831 produced by traffic cloud monitoring or the vehicle ahead, etc., with the three-dimensional data 834 produced based on the sensor information 833 of the own vehicle 801, and by performing this synthesis, constructs the three-dimensional data 835 including the space ahead of the vehicle ahead that cannot be detected by the sensor 815 of the own vehicle.

[0518] Next, the three-dimensional data creating device 810 determines whether the three-dimensional data 835 included in the transmitted space has changed (S802).

[0519] When the three-dimensional data 835 contained in the space changes due to a vehicle or a person entering the sent space from the outside ("Yes" in S802), the three-dimensional data production device 810 sends the three-dimensional data including the three-dimensional data 835 of the changed space to the traffic cloud monitoring or the rear vehicle (S803).

[0520] In addition, although the three-dimensional data production device 810 can send the three-dimensional data of the space that has changed according to the transmission timing of the three-dimensional data at a predetermined interval, it can also send it immediately after the change is detected. That is, the three-dimensional data production device 810 can send the three-dimensional data of the space that has changed in priority to the three-dimensional data sent at a predetermined interval.

[0521] Furthermore, the three-dimensional data production device 810 may send all three-dimensional data of the changed space as the three-dimensional data of the changed space, or may send only the difference in the three-dimensional data (such as information on the appearance or disappearance of three-dimensional points, or information on the displacement of three-dimensional points, etc.).

[0522] Furthermore, the three-dimensional data production device 810 may send metadata related to the danger avoidance work of the own vehicle, such as an emergency brake warning, to the rear vehicle before sending the three-dimensional data of the changed space. Thus, the rear vehicle can confirm the emergency brake of the front vehicle in advance, so as to start the danger avoidance work such as deceleration as soon as possible.

[0523] When the three-dimensional data 835 contained in the sent space has not changed ("No" in S802) or after step S803, the three-dimensional data production device 810 sends the three-dimensional data contained in the space of a specified shape at a distance L in front of its own vehicle 801 to the traffic cloud monitoring or the rear vehicle (S804).

[0524] Furthermore, for example, the processing of steps S801 to S804 is repeatedly executed at predetermined time intervals.

[0525] Furthermore, when there is no difference between the three-dimensional data 835 of the space 802 currently being transmitted and the three-dimensional map, the three-dimensional data creating device 810 may not transmit the three-dimensional data 837 of the space 802 .

[0526] Fig.40 This is a flowchart showing the operation of the three-dimensional data creation device 810 in this case.

[0527] First, the three-dimensional data creation device 810 generates and updates the three-dimensional data 835 of the space including the space 802 on the road ahead of the own vehicle 801 (S811).

[0528] Next, the three-dimensional data production device 810 determines whether there is an update of the difference between the three-dimensional data 835 of the generated space 802 and the three-dimensional map (S812). That is, the three-dimensional data production device 810 determines whether there is a difference between the three-dimensional data 835 of the generated space 802 and the three-dimensional map. Here, the three-dimensional map is three-dimensional map information managed by a device on the infrastructure side such as traffic cloud monitoring. For example, the three-dimensional map can be obtained as three-dimensional data 831.

[0529] If there is an update ("Yes" in S812), the three-dimensional data production device 810 sends the three-dimensional data contained in the space 802 to the traffic cloud monitoring or the rear vehicle (S813) in the same manner as described above.

[0530] In addition, when there is no update (No in S812), the three-dimensional data production device 810 does not send the three-dimensional data contained in the space 802 to the traffic cloud monitoring and the rear vehicle (S814). In addition, the three-dimensional data production device 810 can control the three-dimensional data of the space 802 not to be sent by setting the volume of the space 802 to zero. In addition, the three-dimensional data production device 810 can also send information indicating that there is no update in the space 802 to the traffic cloud monitoring or the rear vehicle.

[0531] As described above, for example, when there is no obstacle on the road, there is no difference between the generated three-dimensional data 835 and the three-dimensional map on the infrastructure side, so the data is not transmitted. In this way, the transmission of unnecessary data can be suppressed.

[0532] In the above description, although the three-dimensional data creation device 810 is mounted on a vehicle as an example, the three-dimensional data creation device 810 is not limited to being mounted on a vehicle, and can be mounted on any mobile object.

[0533] As described above, the three-dimensional data creation device 810 involved in this embodiment is mounted on a mobile body having a sensor 815 and a communication unit (data receiving unit 811 or data sending unit 822, etc.) for sending and receiving three-dimensional data to and from the outside. The three-dimensional data creation device 810 creates three-dimensional data 835 (second three-dimensional data) based on sensor information 833 detected by the sensor 815 and three-dimensional data 831 (first three-dimensional data) received by the data receiving unit 811. The three-dimensional data creation device 810 sends three-dimensional data 837, which is a part of the three-dimensional data 835, to the outside.

[0534] Thus, the three-dimensional data generating device 810 can generate three-dimensional data of a range that cannot be detected by the own vehicle, and can send three-dimensional data of a range that cannot be detected by other vehicles to the other vehicles.

[0535] Furthermore, the three-dimensional data creation device 810 repeatedly creates three-dimensional data 835 and transmits three-dimensional data 837 at predetermined intervals. The three-dimensional data 837 is three-dimensional data of a small space 802 having a predetermined size located at a predetermined distance L ahead of the current position of the vehicle 801 in the moving direction of the vehicle 801.

[0536] According to this, since the range of the three-dimensional data 837 to be transmitted is limited, the data amount of the three-dimensional data 837 to be transmitted can be reduced.

[0537] Furthermore, the prescribed distance L changes according to the moving speed V of the vehicle 801. For example, the greater the moving speed V, the longer the prescribed distance L. Accordingly, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801, and can send the three-dimensional data 837 of the small space 802 to the rear vehicle, etc.

[0538] Furthermore, the prescribed size changes according to the moving speed V of the vehicle 801. For example, the greater the moving speed V, the greater the prescribed size. For example, the greater the moving speed V, the greater the depth D, which is the length of the small space 802 in the moving direction of the vehicle. Accordingly, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801, and can send the three-dimensional data 837 of the small space 802 to the rear vehicle, etc.

[0539] Furthermore, the three-dimensional data production device 810 determines whether there is a change in the three-dimensional data 835 of the small space 802 corresponding to the transmitted three-dimensional data 837. If the three-dimensional data production device 810 determines that there is a change, the three-dimensional data 837 (fourth three-dimensional data) which is at least a part of the changed three-dimensional data 835 is transmitted to an external rear vehicle or the like.

[0540] Accordingly, the vehicle 801 can transmit the three-dimensional data 837 of the changed space to the following vehicle and the like.

[0541] Furthermore, the three-dimensional data production device 810 sends the changed three-dimensional data 837 (the fourth three-dimensional data) in priority over the regular three-dimensional data 837 (the third three-dimensional data) that is sent periodically. Specifically, the three-dimensional data production device 810 sends the changed three-dimensional data 837 (the fourth three-dimensional data) before sending the regular three-dimensional data 837 (the third three-dimensional data) that is sent periodically. That is, the three-dimensional data production device 810 does not wait for the regular three-dimensional data 837 that is sent periodically to be sent, but sends the changed three-dimensional data 837 (the fourth three-dimensional data) non-periodically.

[0542] According to this, since the vehicle 801 can preferentially send the three-dimensional data 837 of the changed space to the rear vehicle, etc., the rear vehicle, etc. can quickly make a judgment based on the three-dimensional data.

[0543] Furthermore, the changed three-dimensional data 837 (fourth three-dimensional data) shows the difference between the three-dimensional data 835 of the small space 802 corresponding to the transmitted three-dimensional data 837 and the changed three-dimensional data 835. Thus, the amount of the transmitted three-dimensional data 837 can be reduced.

[0544] Furthermore, the three-dimensional data creation device 810 does not send the three-dimensional data 837 of the small space 802 when there is no difference between the three-dimensional data 837 of the small space 802 and the three-dimensional data 831 of the small space 802. Furthermore, the three-dimensional data creation device 810 may send information indicating that there is no difference between the three-dimensional data 837 of the small space 802 and the three-dimensional data 831 of the small space 802 to the outside.

[0545] According to this, since it is possible to suppress the transmission of unnecessary three-dimensional data 837, the data amount of the three-dimensional data 837 to be transmitted can be reduced.

[0546] (Implementation 6)

[0547] In the present embodiment, a display device and a display method for displaying information obtained from a three-dimensional map or the like, and a storage device and a storage method for a three-dimensional map or the like are described.

[0548] For the purpose of automatic driving of the vehicle or autonomous movement of the robot, a mobile body such as a vehicle or a robot flexibly uses a three-dimensional map obtained through communication with a server or other vehicles, a two-dimensional image obtained from a sensor mounted on the vehicle, or three-dimensional data detected by the vehicle. The data that the user wants to view or save varies depending on the situation. The following describes a display device that switches the display according to different situations.

[0549] Fig.41 1 is a flowchart showing an overview of a display method by a display device. The display device is mounted on a mobile body such as a vehicle or a robot. In addition, the following description is made using an example in which the mobile body is a vehicle (automobile).

[0550] First, the display device determines which of the two-dimensional surrounding information and the three-dimensional surrounding information to display according to the driving condition of the vehicle (S901). In addition, the two-dimensional surrounding information corresponds to the first surrounding information in the technical solution, and the three-dimensional surrounding information corresponds to the second surrounding information in the technical solution. Here, the surrounding information is information showing the surroundings of the moving body, such as an image when viewed from the vehicle in a specified direction, or a map of the surroundings of the vehicle.

[0551] The two-dimensional surrounding information is information generated using two-dimensional data. Here, the two-dimensional data refers to two-dimensional map information or images. For example, the two-dimensional surrounding information is a map of the vehicle's surroundings obtained from a two-dimensional map, or an image obtained by a camera mounted on the vehicle. Furthermore, the two-dimensional surrounding information, for example, does not include three-dimensional information. That is, when the two-dimensional surrounding information is a map of the vehicle's surroundings, the map does not include information in the height direction. Furthermore, when the two-dimensional surrounding information is an image obtained by a camera, the image does not include information in the depth direction.

[0552] Furthermore, the three-dimensional surrounding information is information generated using three-dimensional data. Here, the three-dimensional data is, for example, a three-dimensional map. In addition, the three-dimensional data may be information showing the three-dimensional position or three-dimensional shape of an object around the vehicle obtained from another vehicle or a server or detected by the vehicle itself. For example, the three-dimensional surrounding information is a two-dimensional or three-dimensional image or map around the vehicle generated using a three-dimensional map. Furthermore, the three-dimensional surrounding information includes, for example, three-dimensional information. For example, in the case where the three-dimensional surrounding information is an image in front of the vehicle, the image includes information showing the distance to the object in the image. Furthermore, in the image, for example, pedestrians blocked by the vehicle in front are displayed. Furthermore, in the three-dimensional surrounding information, the information showing these distances or pedestrians may be superimposed on the image obtained by the sensor mounted on the vehicle. Furthermore, in the three-dimensional surrounding information, the information in the height direction may also be superimposed on the two-dimensional map.

[0553] Furthermore, three-dimensional data can be displayed three-dimensionally, and a two-dimensional image or two-dimensional map obtained from the three-dimensional data can also be displayed on a two-dimensional display or the like.

[0554] In step S901, when it is determined to display the three-dimensional surrounding information ("Yes" in S902), the display device displays the three-dimensional surrounding information (S903). Also, in step S901, when it is determined to display the two-dimensional surrounding information ("No" in S902), the display device displays the two-dimensional surrounding information (S904). In this way, the display device displays the three-dimensional surrounding information or the two-dimensional surrounding information to be displayed as determined in step S901.

[0555] The following is an explanation of specific examples. In the first example, the display device switches the surrounding information to be displayed according to whether the vehicle is automatically driven or manually driven. Specifically, during automatic driving, since the driver does not need to know the detailed surrounding road information in detail, the display device displays two-dimensional surrounding information (such as a two-dimensional map). In addition, during manual driving, three-dimensional surrounding information (such as a three-dimensional map) is displayed for safe driving so that the details of the surrounding road information can be known.

[0556] Furthermore, during automatic driving, since the user is shown what information the vehicle is driving based on, the display device can display information that affects the driving operation (e.g., SWLD, lanes, road signs, and surrounding condition detection results used in estimating the vehicle's position). For example, the display device can add this information to a two-dimensional map.

[0557] In addition, the surrounding information displayed during the above-mentioned automatic driving and manual driving is only an example, and the display device can also display three-dimensional surrounding information during automatic driving and two-dimensional surrounding information during manual driving. Moreover, in at least one of the automatic driving and manual driving, in addition to the two-dimensional or three-dimensional map or image, the display device can also display metadata or surrounding condition detection results, and can display metadata or surrounding condition detection results instead of the two-dimensional or three-dimensional map or image. Here, metadata refers to information showing the three-dimensional position or three-dimensional shape of an object obtained from a server or other vehicles. Moreover, the surrounding condition detection result is information showing the three-dimensional position or three-dimensional shape of an object detected by the vehicle itself.

[0558] In the second example, the display device switches the surrounding information to be displayed according to the driving environment. For example, the display device switches the surrounding information to be displayed according to the brightness of the outside world. Specifically, when the surroundings of the own vehicle are bright, the display device displays a two-dimensional image obtained by a camera mounted on the own vehicle, or displays three-dimensional surrounding information produced using the two-dimensional image. In addition, when the surroundings of the own vehicle are dark, the display device displays three-dimensional surrounding information produced using a laser radar or a millimeter-wave radar because the two-dimensional image obtained by the camera mounted on the own vehicle is dark and difficult to see.

[0559] Furthermore, the display device may also switch the surrounding information to be displayed according to the area where the vehicle is currently located, i.e., the driving area. For example, the display device may display three-dimensional surrounding information in a manner that provides information about surrounding buildings in tourist spots, city centers, or near the destination to the user. In addition, the display device may display two-dimensional surrounding information in consideration of the fact that detailed information about the surrounding areas is not often required, such as in the mountains or in the suburbs.

[0560] Furthermore, the display device may also switch the surrounding information to be displayed according to the weather conditions. For example, in the case of sunny days, the display device displays the three-dimensional surrounding information produced by the camera or the laser radar. In addition, in the case of rainy days or dense fog, since the three-dimensional surrounding information produced by the camera or the laser radar is prone to contain noise, the display device displays the three-dimensional surrounding information produced by the millimeter wave radar.

[0561] Furthermore, the switching of these displays may be performed automatically by the system or manually by the user.

[0562] Furthermore, the three-dimensional surrounding information is generated based on one or more of the following data, namely: dense point group data generated based on WLD, grid data generated based on MWLD, sparse data generated based on SWLD, lane data generated based on lane world space, two-dimensional map data including three-dimensional shape information of roads and intersections, and metadata or vehicle detection results including three-dimensional position or three-dimensional shape information that changes according to actual time.

[0563] In addition, the above-mentioned WLD is three-dimensional point group data, and SWLD is data obtained by extracting a point group whose feature value is above a threshold from WLD. And MWLD is data with a mesh structure generated from WLD. Lane world space is data obtained by extracting a point group whose feature value is above a threshold and is required for self-position estimation, driving support, or autonomous driving from WLD.

[0564] Here, MWLD and SWLD have less data than WLD. Therefore, when more detailed data is required, WLD can be used, and in other cases, MWLD or SWLD can be used to appropriately reduce the amount of communication data and the amount of processing. In addition, the lane world space has less data than SWLD. Therefore, by using the lane world space, the amount of communication data and the amount of processing can be further reduced.

[0565] Furthermore, although the above description uses the switching between two-dimensional surrounding information and three-dimensional surrounding information as an example, the display device may also switch the type of data (WLD, SWLD, etc.) used when generating the three-dimensional surrounding information according to the above conditions. That is, in the above description, in the example where the display device displays three-dimensional surrounding information, the three-dimensional surrounding information generated based on the first data (such as WLD or SWLD) with a larger amount of data may be displayed, and in the example where the two-dimensional surrounding information is displayed, the three-dimensional surrounding information generated based on the second data (such as SWLD or lane world space) instead of the two-dimensional surrounding information may be displayed, and the second data is data with a smaller amount of data than the first data described above.

[0566] Furthermore, the display device displays the two-dimensional surrounding information or the three-dimensional surrounding information, for example, on a two-dimensional display, a head-up display, or a head-mounted display mounted on the vehicle itself. Alternatively, the display device may transmit the two-dimensional surrounding information or the three-dimensional surrounding information to a mobile terminal such as a smart phone through wireless communication and display it. That is, the display device is not limited to being mounted on a moving body, but can operate in coordination with the moving body. For example, when a user holding a display device such as a smart phone is riding on a moving body, or driving a moving body, information about the moving body such as the position of the moving body estimated based on the moving body's own position is displayed on the display device, or this information is displayed on the display device together with the surrounding information.

[0567] Furthermore, when displaying a three-dimensional map, the display device may render the three-dimensional map and display it as two-dimensional data, or may utilize a three-dimensional display or a three-dimensional hologram and display it as three-dimensional data.

[0568] Next, the method of saving the three-dimensional map is described. Since the moving body such as a vehicle or a robot is an automatic driving of the vehicle or the autonomous movement of the robot, the three-dimensional map obtained from the communication with the server or other vehicles, the two-dimensional image obtained by the sensor carried by the vehicle itself, or the three-dimensional data detected by the vehicle itself is used. It can be considered that among these data, the data that the user wants to view or save will be different depending on the situation. The following describes the method of saving data corresponding to the situation.

[0569] The storage device is mounted on a mobile body such as a vehicle or a robot. In the following description, an example in which the mobile body is a vehicle (automobile) is used. Furthermore, the storage device may also be included in the above-mentioned display device.

[0570] In the first example, the storage device determines whether to store the three-dimensional map based on the region. Here, by storing the three-dimensional map in the storage medium of the vehicle itself, autonomous driving can be performed within the storage space without communicating with the server. However, due to the limited storage capacity, only limited data can be stored. For this reason, the storage device is limited to the storage area shown below.

[0571] For example, the storage device may prioritize the three-dimensional map of frequently-used areas such as the route to work or the vicinity of one's home. As a result, data of frequently-used areas does not need to be acquired every time, thereby effectively reducing the amount of communication data. In addition, priority storage means that high-priority data is stored within a predetermined storage capacity. For example, when new data cannot be stored within the storage capacity, data with a lower priority than the new data is deleted.

[0572] Furthermore, the storage device preferentially stores the three-dimensional map of the area with poor communication environment. Thus, in the area with poor communication environment, since data does not need to be obtained through communication, the situation where the three-dimensional map cannot be obtained due to poor communication can be suppressed.

[0573] Alternatively, the storage device preferentially stores the three-dimensional map of the area with heavy traffic. Thus, the three-dimensional map of the area with frequent accidents can be preferentially stored. Thus, in such an area, the reduction in the accuracy of the automatic driving or driving assistance caused by the failure to obtain the three-dimensional map due to poor communication can be suppressed.

[0574] Alternatively, the storage device may prioritize the storage of three-dimensional maps of areas with less traffic. In areas with less traffic, the possibility that the automatic driving mode that automatically follows the vehicle ahead cannot be used increases. As a result, more detailed surrounding information may be required. Therefore, by prioritizing the storage of three-dimensional maps of areas with less traffic, the accuracy of automatic driving or driving assistance in such areas can be improved.

[0575] In addition, the above-mentioned multiple storage methods may be combined. Furthermore, the regions where the three-dimensional maps are preferentially stored may be automatically determined by the system or may be specified by the user.

[0576] Furthermore, the storage device can delete the three-dimensional map after a specified period of time has passed since the storage, or update it to the latest data. Accordingly, the old map data will not be used. Furthermore, when updating the map data, the storage device compares the old map with the new map to detect the difference spatial area with difference, that is, the difference area, and adds the data of the difference area of ​​the new map to the old map, or removes the data of the difference area from the old map, so that only the data of the changed area can be updated.

[0577] Furthermore, in this example, the stored three-dimensional map is used for autonomous driving. Therefore, by using SWLD as the three-dimensional map, the amount of communication data can be reduced. In addition, the three-dimensional map is not limited to SWLD, and can also be other types of data such as WLD.

[0578] In the second example, the saving means saves the three-dimensional map according to the event.

[0579] For example, the storage device stores a special event encountered during driving as a three-dimensional map. Based on this, the user can view and listen to the details of the event afterwards. The following shows an example of an event stored as a three-dimensional map. In addition, the storage device can also store three-dimensional surrounding information generated from the three-dimensional map.

[0580] For example, the storage device stores the three-dimensional map before and after a collision accident or when danger is sensed.

[0581] Alternatively, the storage device holds a three-dimensional map of a characteristic scene such as a beautiful scenery, a place where people gather, or a tourist attraction.

[0582] These events to be saved can be automatically determined by the system or specified in advance by the user. For example, machine learning can be used as a method for determining these events.

[0583] Furthermore, in this example, the stored three-dimensional map is used for viewing. Therefore, by using WLD as the three-dimensional map, high-quality images can be provided. In addition, the three-dimensional map is not limited to WLD, and can also be other types of data such as SWLD.

[0584] The following is a method for controlling the display of the display device according to the user. When the display device overlays the surrounding condition detection results obtained through inter-vehicle communication on the map and displays it, the surrounding vehicles are represented as wireframes or the surrounding vehicles are represented with transparency, so that the detection objects hidden by the surrounding vehicles can be seen. Alternatively, the display device can display the image seen from an overhead viewpoint, or the own vehicle, surrounding vehicles and the surrounding condition detection results can be seen from a bird's-eye view.

[0585] When using a head-up display to overlay the surrounding condition detection results or point group data on the transparent Fig.42 In the case of the surrounding environment viewed through the windshield shown, the position of the superimposed information may deviate due to differences in the user's posture, body shape, or eye position. Fig.43 An example of a display on the head-up display when a positional deviation occurs is shown.

[0586] To correct this deviation, the display device detects the user's posture, body shape, or eye position using information from a camera in the vehicle or a sensor mounted on a seat, and adjusts the position of the superimposed information according to the detected user's posture, body shape, or eye position. Fig.44 An example of a display on the head-up display after adjustment is shown.

[0587] In addition, such adjustment of the overlapping position may be performed manually by the user using a control device mounted on the vehicle.

[0588] Furthermore, the display device can display safe places in the event of a disaster on a map and prompt the user. Alternatively, the vehicle can convey the details of the disaster and a message to move to a safe place to the user, and perform automatic driving until reaching the safe place.

[0589] For example, if a vehicle sets a high altitude area that will not be affected by a tsunami when an earthquake occurs as a destination, the vehicle can communicate with the server to obtain information about roads that are difficult to pass due to the earthquake and handle the disaster content according to the route avoiding such roads.

[0590] In addition, autonomous driving can include multiple modes such as mobile mode and long-distance driving mode.

[0591] In mobile mode, the vehicle decides the route to the destination taking into account early arrival time, low cost, short driving distance, short travel distance, and low fuel consumption, and drives automatically according to the determined route.

[0592] In the driving travel mode, the vehicle automatically determines the route in such a way that it arrives at the destination at the time specified by the user. For example, when the user sets the destination and arrival time, the vehicle determines the route in such a way that it can travel around and arrive at the destination at the set time, and drives automatically according to the determined route.

[0593] (Implementation 7)

[0594] In the fifth embodiment, a client device such as a vehicle sends three-dimensional data to another vehicle or a server such as a traffic cloud monitoring device. In this embodiment, the client device sends sensor information obtained by the sensor to the server or other client devices.

[0595] First, the configuration of a system according to this embodiment will be described. Fig.45 1 shows the structure of the system for transmitting and receiving three-dimensional maps and sensor information according to the present embodiment. The system includes a server 901 and client devices 902A and 902B. In addition, when the client devices 902A and 902B are not specifically distinguished, they are also referred to as client devices 902.

[0596] The client device 902 is, for example, an in-vehicle device mounted on a mobile body such as a vehicle. The server 901 is, for example, a traffic cloud monitoring system, and can communicate with a plurality of client devices 902 .

[0597] The server 901 transmits a three-dimensional map composed of point cloud data to the client device 902. In addition, the composition of the three-dimensional map is not limited to point cloud data, and can also be expressed by other three-dimensional data such as a grid structure.

[0598] The client device 902 sends the sensor information obtained by the client device 902 to the server 901. The sensor information includes, for example, at least one of LIDAR information, visible light images, infrared images, depth images, sensor position information, and speed information.

[0599] The data sent and received between the server 901 and the client device 902 may be compressed when it is desired to reduce the data, and may not be compressed when it is desired to maintain the accuracy of the data. When compressing the data, a three-dimensional compression method based on an octree may be used in point cloud data, for example. In addition, a two-dimensional image compression method may be used in visible light images, infrared images, and depth images. The two-dimensional image compression method is, for example, MPEG-4AVC or HEVC standardized by MPEG.

[0600] Furthermore, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in accordance with the three-dimensional map transmission request from the client device 902. In addition, the server 901 may transmit the three-dimensional map without waiting for the three-dimensional map transmission request from the client device 902. For example, the server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, the server 901 may transmit the three-dimensional map suitable for the location of the client device 902 at regular intervals to the client device 902 that has received a transmission request once. Furthermore, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

[0601] The client device 902 sends a three-dimensional map transmission request to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 sends a three-dimensional map transmission request to the server 901.

[0602] In addition, in the following cases, the client device 902 may also issue a three-dimensional map transmission request to the server 901. In the case where the three-dimensional map held by the client device 902 is relatively old, the client device 902 may also issue a three-dimensional map transmission request to the server 901. For example, in the case where the client device 902 obtains the three-dimensional map and a certain period of time has passed, the client device 902 may also issue a three-dimensional map transmission request to the server 901.

[0603] Alternatively, the client device 902 may send a three-dimensional map transmission request to the server 901 before a certain time when the client device 902 is about to leave the space shown on the three-dimensional map held by the client device 902. For example, the client device 902 may send a three-dimensional map transmission request to the server 901 when the client device 902 is within a predetermined distance from the boundary of the space shown on the three-dimensional map held by the client device 902. Furthermore, when the moving path and moving speed of the client device 902 are known, the time when the client device 902 leaves the space shown on the three-dimensional map held by the client device 902 may be predicted based on the known moving path and moving speed.

[0604] When the error between the three-dimensional data generated by the client device 902 based on the sensor information and the position of the three-dimensional map is greater than a certain range, the client device 902 may send a request to the server 901 to send the three-dimensional map.

[0605] The client device 902 transmits the sensor information to the server 901 in accordance with the sensor information transmission request transmitted from the server 901. In addition, the client device 902 may transmit the sensor information to the server 901 without waiting for the sensor information transmission request from the server 901. For example, when the client device 902 has received a sensor information transmission request from the server 901 once, the client device 902 may periodically transmit the sensor information to the server 901 within a certain period. Furthermore, when the error between the three-dimensional data generated by the client device 902 based on the sensor information and the position of the three-dimensional map obtained from the server 901 is greater than a certain range, the client device 902 may determine that there is a possibility that the three-dimensional map around the client device 902 has changed, and transmit the determination result together with the sensor information to the server 901.

[0606] The server 901 issues a request to send sensor information to the client device 902. For example, the server 901 receives location information of the client device 902 such as GPS from the client device 902. When the server 901 determines that the client device 902 is close to a space with little information in the three-dimensional map managed by the server 901 based on the location information of the client device 902, the server 901 issues a request to send sensor information to the client device 902 in order to regenerate the three-dimensional map. In addition, the server 901 may issue a request to send sensor information when it is desired to update the three-dimensional map, when it is desired to check the road conditions such as when snow is accumulated or when a disaster occurs, when it is desired to check the congestion conditions or the accident conditions.

[0607] Furthermore, the client device 902 may set the data volume of the sensor information to be sent to the server 901 according to the communication state or the frequency band when receiving the sensor information sending request received from the server 901. Setting the data volume of the sensor information to be sent to the server 901 means, for example, increasing or decreasing the data itself or selecting an appropriate compression method.

[0608] Fig.46 is a block diagram showing an example of the configuration of the client device 902. The client device 902 receives a three-dimensional map composed of point cloud data or the like from the server 901, and estimates the position of the client device 902 itself based on the three-dimensional data produced based on the sensor information of the client device 902. Furthermore, the client device 902 transmits the obtained sensor information to the server 901.

[0609] The client device 902 includes: a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data production unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a sending control unit 1021, and a data sending unit 1022.

[0610] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including point cloud data such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0611] The communication unit 1012 communicates with the server 901 and transmits a data transmission request (for example, a transmission request of a three-dimensional map) and the like to the server 901 .

[0612] The reception control unit 1013 exchanges information such as the corresponding format with the communication partner via the communication unit 1012, and establishes communication with the communication partner.

[0613] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, the format conversion unit 1014 performs decompression or decoding processing when the three-dimensional map 1031 is compressed or encoded. In addition, the format conversion unit 1014 does not perform decompression or decoding processing when the three-dimensional map 1031 is non-compressed data.

[0614] The plurality of sensors 1015 are a group of sensors mounted on the client device 902 such as LIDAR, visible light camera, infrared camera, or depth sensor for obtaining information outside the vehicle, and generate sensor information 1033. For example, when the sensor 1015 is a laser sensor such as LIDAR, the sensor information 1033 is three-dimensional data such as point cloud data (point group data). In addition, the number of sensors 1015 may not be multiple.

[0615] The three-dimensional data generating unit 1016 generates three-dimensional data 1034 of the surroundings of the own vehicle based on the sensor information 1033. For example, the three-dimensional data generating unit 1016 generates point cloud data having color information of the surroundings of the own vehicle using information obtained by LIDAR and visible light images obtained by a visible light camera.

[0616] The three-dimensional image processing unit 1017 uses the received three-dimensional map 1032 such as point cloud data and the three-dimensional data 1034 of the surroundings of the own vehicle generated based on the sensor information 1033 to perform the own vehicle position estimation processing. In addition, the three-dimensional image processing unit 1017 may synthesize the three-dimensional map 1032 and the three-dimensional data 1034 to produce three-dimensional data 1035 of the surroundings of the own vehicle, and use the produced three-dimensional data 1035 to perform the own vehicle position estimation processing.

[0617] The three-dimensional data storage unit 1018 stores the three-dimensional map 1032 , the three-dimensional data 1034 , the three-dimensional data 1035 , and the like.

[0618] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format corresponding to the receiving side. In addition, the format conversion unit 1019 can reduce the amount of data by compressing or encoding the sensor information 1037. In addition, when format conversion is not required, the format conversion unit 1019 can omit the processing. In addition, the format conversion unit 1019 can control the amount of data to be transmitted according to the designation of the transmission range.

[0619] The communication unit 1020 communicates with the server 901 , and receives a data transmission request (a sensor information transmission request) and the like from the server 901 .

[0620] The transmission control unit 1021 exchanges information such as a corresponding format with the communication partner via the communication unit 1020, thereby establishing communication.

[0621] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information obtained by the plurality of sensors 1015, such as information obtained by LIDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0622] Next, the configuration of the server 901 will be described. Fig.47 1 is a block diagram showing an example of the configuration of the server 901. The server 901 receives sensor information sent from the client device 902, and creates three-dimensional data based on the received sensor information. The server 901 updates the three-dimensional map managed by the server 901 using the created three-dimensional data. Furthermore, the server 901 sends the updated three-dimensional map to the client device 902 in accordance with the three-dimensional map sending request from the client device 902.

[0623] The server 901 includes: a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data production unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a sending control unit 1121, and a data sending unit 1122.

[0624] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information obtained by LIDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0625] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a sensor information transmission request) and the like to the client device 902 .

[0626] The reception control unit 1113 exchanges information such as the corresponding format with the communication partner via the communication unit 1112, thereby establishing communication.

[0627] When the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. When the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0628] The 3D data generating unit 1116 generates 3D data 1134 of the surroundings of the client device 902 based on the sensor information 1132. For example, the 3D data generating unit 1116 generates point cloud data having color information of the surroundings of the client device 902 using information obtained by LIDAR and visible light images obtained by a visible light camera.

[0629] The three-dimensional data synthesis unit 1117 synthesizes the three-dimensional data 1134 generated based on the sensor information 1132 with the three-dimensional map 1135 managed by the server 901 , thereby updating the three-dimensional map 1135 .

[0630] The three-dimensional data storage unit 1118 stores the three-dimensional map 1135 and the like.

[0631] The format conversion unit 1119 generates the three-dimensional map 1031 by converting the three-dimensional map 1135 into a format corresponding to the receiving side. In addition, the format conversion unit 1119 can also reduce the amount of data by compressing or encoding the three-dimensional map 1135. In addition, when format conversion is not required, the format conversion unit 1119 can also omit the processing. In addition, the format conversion unit 1119 can control the amount of data sent according to the designation of the sending range.

[0632] The communication unit 1120 communicates with the client device 902 , and receives a data transmission request (a transmission request of a three-dimensional map) and the like from the client device 902 .

[0633] The transmission control unit 1121 exchanges information such as a corresponding format with the communication partner via the communication unit 1120 , thereby establishing communication.

[0634] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including point cloud data such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0635] Next, the workflow of the client device 902 will be described. Fig.48 1 is a flowchart showing the operation performed by the client device 902 when obtaining a three-dimensional map.

[0636] First, the client device 902 requests the server 901 to send a three-dimensional map (point cloud data, etc.) (S1001). At this time, the client device 902 also sends the location information of the client device 902 obtained by GPS, etc., and accordingly, the server 901 can be requested to send a three-dimensional map related to the location information.

[0637] Next, the client device 902 receives the three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate a non-compressed three-dimensional map (S1003).

[0638] Next, the client device 902 creates three-dimensional data 1034 of the surroundings of the client device 902 based on the sensor information 1033 obtained from the plurality of sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created based on the sensor information 1033 (S1005).

[0639] Fig.49 101 is a flowchart showing the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). The client device 902 that has received the transmission request transmits the sensor information 1037 to the server 901 (S1012). In addition, when the sensor information 1033 includes a plurality of information obtained by a plurality of sensors 1015, the client device 902 compresses each information in a compression method suitable for each information, thereby generating the sensor information 1037.

[0640] Next, the workflow of the server 901 is described. Fig.50 1 is a flowchart showing the operation of the server 901 when acquiring sensor information. First, the server 901 requests the client device 902 to send sensor information (S1021). Next, the server 901 receives the sensor information 1037 sent from the client device 902 in accordance with the request (S1022). Next, the server 901 uses the received sensor information 1037 to create three-dimensional data 1134 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 on the three-dimensional map 1135 (S1024).

[0641] Fig.511 is a flowchart showing the work performed by the server 901 when sending a three-dimensional map. First, the server 901 receives a request to send a three-dimensional map from the client device 902 (S1031). The server 901 that has received the request to send a three-dimensional map sends the three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 can extract a three-dimensional map in the vicinity corresponding to the location information of the client device 902, and send the extracted three-dimensional map. Furthermore, the server 901 can compress the three-dimensional map composed of point cloud data, for example, using an octree compression method, and send the compressed three-dimensional map.

[0642] Hereinafter, modified examples of the present embodiment will be described.

[0643] The server 901 uses the sensor information 1037 received from the client device 902 to create three-dimensional data 1134 near the location of the client device 902. Next, the server 901 matches the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by the server 901, and calculates the difference between the three-dimensional data 1134 and the three-dimensional map 1135. When the difference is greater than a predetermined threshold, the server 901 determines that some abnormality has occurred around the client device 902. For example, when the ground subsidence occurs due to a natural disaster such as an earthquake, it can be considered that there will be a large difference between the three-dimensional map 1135 managed by the server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0644] The sensor information 1037 may also include at least one of the type of sensor, the performance of the sensor, and the model of the sensor. In addition, a category ID corresponding to the performance of the sensor may be added to the sensor information 1037. For example, when the sensor information 1037 is information obtained by LIDAR, an identifier may be assigned in consideration of the performance of the sensor, for example, category 1 may be assigned to a sensor that can obtain information with an accuracy of several mm, category 2 may be assigned to a sensor that can obtain information with an accuracy of several cm, and category 3 may be assigned to a sensor that can obtain information with an accuracy of several m. In addition, the server 901 may estimate the performance information of the sensor from the model of the client device 902. For example, when the client device 902 is mounted on a vehicle, the server 901 may determine the specification information of the sensor according to the model of the vehicle. In this case, the server 901 may obtain the information of the model of the vehicle in advance, or include the information in the sensor information. In addition, the server 901 may use the obtained sensor information 1037 to switch the degree of correction of the three-dimensional data 1134 produced using the sensor information 1037. For example, when the sensor performance is high accuracy (category 1), the server 901 does not perform correction on the three-dimensional data 1134. When the sensor performance is low accuracy (category 3), the server 901 applies correction suitable for the accuracy of the sensor to the three-dimensional data 1134. For example, the server 901 increases the degree (intensity) of correction as the accuracy of the sensor is lower.

[0645] The server 901 may also simultaneously send a request to send sensor information to multiple client devices 902 existing in a certain space. When the server 901 receives multiple sensor information from multiple client devices 902, it is not necessary to use all the sensor information to create the three-dimensional data 1134. For example, the server 901 may select the sensor information to be used according to the performance of the sensor. For example, when the server 901 updates the three-dimensional map 1135, it may select high-precision sensor information (category 1) from the received multiple sensor information and use the selected sensor information to create the three-dimensional data 1134.

[0646] The server 901 is not limited to servers such as traffic cloud monitoring, but can also be other client devices (car-mounted). Fig.52 The system configuration in this case is shown.

[0647] For example, the client device 902C sends a request to send sensor information to the client device 902A that is nearby, and obtains the sensor information from the client device 902A. Then, the client device 902C uses the obtained sensor information of the client device 902A to create three-dimensional data, and updates the three-dimensional map of the client device 902C. In this way, the client device 902C can utilize the performance of the client device 902C to generate a three-dimensional map of the space that can be obtained from the client device 902A. For example, when the performance of the client device 902C is high, this situation can be considered to occur.

[0648] In this case, the client device 902A that has provided the sensor information is given the right to obtain the high-precision three-dimensional map generated by the client device 902C. The client device 902A receives the high-precision three-dimensional map from the client device 902C in accordance with the right.

[0649] Furthermore, the client device 902C may send a request to send sensor information to multiple client devices 902 (client device 902A and client device 902B) in the vicinity. If the sensor of the client device 902A or the client device 902B is high-performance, the client device 902C can use the sensor information obtained by the high-performance sensor to create three-dimensional data.

[0650] Fig.53 1 is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes a 3D map and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0651] The client device 902 includes: a three-dimensional map decoding processing unit 1211, and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives the coded data of the compressed three-dimensional map, decodes the coded data and obtains the three-dimensional map. The sensor information compression processing unit 1212 does not compress the three-dimensional data produced by the obtained sensor information, but compresses the sensor information itself, and sends the compressed coded data of the sensor information to the server 901. According to this structure, the client device 902 can keep the processing unit (device or LSI) for decoding the three-dimensional map (point cloud data, etc.) inside, without having to keep the processing unit for compressing the three-dimensional data of the three-dimensional map (point cloud data, etc.) inside. In this way, the cost and power consumption of the client device 902 can be suppressed.

[0652] As described above, the client device 902 involved in this embodiment is mounted on the mobile body, and generates three-dimensional data 1034 of the surroundings of the mobile body based on the sensor information 1033 indicating the surrounding conditions of the mobile body obtained by the sensor 1015 mounted on the mobile body. The client device 902 estimates the own position of the mobile body using the generated three-dimensional data 1034. The client device 902 transmits the obtained sensor information 1033 to the server 901 or another mobile body 902.

[0653] Based on this, the client device 902 transmits the sensor information 1033 to the server 901 or the like. In this way, the amount of data to be transmitted may be reduced compared to the case of transmitting three-dimensional data. Furthermore, since it is not necessary to perform processing such as compression or encoding of three-dimensional data on the client device 902, the amount of processing on the client device 902 can be reduced. Therefore, the client device 902 can reduce the amount of data to be transmitted or simplify the configuration of the device.

[0654] Furthermore, the client device 902 further sends a request to send a three-dimensional map to the server 901, and receives the three-dimensional map 1031 from the server 901. The client device 902 estimates its own position using the three-dimensional data 1034 and the three-dimensional map 1032.

[0655] Furthermore, the sensor information 1033 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0656] Also, the sensor information 1033 includes information showing the performance of the sensor.

[0657] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile body 902. Thus, the client device 902 can reduce the amount of data to be transmitted.

[0658] For example, the client device 902 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0659] Furthermore, the server 901 according to the present embodiment can communicate with the client device 902 mounted on the mobile body, and receive sensor information 1037 indicating the surrounding conditions of the mobile body obtained by the sensor 1015 mounted on the mobile body from the client device 902. The server 901 generates three-dimensional data 1134 of the surroundings of the mobile body based on the received sensor information 1037.

[0660] Accordingly, the server 901 uses the sensor information 1037 sent from the client device 902 to create the three-dimensional data 1134. In this way, compared with the case where the client device 902 sends the three-dimensional data, it is possible to reduce the amount of data to be sent. In addition, since it is not necessary to perform processing such as compression or encoding of the three-dimensional data on the client device 902, the processing amount of the client device 902 can be reduced. In this way, the server 901 can reduce the amount of data to be transmitted or simplify the configuration of the device.

[0661] Furthermore, the server 901 further transmits a request for transmitting sensor information to the client device 902 .

[0662] Furthermore, the server 901 further updates the three-dimensional map 1135 using the created three-dimensional data 1134 , and transmits the three-dimensional map 1135 to the client device 902 in response to a transmission request for the three-dimensional map 1135 from the client device 902 .

[0663] Furthermore, the sensor information 1037 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0664] Also, the sensor information 1037 includes information showing the performance of the sensor.

[0665] Furthermore, the server 901 further calibrates the three-dimensional data according to the performance of the sensor. Accordingly, the three-dimensional data production method can improve the quality of the three-dimensional data.

[0666] Furthermore, when receiving sensor information, the server 901 receives a plurality of sensor information 1037 from a plurality of client devices 902, and selects the sensor information 1037 used in the production of the three-dimensional data 1134 based on a plurality of information indicating the performance of the sensors included in the plurality of sensor information 1037. In this way, the server 901 can improve the quality of the three-dimensional data 1134.

[0667] Furthermore, the server 901 decodes or decompresses the received sensor information 1137, and creates three-dimensional data 1134 based on the decoded or decompressed sensor information 1132. In this way, the server 901 can reduce the amount of data to be transmitted.

[0668] For example, the server 901 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0669] (Implementation 8)

[0670] In this embodiment, a modification example of the above-mentioned Embodiment 7 is described. Fig.54 It is a diagram showing the configuration of a system according to this embodiment. Fig.54 The system shown includes a server 2001, a client device 2002A, and a client device 2002B.

[0671] The client device 2002A and the client device 2002B are mounted on a mobile object such as a vehicle, and transmit sensor information to the server 2001. The server 2001 transmits a three-dimensional map (point cloud data) to the client device 2002A and the client device 2002B.

[0672] The client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission possibility determination unit 2013. The configuration of the client device 2002B is the same. In the following, when the client device 2002A and the client device 2002B are not particularly distinguished, they are also referred to as the client device 2002.

[0673] Fig.55 This is a flowchart showing the operation of the client device 2002 according to this embodiment.

[0674] The sensor information acquisition unit 2011 obtains various sensor information using sensors (sensor groups) mounted on the mobile body. That is, the sensor information acquisition unit 2011 obtains sensor information obtained by the sensors (sensor groups) mounted on the mobile body, which indicates the surrounding conditions of the mobile body. In addition, the sensor information acquisition unit 2011 stores the obtained sensor information in the storage unit 2012. The sensor information includes at least one of LiDAR acquisition information, visible light images, infrared images, and depth images. In addition, the sensor information may also include at least one of sensor position information, speed information, acquisition time information, and acquisition location information. The sensor position information indicates the position of the sensor that obtained the sensor information. The speed information indicates the speed of the mobile body when the sensor obtained the sensor information. The acquisition time information indicates the time when the sensor information was obtained by the sensor. The acquisition location information indicates the position of the mobile body or the sensor when the sensor information was obtained by the sensor.

[0675] Next, the data transmission possibility determination unit 2013 determines whether the mobile body (client device 2002) is in an environment where the sensor information can be transmitted to the server 2001 (S2002). For example, the data transmission possibility determination unit 2013 may determine the location and time of the client device 2002 using information such as GPS, and determine whether data can be transmitted. In addition, the data transmission possibility determination unit 2013 may determine whether data can be transmitted by determining whether it can connect to a specific access point.

[0676] When the client device 2002 determines that the mobile body exists in an environment where the sensor information can be sent to the server 2001 ("Yes" in S2002), the sensor information is sent to the server 2001 (S2003). That is, at the time when the client device 2002 becomes in a state where the sensor information can be sent to the server 2001, the client device 2002 sends the sensor information it holds to the server 2001. For example, a millimeter wave access point capable of high-speed communication is set at an intersection or the like. At the time when the client device 2002 enters the intersection, the sensor information held by the client device 2002 is sent to the server 2001 at high speed using millimeter wave communication.

[0677] Next, the client device 2002 deletes the sensor information sent to the server 2001 from the storage unit 2012 (S2004). In addition, the client device 2002 may delete the sensor information when the sensor information sent to the server 2001 does not meet the prescribed conditions. For example, the client device 2002 may delete the sensor information from the storage unit 2012 when the time at which the sensor information is obtained is earlier than a certain time from the current time. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the time at which the sensor information is obtained by the sensor and the current time exceeds a preset time. In addition, the client device 2002 may delete the sensor information from the storage unit 2012 when the distance between the place at which the sensor information is obtained and the current location is greater than a certain distance. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the position of the moving body or sensor when the sensor information is obtained by the sensor and the current position of the moving body or sensor exceeds a preset distance. This makes it possible to reduce the capacity of the storage unit 2012 of the client device 2002 .

[0678] If the acquisition of sensor information by the client device 2002 has not been completed (No in S2005), the client device 2002 performs the processing after step S2001 again. If the acquisition of sensor information by the client device 2002 has been completed (Yes in S2005), the client device 2002 ends the processing.

[0679] In addition, the client device 2002 may also select sensor information to be sent to the server 2001 according to the communication conditions. For example, when high-speed communication is possible, the client device 2002 will preferentially send sensor information of large size (such as LiDAR acquisition information, etc.) stored in the storage unit 2012. In addition, when high-speed communication is difficult, the client device 2002 will send sensor information of small size and high priority (such as visible light images) stored in the storage unit 2012. Thus, the client device 2002 can effectively send the sensor information stored in the storage unit 2012 to the server 2001 according to the network conditions.

[0680] In addition, the client device 2002 may also obtain the time information indicating the current time and the location information indicating the current location from the server 2001. In addition, the client device 2002 may also determine the acquisition time and acquisition location of the sensor information based on the obtained time information and location information. That is, the client device 2002 may also obtain the time information from the server 2001 and generate the acquisition time information using the obtained time information. In addition, the client device 2002 may also obtain the location information from the server 2001 and generate the acquisition location information using the obtained location information.

[0681] For example, regarding time information, the server 2001 and the client device 2002 use mechanisms such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) to synchronize time. As a result, the client device 2002 can obtain correct time information. In addition, since the time can be synchronized between the server 2001 and multiple client devices, the time in the sensor information obtained by different client devices 2002 can be synchronized. Therefore, the server 2001 can handle sensor information representing the synchronized time. In addition, the mechanism for time synchronization can also be any method other than NTP or PTP. In addition, GPS information can also be used as the above-mentioned time information and location information.

[0682] The server 2001 may also obtain sensor information from multiple client devices 2002 by specifying a time or a place. For example, when a certain accident occurs, in order to find a client in the vicinity, the server 2001 specifies the time and place of the accident and broadcasts a sensor information transmission request to multiple client devices 2002. Then, the client device 2002 having the sensor information of the corresponding time and place transmits the sensor information to the server 2001. That is, the client device 2002 receives the sensor information transmission request including the designated information of the designated place and time from the server 2001. When the client device 2002 stores the sensor information obtained at the place and time indicated by the designated information in the storage unit 2012 and determines that the mobile body exists in the environment where the sensor information can be transmitted to the server 2001, the sensor information obtained at the place and time indicated by the designated information is transmitted to the server 2001. In this way, the server 2001 can obtain sensor information related to the occurrence of the accident from multiple client devices 2002 and use it for accident analysis, etc.

[0683] In addition, the client device 2002 may refuse to send the sensor information when receiving the sensor information sending request from the server 2001. In addition, the client device 2002 may set in advance which sensor information can be sent among the multiple sensor information. Alternatively, the server 2001 may ask the client device 2002 whether the sensor information can be sent each time.

[0684] In addition, a score may also be assigned to the client device 2002 that has sent sensor information to the server 2001. The score may be used, for example, to pay for gasoline purchase fees, EV (Electric Vehicle) charging fees, highway tolls, or car rental fees. In addition, after obtaining the sensor information, the server 2001 may delete the information of the client device 2002 used to determine the source of the sensor information. For example, the information is information such as the network address of the client device 2002. In this way, the sensor information can be anonymized, so the user of the client device 2002 can safely send the sensor information from the client device 2002 to the server 2001. In addition, the server 2001 may also be composed of multiple servers. For example, by sharing sensor information by multiple servers, even if a server fails, other servers can communicate with the client device 2002. In this way, the service stop caused by server failure can be avoided.

[0685] In addition, the designated place specified by the sensor information transmission request indicates the location where the accident occurred, etc., and is sometimes different from the location of the client device 2002 at the designated time specified by the sensor information transmission request. Therefore, the server 2001 can, for example, request information acquisition from the client device 2002 that exists within the range by specifying a range such as within XXm around as the designated place. The same is true for the designated time. The server 2001 can also specify a range such as within N seconds before and after a certain time. Thus, the server 2001 can obtain sensor information from the client device 2002 that exists in the "location: within XXm from the absolute position S" at "time: t-N to t+N". The client device 2002 can also send data generated immediately after time t when sending three-dimensional data such as LiDAR.

[0686] In addition, the server 2001 may separately designate information indicating the location of the client device 2002 from which the sensor information is to be obtained and the location where the sensor information is to be obtained as designated locations. For example, the server 2001 specifies that sensor information at least including a range of YYm from the absolute position S is to be obtained from the client device 2002 that is within XXm from the absolute position S. When the client device 2002 selects the three-dimensional data to be sent, it selects one or more randomly accessible units of three-dimensional data in a manner that includes at least the sensor information of the designated range. In addition, when the client device 2002 sends a visible light image, it may also send a plurality of image data that are continuous in time and include at least a frame immediately before or immediately after the time t.

[0687] In the case where the client device 2002 can utilize multiple physical networks such as 5G or WiFi, or multiple modes of 5G in sending sensor information, the client device 2002 may also select the network to be used according to the priority notified from the server 2001. Alternatively, the client device 2002 itself may select a network that can ensure an appropriate bandwidth based on the size of the data to be sent. Alternatively, the client device 2002 may also select the network to be used based on the cost spent on data transmission, etc. In addition, the sending request from the server 2001 may also include information indicating a sending deadline, such as sending when the client device 2002 can start sending before time T. If the server 2001 cannot obtain sufficient sensor information within the deadline, it may also issue a sending request again.

[0688] The sensor information may also include header information indicating the characteristics of the sensor data together with the compressed or uncompressed sensor data. The client device 2002 may also send the header information to the server 2001 via a physical network or communication protocol different from the sensor data. For example, the client device 2002 may also send the header information to the server 2001 before sending the sensor data. The server 2001 determines whether to obtain the sensor data of the client device 2002 based on the analysis result of the header information. For example, the header information may also include information indicating the density, elevation angle, or frame rate of the point group obtained by LiDAR, or the resolution, SN ratio, or frame rate of the visible light image. Thus, the server 2001 can obtain the sensor information from the client device 2002 having the sensor data of the determined quality.

[0689] As described above, the client device 2002 is mounted on the mobile body, obtains sensor information indicating the surrounding conditions of the mobile body obtained by the sensor mounted on the mobile body, and stores the sensor information in the storage unit 2012. The client device 2002 determines whether the mobile body exists in an environment where the sensor information can be sent to the server 2001, and sends the sensor information to the server 2001 if it is determined that the mobile body exists in an environment where the sensor information can be sent to the server.

[0690] Furthermore, the client device 2002 creates three-dimensional data of the surroundings of the moving object based on the sensor information, and estimates the own position of the moving object using the created three-dimensional data.

[0691] Furthermore, the client device 2002 further transmits a request to transmit the three-dimensional map to the server 2001, and receives the three-dimensional map from the server 2001. The client device 2002 estimates its own position by using the three-dimensional data and the three-dimensional map.

[0692] In addition, the processing performed by the above-mentioned client device 2002 can also be implemented as an information sending method in the client device 2002.

[0693] Furthermore, the client device 2002 may include a processor and a memory, and the processor may perform the above-mentioned processing using the memory.

[0694] Next, the sensor information collection system according to the present embodiment will be described. Fig.56 FIG. 1 is a diagram showing the structure of the sensor information collection system according to the present embodiment. Fig.56As shown, the sensor information collection system involved in this embodiment includes a terminal 2021A, a terminal 2021B, a communication device 2022A, a communication device 2022B, a network 2023, a data collection server 2024, a map server 2025, and a client device 2026. In addition, when the terminal 2021A and the terminal 2021B are not particularly distinguished, they are also recorded as the terminal 2021. When the communication device 2022A and the communication device 2022B are not particularly distinguished, they are also recorded as the communication device 2022.

[0695] The data collection server 2024 collects data such as sensor data obtained by the sensor included in the terminal 2021 as position-related data associated with positions in a three-dimensional space.

[0696] Sensor data refers to data obtained using sensors provided by the terminal 2021, such as the state of the surroundings of the terminal 2021 or the state of the inside of the terminal 2021. The terminal 2021 sends sensor data collected from one or more sensor devices at a location where the terminal 2021 can communicate directly or through one or more relay devices via the same communication method to the data collection server 2024.

[0697] The data included in the location-related data may also include, for example, information indicating the operation status of the terminal itself or a device possessed by the terminal, an operation log, the utilization status of a service, etc. In addition, the data included in the location-related data may also include information that establishes a correspondence between the identifier of the terminal 2021 and the location or movement path of the terminal 2021.

[0698] The information indicating the position included in the position-related data is associated with the information indicating the position in three-dimensional data such as three-dimensional map data. The details of the information indicating the position will be described later.

[0699] The location-related data may include at least one of the above-mentioned time information and information indicating the attributes of the data included in the location-related data or the type (e.g., model, etc.) of the sensor that generated the data, in addition to the location information that is information indicating the location. The location information and the time information may also be stored in the header area of ​​the location-related data or the header area of ​​the frame storing the location-related data. In addition, the location information and the time information may also be sent and / or stored separately from the location-related data as metadata associated with the location-related data.

[0700] The map server 2025 is connected to the network 2023, for example, and transmits three-dimensional data such as three-dimensional map data in response to a request from other devices such as the terminal 2021. In addition, as described in the aforementioned embodiments, the map server 2025 may also have a function of updating the three-dimensional data using sensor information transmitted from the terminal 2021.

[0701] The data collection server 2024 is connected to the network 2023, for example, to collect location-related data from other devices such as the terminal 2021, and store the collected location-related data in a storage device inside or in another server. In addition, the data collection server 2024 sends the collected location-related data or metadata of three-dimensional map data generated based on the location-related data to the terminal 2021 in accordance with a request from the terminal 2021.

[0702] The network 2023 is a communication network such as the Internet. The terminal 2021 is connected to the network 2023 via the communication device 2022. The communication device 2022 communicates with the terminal 2021 by one communication method or by switching between multiple communication methods. The communication device 2022 is, for example, (1) a base station of LTE (Long Term Evolution), (2) an access point (AP) of WiFi or millimeter wave communication, (3) a gateway of an LPWA (Low Power Wide Area) network such as SIGFOX, LoRaWAN or Wi-SUN, or (4) a communication satellite that communicates using a satellite communication method such as DVB-S2.

[0703] In addition, the base station may communicate with the terminal 2021 using a LPWA method classified as NB-IoT (Narrow Band-IoT) or LTE-M, or may communicate with the terminal 2021 while switching between these methods.

[0704] Here, the case where the terminal 2021 has the function of communicating with the communication device 2022 using two communication methods, and uses one of these communication methods, or switches between these multiple communication methods and the communication device 2022 as the direct communication partner while communicating with the map server 2025 or the data collection server 2024 is cited as an example, but the configuration of the sensor information collection system and the terminal 2021 is not limited to this. For example, the terminal 2021 may not have the communication function under multiple communication methods, but may have the function of communicating in a certain communication method. In addition, the terminal 2021 may also correspond to more than three communication methods. In addition, the corresponding communication method may be different for each terminal 2021.

[0705] The terminal 2021 has, for example, Fig.46 The configuration of the client device 902 is shown. The terminal 2021 estimates the position of its own position etc. using the received three-dimensional data. In addition, the terminal 2021 associates the sensor data obtained from the sensor with the position information obtained by the position estimation process to generate position-related data.

[0706] The location information added to the location-related data indicates, for example, the location in the coordinate system used in the three-dimensional data. For example, the location information is a coordinate value represented by the values ​​of latitude and longitude. In this case, the terminal 2021 may also include, in the location information, the coordinate system that serves as the basis for the coordinate value and information representing the three-dimensional data used in the location estimation. In addition, the coordinate value may also include information on the altitude.

[0707] In addition, the position information may also correspond to a unit of data or a unit of space that can be used in the encoding of the aforementioned three-dimensional data. The unit may be, for example, WLD, GOS, SPC, VLM, or VXL. In this case, the position information may be expressed, for example, by an identifier of a data unit such as SPC that is used to determine the position-associated data. In addition, in addition to the identifier of the data unit such as SPC that is used to determine the position information, the position information may also include information indicating the three-dimensional data obtained by encoding the three-dimensional space including the data unit such as the SPC, or information indicating the detailed position within the SPC. The information indicating the three-dimensional data may be, for example, the file name of the three-dimensional data.

[0708] In this way, the system generates position-related data corresponding to position information based on position estimation using three-dimensional data, and can provide sensor information with more accurate position information than when sensor information is attached with position information based on the client device (terminal 2021)'s own position obtained using GPS. As a result, when other devices use the position-related data in other services, they can also more accurately determine the position corresponding to the position-related data in real space by performing position estimation based on the same three-dimensional data.

[0709] In addition, in this embodiment, the case where the data sent from the terminal 2021 is position-related data is described as an example, but the data sent from the terminal 2021 may also be data that is not associated with position information. That is, the three-dimensional data or sensor data described in other embodiments may also be sent and received via the network 2023 described in this embodiment.

[0710] Next, different examples of position information representing a position in a three-dimensional or two-dimensional real space or map space are described. The position information added to the position-related data may also be information representing a relative position relative to a feature point in the three-dimensional data. Here, the feature point serving as a reference for the position information is, for example, a feature point encoded as SWLD and notified to the terminal 2021 as three-dimensional data.

[0711] The information indicating the relative position relative to the feature point may be, for example, also represented by a vector from the feature point to the point represented by the position information, and is information indicating the direction and distance from the feature point to the point represented by the position information. Alternatively, the information indicating the relative position relative to the feature point may also be information indicating the displacement of the X-axis, Y-axis, and Z-axis from the feature point to the point represented by the position information. In addition, the information indicating the relative position relative to the feature point may also be information indicating the distance from each of three or more feature points to the point represented by the position information. In addition, the relative position may not be the relative position of the point represented by the position information expressed with respect to each feature point as a reference, but the relative position of each feature point expressed with respect to the point represented by the position information as a reference. An example of position information based on the relative position relative to the feature point includes information for determining the feature point used as a reference, and information indicating the relative position of the point represented by the position information relative to the feature point. In addition, when the information representing the relative position relative to the feature point is provided separately from the three-dimensional data, the information representing the relative position relative to the feature point may also include information representing the coordinate axes used in deriving the relative position, information representing the type of three-dimensional data, and / or information representing the size per unit amount (scale, etc.) of the value of the information representing the relative position.

[0712] In addition, the location information may also include information indicating the relative position of each feature point for multiple feature points. In the case where the location information is indicated by the relative position relative to multiple feature points, the terminal 2021 that wants to determine the position indicated by the location information in the real space can also calculate the candidate point of the position indicated by the location information based on the position of the feature point estimated based on the sensor data for each feature point, and determine the point obtained by averaging the calculated multiple candidate points as the point indicated by the location information. According to this configuration, the influence of the error when estimating the position of the feature point based on the sensor data can be reduced, so the estimation accuracy of the point indicated by the location information in the real space can be improved. In addition, in the case where the location information includes information indicating the relative position relative to multiple feature points, even if there are feature points that cannot be detected due to the type or performance of the sensor possessed by the terminal 2021, as long as one of the multiple feature points can be detected, the value of the point indicated by the location information can be estimated.

[0713] As feature points, points that can be identified based on sensor data can be used. Points that can be identified based on sensor data are, for example, points or points within an area that satisfy the conditions for feature point detection such as the three-dimensional feature quantity or the feature quantity of visible light data being above a threshold.

[0714] In addition, a marker set in real space may be used as a feature point. In this case, the marker can be detected and its position can be determined based on data obtained by a sensor such as a LiDER or a camera. For example, the marker may be represented by a change in color or brightness value (reflectivity) or a three-dimensional shape (concave and convex, etc.). In addition, a coordinate value indicating the position of the marker or a QR code or barcode generated based on the identifier of the marker may be used.

[0715] In addition, a light source that transmits an optical signal may be used as a marker. When a light source that transmits an optical signal is used as a marker, not only information such as coordinate values ​​or identifiers used to obtain a location is transmitted, but other data may also be transmitted using an optical signal. For example, an optical signal may also include the content of a service corresponding to the location of the marker, an address such as a URL used to obtain the content, or an identifier of a wireless communication device used to receive the provision of the service, information indicating a wireless communication method used to connect to the wireless communication device, and the like. By using an optical communication device (light source) as a marker, the transmission of data other than information indicating a location becomes easy, and the data can be switched dynamically.

[0716] The terminal 2021 grasps the correspondence between feature points of different data using, for example, an identifier commonly used between data or information or a table indicating the correspondence between feature points between data. In addition, in the absence of information indicating the correspondence between feature points, the terminal 2021 may also determine the feature points at the closest distance when the coordinates of the feature points in the three-dimensional data of one party are transformed into the positions in the three-dimensional data space of the other party as corresponding feature points.

[0717] When the position information based on relative position described above is used, even between terminals 2021 or services using different three-dimensional data, the position represented by the position information can be determined or estimated based on the common feature points included in or corresponding to each three-dimensional data. As a result, the same position can be determined or estimated with higher accuracy between terminals 2021 or services using different three-dimensional data.

[0718] Furthermore, even when using map data or three-dimensional data expressed using different coordinate systems, the influence of errors associated with coordinate system transformation can be reduced, thereby enabling collaboration of services based on more accurate position information.

[0719] The following describes an example of the function provided by the data collection server 2024. The data collection server 2024 may also transmit the received location-related data to other data servers. In the case of multiple data servers, the data collection server 2024 determines to which data server the received location-related data is to be transmitted, and transmits the location-related data to the data server determined as the transmission destination.

[0720] The data collection server 2024 determines the transfer destination based on, for example, a transfer destination server determination rule set in advance in the data collection server 2024. The transfer destination server determination rule is set, for example, using a transfer destination table that associates an identifier corresponding to each terminal 2021 with a data server of the transfer destination.

[0721] The terminal 2021 adds an identifier corresponding to the terminal 2021 to the transmitted location-related data, and sends it to the data collection server 2024. The data collection server 2024 determines the data server of the transmission destination corresponding to the identifier attached to the location-related data based on the determination rule of the transmission destination server using the transmission destination table, and sends the location-related data to the determined data server. In addition, the determination rule of the transmission destination server can also be specified by using the determination conditions such as the time or place at which the location-related data is obtained. Here, the identifier corresponding to the terminal 2021 of the transmission source mentioned above is, for example, an identifier inherent to each terminal 2021, or an identifier indicating the group to which the terminal 2021 belongs.

[0722] In addition, the transmission destination table may not directly establish a correspondence between the identifier corresponding to the terminal of the transmission source and the data server of the transmission destination. For example, the data collection server 2024 maintains a management table and a transmission destination table, and the management table stores label information assigned according to the identifier inherent to the terminal 2021, and the transmission destination table establishes a correspondence between the label information and the data server of the transmission destination. The data collection server 2024 may also use the management table and the transmission destination table to determine the data server of the transmission destination based on the label information. Here, the label information is, for example, the type, model, owner, group to which the terminal 2021 corresponding to the identifier belongs, or other control information for management or control information for service provision assigned to the identifier. In addition, in the transmission destination table, an identifier inherent to each sensor may be used instead of an identifier corresponding to the terminal 2021 of the transmission source. In addition, the determination rule of the transmission destination server may also be set from the client device 2026.

[0723] The data collection server 2024 may also determine a plurality of data servers as transmission destinations, and transmit the received location-related data to the plurality of data servers. According to this structure, for example, when the location-related data is automatically backed up, or when the location-related data needs to be sent to the data servers for providing each service in order to use the location-related data in common in different services, the intended data transmission can be achieved by changing the setting of the data collection server 2024. As a result, compared with the case where the transmission destination of the location-related data is set for a separate terminal 2021, the workload required for system construction and modification can be reduced.

[0724] The data collection server 2024 may also register the data server specified by the transmission request signal as a new transmission destination based on the transmission request signal received from the data server, and transmit the position-related data received thereafter to the data server.

[0725] The data collection server 2024 may also store the location-related data received from the terminal 2021 in a recording device, and based on a sending request signal received from the terminal 2021 or the data server, send the location-related data specified by the sending request signal to the terminal 2021 or the data server of the request source.

[0726] The data collection server 2024 may determine whether the location-related data can be provided to the data server or terminal 2021 of the request source, and transmit or send the location-related data to the data server or terminal 2021 of the request source if it is determined that the data can be provided.

[0727] When receiving the current location-related data request from the client device 2026, the data collection server 2024 can request the terminal 2021 to send the location-related data even if it is not the location-related data sending timing of the terminal 2021, and the terminal 2021 sends the location-related data according to the sending request.

[0728] In the above description, it is assumed that the terminal 2021 sends location information data to the data collection server 2024, but the data collection server 2024 may also have, for example, functions for managing the terminal 2021, functions required for collecting location-related data from the terminal 2021, or functions used when collecting location-related data from the terminal 2021.

[0729] The data collection server 2024 may also have a function of sending a data request signal to the terminal 2021 and collecting location-related data. The data request signal is a signal requesting the transmission of location information data.

[0730] In the data collection server 2024, management information such as an address for communicating with the terminal 2021 as the data collection target or an identifier unique to the terminal 2021 is registered in advance. Based on the registered management information, the data collection server 2024 collects location-related data from the terminal 2021. The management information may also include information such as the type of sensor provided by the terminal 2021, the number of sensors provided by the terminal 2021, and the communication method supported by the terminal 2021.

[0731] The data collection server 2024 may also collect information such as the working status or current location of the terminal 2021 from the terminal 2021 .

[0732] The management information may be registered from the client device 2026, or the terminal 2021 may start the registration process by sending a registration request to the data collection server 2024. The data collection server 2024 may also have a function of controlling communication with the terminal 2021.

[0733] The communication linking the data collection server 2024 and the terminal 2021 may also be a dedicated line provided by a service provider such as an MNO (Mobile Network Operator) or an MVNO (Mobile Virtual Network Operator), or a virtual dedicated line formed by a VPN (Virtual Private Network), etc. According to this structure, the communication between the terminal 2021 and the data collection server 2024 can be performed securely.

[0734] The data collection server 2024 may also have a function of authenticating the terminal 2021 or encrypting the data sent and received to and from the terminal 2021. Here, the authentication process of the terminal 2021 or the encryption process of the data is performed using an identifier inherent to the terminal 2021 or an identifier inherent to a terminal group including a plurality of terminals 2021, which is shared in advance between the data collection server 2024 and the terminal 2021. The identifier is, for example, an IMSI (International Mobile Subscriber Identity) which is a unique number stored in a SIM (Subscriber Identity Module) card. The identifier used in the authentication process and the identifier used in the data encryption process may be the same or different.

[0735] The authentication or data encryption process between the data collection server 2024 and the terminal 2021 can be provided as long as both the data collection server 2024 and the terminal 2021 have the function of implementing the process, and is independent of the communication method used by the communication device 2022 for relaying. Therefore, it is possible to use a common authentication or encryption process regardless of which communication method the terminal 2021 uses, so the convenience of the user's system construction is improved. However, being independent of the communication method used by the communication device 2022 for relaying means that it is not necessary to change according to the communication method. That is, the authentication or data encryption process between the data collection server 2024 and the terminal 2021 can also be switched according to the communication method used by the relay device for the purpose of improving transmission efficiency or ensuring security.

[0736] The data collection server 2024 may also provide the client device 2026 with a UI for managing data collection rules such as the type of location-related data collected from the terminal 2021 and the data collection schedule. Thus, the user can specify the terminal 2021 from which data is to be collected, the data collection time and frequency, etc., using the client device 2026. In addition, the data collection server 2024 may also specify an area on a map from which data is to be collected, and collect location-related data from the terminal 2021 included in the area.

[0737] When managing data collection rules for each terminal 2021, the client device 2026 displays a list of terminals 2021 or sensors to be managed on the screen, for example. The user sets whether or not to collect data, the collection schedule, etc. for each item in the list.

[0738] When specifying an area on a map where data collection is desired, the client device 2026, for example, displays a two-dimensional or three-dimensional map of the area to be managed on the screen. The user selects an area to collect data on the displayed map. The area selected on the map can be a circular or rectangular area centered on a point specified on the map, or a circular or rectangular area that can be determined by dragging. In addition, the client device 2026 can also select an area in pre-set units such as a city, an area within a city, a block, or a main road. In addition, instead of using a map to specify an area, the area can be set by inputting values ​​of latitude and longitude, or the area can be selected from a list of candidate areas derived based on input text information. The text information is, for example, the name of a region, city, or landmark.

[0739] Furthermore, the user may designate one or more terminals 2021 and set conditions such as a range of 100 meters around the terminal 2021, thereby collecting data while dynamically changing the designated area.

[0740] In addition, when the client device 2026 has a sensor such as a camera, an area on the map can also be specified based on the position of the client device 2026 in real space obtained based on the sensor data. For example, the client device 2026 can also use the sensor data to estimate its own position, and designate an area within a range of a preset distance or a user-specified distance from a point on the map corresponding to the estimated position as an area for collecting data. In addition, the client device 2026 can also designate the sensing area of ​​the sensor, that is, the area corresponding to the obtained sensor data, as an area for collecting data. Alternatively, the client device 2026 can also designate an area based on a position corresponding to the sensor data specified by the user as an area for collecting data. The estimation of the area or position on the map corresponding to the sensor data can be performed by the client device 2026 or by the data collection server 2024.

[0741] When specifying an area on a map, the data collection server 2024 may also determine the terminal 2021 in the specified area by collecting the current location information of each terminal 2021, and request the determined terminal 2021 to send the location-related data. In addition, instead of the data collection server 2024 determining the terminal 2021 in the area, the data collection server 2024 may send information indicating the specified area to the terminal 2021, and the terminal 2021 may determine whether it is in the specified area, and send the location-related data if it is determined to be in the specified area.

[0742] The data collection server 2024 transmits data such as a list or a map for providing the above-mentioned UI (User Interface) in the application executed by the client device 2026 to the client device 2026. The data collection server 2024 may transmit not only the data such as a list or a map but also the application program to the client device 2026. In addition, the above-mentioned UI may also be provided as content created by HTML etc. which can be displayed by a browser. In addition, some data such as map data may also be provided from a server other than the data collection server 2024 such as the map server 2025.

[0743] When the user performs input such as pressing a setting button to notify that the input is completed, the client device 2026 transmits the input information as setting information to the data collection server 2024. Based on the setting information received from the client device 2026, the data collection server 2024 transmits a request for location-related data or a signal notifying a collection rule of location-related data to each terminal 2021, and collects location-related data.

[0744] Next, an example of controlling the operation of the terminal 2021 based on the additional information added to the three-dimensional or two-dimensional map data will be described.

[0745] In this structure, object information indicating the position of a power supply unit such as a wireless power supply antenna or power supply coil buried in a road or parking lot is included in the three-dimensional data, or is corresponded to the three-dimensional data and provided to a terminal 2021 such as a vehicle or a drone.

[0746] The vehicle or drone that has obtained the object information for charging moves the position of the vehicle itself through automatic driving so that the position of the charging unit such as the charging antenna or charging coil of the vehicle becomes opposite to the area indicated by the object information, and starts charging. In addition, in the case of a vehicle or drone that does not have an automatic driving function, the driver or operator is prompted with an image or sound displayed on the screen to indicate the direction to move or the operation to be performed. Furthermore, if it is determined that the position of the charging unit calculated based on the estimated own position enters the area indicated by the object information or is within a range of a specified distance from the area, the prompted image or sound is switched to content that causes driving or operation to be terminated, and charging is started.

[0747] Furthermore, the object information may not be information indicating the position of the power supply unit, but information indicating an area where a charging efficiency above a predetermined threshold value can be obtained if the charging unit is arranged in the area. The position of the object information may be indicated by a point at the center of the area indicated by the object information, or by an area or line in a two-dimensional plane, or an area, line or plane in a three-dimensional space.

[0748] According to this structure, since the position of the power supply antenna which cannot be grasped by the sensing data of the LiDER or the image taken by the camera can be grasped, the wireless charging antenna of the terminal 2021 of the vehicle etc. and the wireless power supply antenna buried in the road etc. can be aligned with higher accuracy. As a result, the charging speed during wireless charging can be shortened or the charging efficiency can be improved.

[0749] The object information may also be an object other than a power supply antenna. For example, the three-dimensional data may also include the location of the AP of the millimeter wave wireless communication as object information. Thus, the terminal 2021 can know the location of the AP in advance, so it can start communication by directing the directivity of the beam toward the direction of the object information. As a result, it is possible to improve the communication quality by increasing the transmission speed, shortening the time until the communication starts, and extending the period of communication.

[0750] The object information may also include information indicating the type of the object corresponding to the object information. In addition, the object information may also include information indicating the processing that the terminal 2021 should perform when the terminal 2021 is included in the area in the real space corresponding to the position on the three-dimensional data of the object information, or within a range of a specified distance from the area.

[0751] The object information may be provided from a server different from the server providing the three-dimensional data. When the object information is provided separately from the three-dimensional data, an object group storing object information used in the same service may be provided as different data according to the type of the object service or object device.

[0752] The three-dimensional data used in combination with the object information may be either point group data of WLD or feature point data of SWLD.

[0753] Although the server and the client device and the like according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0754] Furthermore, each processing unit included in the server and client device according to the above-mentioned embodiment can be typically implemented as an LSI of an integrated circuit. These can be made into one chip separately, or part or all of them can be made into one chip.

[0755] Furthermore, integrated circuits are not limited to LSIs, and can be implemented by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays) that are programmable after LSI manufacturing, or reconfigurable processors that can reconfigure the connections or settings of circuit cells within LSIs can also be used.

[0756] Furthermore, in each of the above-mentioned embodiments, each component may be formed by dedicated hardware, or may be implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0757] Furthermore, the present disclosure can be implemented as a three-dimensional data creation method executed by a server, a client device, or the like.

[0758] Furthermore, the division of the functional blocks in the block diagram is an example, and multiple functional blocks can be implemented as one functional block, and one functional block can also be divided into multiple blocks, and a part of the functions can also be moved to other functional blocks. Furthermore, the functions of multiple functional blocks with similar functions can also be processed in parallel or in time division by a single hardware or software.

[0759] Furthermore, the execution order of each step in the flowchart is an example given for the purpose of specifically describing the present disclosure, and may be an order other than the above. Furthermore, part of the above steps may be executed simultaneously (in parallel) with other steps.

[0760] The above describes the three-dimensional data production device and display device involved in one or more forms based on the embodiments, but the present disclosure is not limited to these embodiments. Within the scope of the present disclosure, the forms obtained by performing various modifications that can be thought of by those skilled in the art on the present embodiment and the forms obtained by combining the constituent elements in different embodiments are all included in the scope of one or more forms.

[0761] Industrial Applicability

[0762] The present disclosure can be applied to a client device or the like.

[0763] Explanation of symbols

[0764] 100, 400 3D data encoding device

[0765] 101, 201, 401, 501 obtained

[0766] 102, 402 coding area determination unit

[0767] 103 Division

[0768] 104, 644 Coding Department

[0769] 111, 607 three-dimensional data

[0770] 112, 211, 413, 414, 511, 634 coded 3D data

[0771] 200, 500 3D data decoding device

[0772] 202 Decoding starts GOS decision section

[0773] 203 Decoding SPC determination unit

[0774] 204, 625 decoding department

[0775] 212, 512, 513 decoding 3D data

[0776] 403SWLD Extraction Department

[0777] 404WLD Coding Department

[0778] 405SWLD Coding Department

[0779] 411 Input 3D data

[0780] 412 Extract 3D data

[0781] 502 Header Analysis

[0782] 503WLD decoding department

[0783] 504SWLD decoding unit

[0784] 600 own vehicles

[0785] 601 Surrounding vehicles

[0786] 602, 605 sensor detection range

[0787] Area 603, 606

[0788] 604 occlusion area

[0789] 620, 620A 3D data production device

[0790] 621, 641 3D Data Production Department

[0791] 622 Request Scope Determination Department

[0792] 623 Search Department

[0793] 624, 642 receiving department

[0794] 626 Synthesis Department

[0795] 627 Detection Area Determination Department

[0796] 628 Peripheral Condition Detection Department

[0797] 629 Autonomous Movement Control Unit

[0798] 631, 651 sensor information

[0799] 632 1st 3D data

[0800] 633 Request scope information

[0801] 635 2nd 3D data

[0802] 636 3rd dimensional data

[0803] 637 Commission Signal

[0804] 638 Send data

[0805] 639 Surrounding Condition Detection Results

[0806] 640, 640A three-dimensional data transmission device

[0807] 643 Extraction Department

[0808] 645 Sending Department

[0809] 646 Transmission Permission Judgment Unit

[0810] 652 5th 3D data

[0811] 654 6th 3D data

[0812] 700 Three-dimensional information processing device

[0813] 701 Three-dimensional Map Acquisition Department

[0814] 702 Own vehicle detection data acquisition unit

[0815] 703 Abnormal Situation Judgment Department

[0816] 704 Response Work Decision Department

[0817] 705 Work Control Department

[0818] 711 3D Map

[0819] 712 3D data of vehicle detection

[0820] 801 vehicles

[0821] 802 Space

[0822] 810 3D data production device

[0823] 811 data receiving unit

[0824] 812, 819 Communications Department

[0825] 813 Receiving Control Unit

[0826] 814, 821 format conversion department

[0827] 815 Sensor

[0828] 816 3D Data Production Department

[0829] 817 3D Data Synthesis Department

[0830] 818 3D data storage department

[0831] 820 Sending Control Unit

[0832] 822 Data transmission unit

[0833] 831, 832, 834, 835, 836, 837 three-dimensional data

[0834] 833 sensor information

[0835] 901 Server

[0836] 902, 902A, 902B, 902C client devices

[0837] 1011, 1111 data receiving unit

[0838] 1012, 1020, 1112, 1120 Communications Department

[0839] 1013, 1113 receiving control unit

[0840] 1014, 1019, 11...

Claims

1. A mobile device, wherein: have: communication devices; as well as processor, the processor, obtaining a three-dimensional map including first information and second information, wherein the first information indicates a position of an object, and the second information indicates a process to be performed by the mobile device at a position corresponding to the object, Based on the second information, the process is performed at a position corresponding to the object.

2. The mobile device according to claim 1, wherein: The object is an access point, The second information indicates the process to be performed when the mobile device is within a range of a preset distance from the access point. The processing includes a processing of starting transmission by directing the directivity of the beam toward the direction of the position of the access point using the first information before the mobile device reaches the access point.

3. The mobile device according to claim 2, wherein: The mobile device further comprises a sensor, The processor also obtains sensor data from the sensor, In the transmitting, the sensor data is transmitted to a server via the access point.

4. A method performed by a mobile device, the mobile device comprising a communication device and a processor, wherein: obtaining a three-dimensional map including first information and second information, wherein the first information indicates a position of an object, and the second information indicates a process to be performed by the mobile device at a position corresponding to the object, Based on the second information, the process is performed at a position corresponding to the object.

5. The method of claim 4, wherein: The object is an access point, The second information indicates the process to be performed when the mobile device is within a range of a preset distance from the access point. The processing includes a processing of starting transmission by directing the directivity of the beam toward the direction of the position of the access point using the first information before the mobile device reaches the access point.

6. The method of claim 5, wherein: The mobile device further comprises a sensor, In the method, sensor data is also obtained from the sensor, In the transmitting, the sensor data is transmitted to a server via the access point.

7. A mobile device, wherein: have: communication devices; Sensors; and processor, the processor, receiving information representing location information, Based on the position information, data acquired by the sensor is transmitted.

8. A method performed by a mobile device, the mobile device comprising a communication device, a sensor, and a processor, wherein: receiving information representing location information, Based on the position information, data acquired by the sensor is transmitted.

9. A server, wherein: have: a communication device; and processor, The processor sends a request to transmit data including location information to a plurality of mobile devices.

10. A method executed by a server, the server comprising a communication device and a processor, wherein: A request to transmit data including location information is transmitted to a plurality of mobile devices.

11. A mobile device, wherein: have: communication devices; Sensors; and processor, The processor transmits information indicating a position and data acquired by the sensor.

12. A method performed by a mobile device, the mobile device comprising a communication device, a sensor, and a processor, wherein: Information indicating the position and data acquired by the sensor are transmitted.

13. A server, wherein: have: a communication device; and processor, The processor receives information indicative of a location and data acquired by a sensor.

14. A method executed by a server, the server comprising a communication device and a processor, wherein: receiving information representing location information, Receive information indicating the location and data acquired by the sensor.

15. A device, wherein: have: Processor; and sensor, the processor, Get the position information represented by the relative position with respect to the feature point, Based on the information acquired by the sensor, a position indicated by the position information is determined.

16. A method performed by a device having a processor and a sensor, wherein: Get the position information represented by the relative position with respect to the feature point, Based on the information acquired by the sensor, a position indicated by the position information is determined.

17. A device, wherein: have: Processor; and sensor, The processor generates position information represented by a relative position with respect to a feature point based on information acquired by the sensor.

18. A method performed by a device having a processor and a sensor, wherein: Based on the information acquired by the sensor, position information represented by a relative position with respect to the feature point is generated.

Images