Point group data processing method and point group data processing device

Abstract

The invention relates to a point group data generation method and a point group data generation device. The point group data generation method of the present invention is executed using a processor, the point group data generation method comprising: acquiring a sensing result from a distance sensor; and generating second point group data using the sensing result, the second point group data including three-dimensional coordinates and one or more attribute values of each of the plurality of points.

Description

[0001] This application is a divisional of an invention patent application filed on November 16, 2018, with application number 201880074466.2 and invention name “Three-dimensional point group data generation method, position inference method, three-dimensional point group data generation device and position inference device”. Technical Field

[0002] The present disclosure relates to a point group data generating method, a position estimating method, a point group data generating device, and a position estimating device. Background Art

[0003] Patent document 1 discloses a method for transmitting three-dimensional shape data. In patent document 1, the three-dimensional shape data is sent to the network for each element such as polygon or voxel. Then, on the receiving side, the three-dimensional shape data is acquired, and the image is expanded and displayed for each received element.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 9-237354 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the techniques disclosed in the above patent documents require further improvements.

[0009] Means for solving problems

[0010] In order to achieve the above-mentioned purpose, a point group data processing method of one form of the present invention is executed using a processor, wherein the point group data processing method includes: obtaining a sensing result from a distance sensor; and using the sensing result to generate a second point group data, wherein the second point group data includes the three-dimensional coordinates and one or more attribute values ​​of each point in a plurality of points.

[0011] In addition, a point group data processing device of one form of the present disclosure includes a distance sensor and a processor, wherein the processor is configured to: obtain a sensing result from the distance sensor; and use the sensing result to generate a second point group data, wherein the second point group data includes three-dimensional coordinates and one or more attribute values ​​of each point among a plurality of points.

[0012] In addition, these overall or specific forms can be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0013] Effects of the Invention

[0014] The present disclosure enables further improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram showing an overview of a position estimation system.

[0016] Figure 2 This is a block diagram showing an example of the functional structure of a position estimation system.

[0017] Figure 3 This is a block diagram showing an example of a functional configuration of a vehicle as a client device.

[0018] Figure 4 This is a timing chart showing an example of the operation of the position estimation system.

[0019] Figure 5 This is a diagram for explaining the operation of the position estimation system.

[0020] Figure 6 This is a block diagram showing an example of the functional structure of the mapping unit.

[0021] Figure 7 This is a flowchart showing an example of detailed processing of the mapping process.

[0022] Figure 8 This is a flowchart showing an example of the operation of the three-dimensional point cloud data generating device in Modification Example 1.

[0023] Fig. 9 This is a flowchart showing an example of detailed processing of calculating the importance.

[0024] Fig.10 This is a diagram showing an example of the data structure of the third three-dimensional point.

[0025] Fig.11 This is a block diagram showing an example of the functional structure of the encoding unit.

[0026] Fig.12 This is a flowchart showing an example of detailed processing of the encoding process. DETAILED DESCRIPTION

[0027] (Insights that form the basis of this disclosure)

[0028] Three-dimensional point group data is popular in manufacturing and construction, and in recent years, it has become important in information technology applications such as autonomous driving. Three-dimensional point group data is usually very inefficient in data storage and data processing. Therefore, in order to use three-dimensional point group data in actual applications, it is required to generate three-dimensional point group data compressed to a compact size. In this way, it is necessary to be able to reduce the amount of data of the three-dimensional point group data. In addition, it is also necessary to use the three-dimensional point group data with reduced data volume to infer one's own position.

[0029] The purpose of the present disclosure is to provide a three-dimensional point group data generation method and a three-dimensional point group data generation device that can effectively reduce the data volume of three-dimensional point group data, and a position estimation method and a position estimation device that can use the three-dimensional point group data with reduced data volume to estimate its own position.

[0030] A three-dimensional point group data generation method in one form of the present invention is a three-dimensional point group generation method that uses a processor to generate a three-dimensional point group including multiple three-dimensional points, obtains a two-dimensional image obtained by photographing a three-dimensional object with a camera and a first three-dimensional point group obtained by sensing the three-dimensional object with a distance sensor, detects from the obtained two-dimensional image one or more attribute values ​​of the two-dimensional image corresponding to positions on the two-dimensional image, and for each of the one or more attribute values ​​detected, (i) determines one or more first three-dimensional points corresponding to the positions of the attribute values ​​on the two-dimensional image among multiple three-dimensional points constituting the first three-dimensional point group, and (ii) adds the attribute values ​​to the one or more first three-dimensional points determined, thereby generating a second three-dimensional point group, wherein the second three-dimensional point group includes one or more second three-dimensional points each having the attribute values.

[0031] Thus, a second three-dimensional point group is generated, wherein the second three-dimensional point group includes a second three-dimensional point in which the attribute value of the two-dimensional image corresponding to the position on the two-dimensional image obtained using the camera is added to the three-dimensional point corresponding to the position on the two-dimensional image, among the plurality of three-dimensional points constituting the first three-dimensional point group obtained using the distance sensor. Therefore, in a position estimation device for estimating its own position, even if the three-dimensional point group is not re-sensed, if the two-dimensional image around the device is re-photographed, the own position can be efficiently estimated by comparing the attribute value corresponding to the position of the captured two-dimensional image with the attribute value of each second three-dimensional point added to the second three-dimensional point group.

[0032] In addition, in the acquisition, a plurality of the two-dimensional images obtained by shooting with the camera at different positions and / or postures are respectively acquired, and in the detection, for each of the plurality of the two-dimensional images obtained, the one or more attribute values ​​are detected, and the three-dimensional point group data generation method also uses the one or more attribute values ​​detected for each of the plurality of two-dimensional images to match the corresponding attribute values ​​in two of the plurality of two-dimensional images, thereby outputting one or more pairs of matched attribute values, and in the determination, for each of the one or more pairs, the positions on the two-dimensional images of the two attribute values ​​constituting the pair and the positions and postures of the cameras when each of the two two-dimensional images was shot are used to determine the one or more first three-dimensional points corresponding to the positions on the two-dimensional images of the two attribute values ​​constituting the pair, or it may be that in the generation, for each of the one or more pairs, attribute values ​​based on the two attribute values ​​constituting the pair are added to the one or more first three-dimensional points determined, thereby generating the second three-dimensional point group.

[0033] Thus, a pair of attribute values ​​determined by matching the attribute values ​​of the two two-dimensional images is determined, and the positions of the attribute values ​​of the pair corresponding to the two-dimensional images are used to determine the three-dimensional point corresponding to the position, and the two attribute values ​​constituting the pair are added to the determined three-dimensional point. Therefore, the three-dimensional point corresponding to the attribute value can be determined with high accuracy.

[0034] Furthermore, in the generating, the second three-dimensional point may be generated by adding a plurality of the attribute values ​​to the determined one first three-dimensional point.

[0035] Therefore, since the number of attribute values ​​attached to one first three-dimensional point is set to multiple, in position inference, the correspondence between the attribute values ​​of the two-dimensional image and the first three-dimensional point can be established with high precision. The attribute values ​​of the two-dimensional image are attribute values ​​corresponding to the position on the two-dimensional image obtained for position inference.

[0036] Furthermore, the plurality of attribute values ​​added to the one first three-dimensional point in the generation may be attribute values ​​detected from a plurality of the two-dimensional images, respectively.

[0037] A first three-dimensional point can be photographed from multiple viewpoints. In addition, in position estimation, since two-dimensional images are photographed at different positions, the obtained two-dimensional images are photographed from multiple different viewpoints. Therefore, by adding multiple attribute values ​​detected from multiple two-dimensional images, in position estimation, even if two-dimensional images photographed from multiple different viewpoints are used, it is easy to establish a correspondence between the attribute values ​​obtained from each two-dimensional image and the first three-dimensional point. Therefore, position estimation can be easily performed.

[0038] Furthermore, the plurality of attribute values ​​added to the one first three-dimensional point in the generation may be attribute values ​​of mutually different attribute types.

[0039] Therefore, since the attribute value attached to a first three-dimensional point is set to a plurality of different types of attribute values, in position inference, the correspondence between the attribute value of the two-dimensional image and the first three-dimensional point can be established with high precision. The attribute value of the two-dimensional image is the attribute value corresponding to the position on the two-dimensional image obtained for position inference.

[0040] Furthermore, in the detection, a feature amount calculated for each of a plurality of regions constituting the acquired two-dimensional image may be detected as the one or more attribute values ​​of the two-dimensional image corresponding to a position on the two-dimensional image.

[0041] Therefore, the feature amount at the position of the two-dimensional image can be easily calculated by a predetermined method.

[0042] In addition, it may also be that, in the generation, for each of the one or more second three-dimensional points, (i) the importance of the second three-dimensional point is calculated based on the attribute value attached to the second three-dimensional point, and (ii) the calculated importance is attached to the second three-dimensional point, thereby generating a third three-dimensional point group, wherein the third three-dimensional point group includes one or more third three-dimensional points each having the attribute value and the importance.

[0043] Therefore, for example, the client device can preferentially use the more important third three-dimensional points, and thus can adjust the data volume of the three-dimensional point group data used in the processing so as not to adversely affect the accuracy of the processing.

[0044] In addition, it is also possible to further receive a threshold from the client device, extract one or more fourth three-dimensional points from the one or more third three-dimensional points to which an importance exceeding the received threshold is attached, and send a fourth three-dimensional point group including the one or more extracted fourth three-dimensional points to the client device.

[0045] Therefore, the data amount of the three-dimensional point group data to be transmitted can be modulated according to the request of the client device.

[0046] In addition, a position inference method in one form of the present disclosure is a position inference method for inferring the current position of a moving body using a processor, obtaining a three-dimensional point group including a plurality of three-dimensional points to which first attribute values ​​as attribute values ​​of a first two-dimensional image obtained by photographing a three-dimensional object are respectively pre-attached, obtaining a second two-dimensional image of the surroundings of the moving body photographed by a camera possessed by the moving body, detecting from the obtained second two-dimensional image at least one second attribute value as attribute values ​​of the second two-dimensional image corresponding to a position on the second two-dimensional image, for each of the one or more second attribute values ​​detected, determining at least one fifth three-dimensional point corresponding to the second attribute value among the plurality of three-dimensional points, thereby generating at least one combination consisting of the second attribute value and the one or more fifth three-dimensional points, obtaining from a storage device the position and posture of the camera relative to the moving body, and calculating the position and posture of the moving body using the one or more generated combinations and the obtained position and posture of the camera.

[0047] Thus, the attribute values ​​of the two-dimensional image obtained by photographing the three-dimensional object are pre-attached to the three-dimensional point group. Therefore, even if the three-dimensional point group is not re-sensed, if the second two-dimensional image around the present device is re-photographed, the own position can be efficiently inferred by comparing the second attribute value and the combination of one or more fifth three-dimensional points corresponding to the second attribute value, and the first attribute value of each three-dimensional point attached to the three-dimensional point group, wherein the second attribute value corresponds to the position of the second two-dimensional image obtained by photographing.

[0048] In addition, these overall or specific forms can be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, or recording media.

[0049] Hereinafter, a three-dimensional point group data generating method, a position estimating method, a three-dimensional point group data generating device, and a position estimating device according to one embodiment of the present invention will be described in detail with reference to the drawings.

[0050] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, materials, constituent elements, configuration positions of constituent elements, connection methods, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, the constituent elements in the following embodiments that are not recorded in the independent claims representing the superordinate concepts are described as arbitrary constituent elements.

[0051] (Implementation Method 1)

[0052] First, an overview of this embodiment will be described. In this embodiment, a position estimation system for estimating the own position of a moving object such as a vehicle using three-dimensional point group data will be described.

[0053] Figure 1 This is a diagram showing an overview of a position estimation system.

[0054] Figure 1 The three-dimensional point group data generation device 100, vehicles 200, 300, a communication network 400, and a base station 410 of a mobile communication system are shown. For example, the position estimation system 1 includes the three-dimensional point group data generation device 100 and vehicles 200, 300 among these components. The vehicle 200 and the vehicle 300 are vehicles of different types. In the position estimation system 1, not only one vehicle 200 is provided, but two or more vehicles 200 may be provided. Similarly, not only one vehicle 300 is provided, but three or more vehicles 300 may be provided.

[0055] The three-dimensional point group data generating device 100 is a device that obtains a plurality of two-dimensional images captured by a camera 210 provided by a vehicle 200 and a three-dimensional point group obtained by a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 230 provided by the vehicle 200, and generates three-dimensional point group data for the vehicle 300 to infer the own position of the vehicle 300. Here, the three-dimensional point group data is data representing a three-dimensional point group including a plurality of three-dimensional points to which feature quantities of feature points are added. The plurality of three-dimensional points to which feature quantities of feature points are added will be described later. The three-dimensional point group data generating device 100 is, for example, a server.

[0056] The vehicle 200 is a vehicle equipped with a camera 210 and a LiDAR 230, which captures a two-dimensional image of a three-dimensional object around the vehicle 200 and detects a three-dimensional point group of the three-dimensional object. The three-dimensional point group is composed of, for example, a plurality of three-dimensional coordinates (X, Y, Z) representing each of a plurality of positions on the surface of the three-dimensional object around the vehicle 200. The vehicle 200, for example, generates a two-dimensional image captured by the camera 210 of the vehicle 200 from the position of the vehicle 200 at each timing by capturing images by the camera 210 at different timings while traveling on a road. That is, the vehicle 200 captures images by the camera 210 at different timings while traveling on a moving path. Therefore, the plurality of two-dimensional images are obtained by capturing images by the camera 210 at different positions and / or postures. In addition, the vehicle 200 generates a three-dimensional point group sensed by the LiDAR 230 of the vehicle 200 from the position of the vehicle 200 at each timing by capturing images by the LiDAR 230 at different timings while traveling on a road, for example.

[0057] The sensor data based on LiDAR230 can also be used to determine the two-dimensional image captured by the camera 210 of the vehicle 200 and the position and posture of the camera 210 when the two-dimensional image was captured. In this case, the accurate three-dimensional map (Accurate 3Dmap) used by a general vehicle to infer its own position can also be generated using a three-dimensional point group obtained by other equipment different from the vehicle 200. The accurate three-dimensional map is an example of the first three-dimensional point group, which is composed of multiple three-dimensional points. Multiple three-dimensional points can also be represented by coordinates in three different directions, for example, the X-axis direction, the Y-axis direction, and the Z-axis direction.

[0058] Sensor data based on LiDAR 230 can also be used to generate an accurate three-dimensional map used by general vehicles to estimate their own positions. In this case, LiDAR 230 is a distance sensor that can measure more three-dimensional points with good accuracy, for example, compared with LiDAR used for general vehicle autonomous driving or driving assistance. In other words, it is a sensing device that obtains a three-dimensional point group and a two-dimensional image for generating an accurate three-dimensional map used by the vehicle 200 to estimate its own position.

[0059] The timing of photographing by the camera 210 and the timing of sensing by the LiDAR 230 may be synchronized or asynchronous. The timing of photographing by the camera 210 and the timing of sensing by the LiDAR 230 may be the same timing or different timing.

[0060] The vehicle 300 is equipped with a camera 310, and is a vehicle that captures a two-dimensional image of a three-dimensional object around the vehicle 300. The vehicle 300 estimates the own position of the vehicle 300 using the two-dimensional image captured by the camera 310 and the three-dimensional point group obtained from the three-dimensional point group data generating device 100. The vehicle 300 uses the estimated result of its own position to perform automatic driving or driving assistance. That is, the vehicle 300 functions as a client device.

[0061] The communication network 400 may be a general network such as the Internet or a dedicated network. The base station 410 is a base station used in a mobile communication system such as a third generation mobile communication system (3G), a fourth generation mobile communication system (4G), LTE (registered trademark), or a fifth generation mobile communication system (5G).

[0062] Next, use Figure 2 A specific example of the functional configuration of the position estimation system will be described.

[0063] Figure 2 This is a block diagram showing an example of the functional structure of a position estimation system.

[0064] First, the functional structure of the three-dimensional point cloud data generating device 100 will be described.

[0065] The three-dimensional point cloud data generating device 100 includes a communication unit 110 , a mapping unit 120 , a coding unit 130 , and an external memory 140 .

[0066] The communication unit 110 communicates with the vehicles 200 and 300 via the communication network 400. The communication unit 110 receives the two-dimensional image and the three-dimensional point group from the vehicle 200 via the communication network 400 to obtain the two-dimensional image and the three-dimensional point group. In addition, the communication unit 110 may also receive the relative position and posture of the camera 210 relative to the LiDAR 230 from the vehicle 200 via the communication network 400. In addition, the communication unit 110 sends the three-dimensional point group with the feature amount of the feature point attached to the vehicle 300 via the communication via the communication network 400. In addition, the communication unit 110 may also send the three-dimensional point group with the feature amount of the feature point attached to the upper server not shown in the figure via the communication via the communication network 400.

[0067] In addition, the 3D point group data generating device 100 may store the relative position and posture of the camera 210 with respect to the LiDAR 230 in advance in a non-volatile storage device such as the external memory 140 as a table of each vehicle type or each vehicle of each vehicle 200. In this case, the 3D point group data generating device 100 may obtain information for determining the vehicle type or vehicle from the vehicle 200, and determine the relative position and posture of the camera 210 in the vehicle 200 with respect to the LiDAR 230 from the table stored in the non-volatile storage device.

[0068] In addition, the communication unit 110 is implemented by a communication interface that can be communicatively connected to the communication network 400. Specifically, the communication unit 110 is communicatively connected to the communication network 400 through a communication connection with a base station 410 of a mobile communication system. The communication unit 110 can also be implemented, for example, by a wireless communication interface suitable for a communication standard used in a mobile communication system such as a 3rd generation mobile communication system (3G), a 4th generation mobile communication system (4G), LTE (registered trademark), a 5th generation mobile communication system (5G), etc. In addition, the communication unit 110 can be implemented, for example, by a wireless LAN (Local Area Network) interface that complies with IEEE802.11a, b, g, n, ac standards, or by a communication connection with a router (for example, a mobile wireless LAN router) not shown in the figure, or by a communication interface that is communicatively connected to the communication network 400.

[0069] The mapping unit 120 uses the two-dimensional image and the three-dimensional point group obtained by the communication unit 110 to perform a mapping process to add the attribute values ​​detected from the two-dimensional image to a plurality of three-dimensional points constituting the three-dimensional point group. Figure 6 and Figure 7 Describe the details of the mapping unit.

[0070] The encoding unit 130 encodes the three-dimensional point group obtained by the mapping process to generate a code stream. The encoding unit 130 may store the generated code stream in the external memory 140. In addition, the encoding unit 130 may also cause the communication unit 110 to transmit the generated code stream to the vehicle 300 via the communication network 400. Figure 8 and Fig. 9 The details of the encoding process performed by the encoding unit 130 will be described.

[0071] Each of the mapping unit 120 and the encoding unit 130 may be implemented by a processor and a memory, or may be implemented by a dedicated circuit. In other words, the mapping unit 120 and the encoding unit 130 may be implemented by software, or may be implemented by hardware.

[0072] The external memory 140 may also store information required by the processor, such as programs. The external memory 140 may also store data generated by the processor during processing, such as a three-dimensional point group with attribute values ​​and a coded stream. The external memory 140 may be implemented, for example, by a non-volatile storage device such as a flash memory or a HDD (Hard Disk Drive).

[0073] The vehicle 200 includes a camera 210 , an acquisition unit 220 , a LiDAR 230 , an acquisition unit 240 , a memory 250 , and a communication unit 260 .

[0074] The camera 210 obtains a two-dimensional image by photographing three-dimensional objects in the three-dimensional space around the vehicle 200. The camera 210 is arranged at a predetermined position and a predetermined posture relative to the vehicle 200. For example, the camera 210 can be arranged inside the vehicle 200 or outside the vehicle 200. In addition, the camera 210 can photograph the front of the vehicle 200, the right side or the left side of the vehicle 200, the rear side of the vehicle 200, and the 360-degree full range of the vehicle 200. The camera 210 can be composed of one camera or more than two cameras. The camera 210 can obtain multiple two-dimensional images by photographing at different multiple timings. The camera 210 can also photograph a moving image including multiple frames as multiple two-dimensional images at a predetermined frame rate.

[0075] The acquisition unit 220 is provided corresponding to the camera 210, stores two-dimensional images obtained by shooting at different multiple timings in the camera 210, and outputs the stored two-dimensional images to the communication unit 260. The two-dimensional image is associated with the timing at which the two-dimensional image is shot. The acquisition unit 220 may be a processing unit built into the camera 210. That is, the camera 210 may have the function of the acquisition unit 220.

[0076] LiDAR230 obtains a three-dimensional point group by sensing a three-dimensional object in the three-dimensional space around the vehicle 200. LiDAR230 is a laser sensor that is maintained on the vehicle 200 and detects the distance to a three-dimensional object located within a detection range of 360 degrees in the horizontal direction of the vehicle 200 and a predetermined angle (e.g., 30 degrees) in the vertical direction. LiDAR230 is an example of a distance sensor. LiDAR230 emits laser light to the surroundings and measures the distance from LiDAR230 to the three-dimensional object by detecting the laser light reflected by the surrounding objects. LiDAR230 measures the distance, for example, in centimeters. In this way, LiDAR230 detects the three-dimensional coordinates of each of the multiple points on the terrain surface around the vehicle 200. That is, LiDAR230 detects the three-dimensional shape of the terrain including the objects around the vehicle 200 by detecting the multiple three-dimensional coordinates of the surrounding terrain surface. In this way, in LiDAR 230, a three-dimensional point group consisting of three-dimensional coordinates of a plurality of points can be obtained, that is, a three-dimensional point group representing the three-dimensional shape of the terrain including objects around the vehicle 200. In addition, the distance sensor is not limited to LiDAR 230, and may be other distance sensors such as millimeter wave radar, ultrasonic sensor, ToF (Time of Flight) camera, stereo camera, etc.

[0077] The acquisition unit 240 is provided corresponding to the LiDAR 230, stores the three-dimensional point group obtained by sensing in the LiDAR 230 at different multiple timings, and outputs the stored three-dimensional point group to the communication unit 260. The three-dimensional point group is associated with the timing at which the three-dimensional point group is sensed. The acquisition unit 240 may be a processing unit built into the LiDAR 230. That is, the LiDAR 230 may have the function of the acquisition unit 240.

[0078] The memory 250 stores the position and posture of the camera 210 relative to the vehicle 200, and the position and posture of the LiDAR 230 relative to the vehicle 200. In addition, the memory 250 may also store the relative position and posture of the camera 210 relative to the LiDAR 230. The relative position and posture of the camera 210 relative to the LiDAR 230 may be detected by matching the intensity of the laser output by the LiDAR 230 and a two-dimensional image using a method such as mutual information, wherein the two-dimensional image is a two-dimensional image of reflected light in a three-dimensional object including the laser of the LiDAR 230 captured by the camera 210, or may be detected using a device other than the camera 210 and the LiDAR 230. The memory 250 is implemented, for example, by a non-volatile storage device.

[0079] The communication unit 260 communicates with the three-dimensional point group data generating device 100 via the communication network 400. The communication unit 260 transmits the two-dimensional image and the three-dimensional point group to the three-dimensional point group data generating device 100 through the communication via the communication network 400. In addition, the communication unit 260 may also transmit the relative position and posture of the camera 210 with respect to the LiDAR 230 to the three-dimensional point group data generating device 100 through the communication via the communication network 400.

[0080] In addition, the communication unit 260 is implemented by a communication interface capable of communicating with the communication network 400. Specifically, the communication unit 260 is connected to the communication network 400 through a communication connection with a base station 410 of the mobile communication system. The communication unit 260 may be implemented by a wireless communication interface suitable for a communication standard used in a mobile communication system such as a third generation mobile communication system (3G), a fourth generation mobile communication system (4G), LTE (registered trademark), or a fifth generation mobile communication system (5G).

[0081] Next, the functional configuration of vehicle 300 will be described.

[0082] Figure 3 This is a block diagram showing an example of a functional configuration of a vehicle as a client device.

[0083] The vehicle 300 includes a camera 310 , an acquisition unit 320 , a communication unit 330 , a decoding unit 340 , a position estimation unit 350 , a control unit 360 , and an external memory 370 .

[0084] The camera 310 is arranged at a predetermined position and a predetermined posture relative to the vehicle 300, and photographs a three-dimensional object in the three-dimensional space around the vehicle 300. The camera 310 is different from the camera 210 in that it is arranged in the vehicle 300, but has the same structure as the camera 210 otherwise. Therefore, in the description of the camera 210, the camera 210 can be replaced by the camera 310, and the vehicle 200 can be replaced by the vehicle 300. Therefore, the detailed description of the camera 310 is omitted.

[0085] The acquisition unit 320 is provided corresponding to the camera 310, stores the two-dimensional image captured by the camera 310, and outputs the stored two-dimensional image to the position estimation unit 350. The acquisition unit 320 may also be a processing unit built into the camera 310. That is, the camera 310 may also have the function of the acquisition unit 320.

[0086] The communication unit 330 communicates with the three-dimensional point group data generating device 100 via the communication network 400. The communication unit 330 receives the coded stream from the three-dimensional point group data generating device 100 through the communication via the communication network 400, thereby obtaining the coded stream. In addition, in order to reduce the communication load, the communication unit 330 may also send a threshold value related to the importance described later to the three-dimensional point group data generating device 100 via the communication network 400.

[0087] The communication unit 330 may also obtain all of the multiple three-dimensional points included in the three-dimensional point group of the three-dimensional point group data generating device 100. In addition, when the communication unit 330 obtains the three-dimensional point group from the three-dimensional point group data generating device 100, it may obtain the three-dimensional point group of a specified area based on the position of the vehicle 300 detected by a position detection device with a rough accuracy such as a GPS (Global Positioning System) not shown as all the three-dimensional points mentioned above. In this way, it is also possible not to obtain the three-dimensional point group around the moving path of the moving body such as all the roads in the world, and the amount of data of the obtained three-dimensional point group can be reduced. In addition, the communication unit 330 may not obtain all the three-dimensional points, and may also obtain a plurality of three-dimensional points with a greater importance than the threshold value among the plurality of three-dimensional points mentioned above by sending the threshold value to the three-dimensional point group data generating device 100, and not obtain a plurality of three-dimensional points with an importance below the threshold value.

[0088] In addition, the communication unit 330 is implemented by a communication interface capable of communicating with the communication network 400. Specifically, the communication unit 330 is connected to the communication network 400 through a communication connection with a base station 410 of the mobile communication system. The communication unit 330 may be implemented by a wireless communication interface suitable for a communication standard used in a mobile communication system such as a third generation mobile communication system (3G), a fourth generation mobile communication system (4G), LTE (registered trademark), or a fifth generation mobile communication system (5G).

[0089] The decoding unit 340 decodes the coded stream received from the communication unit 330 to generate a three-dimensional point group to which attribute values ​​are added.

[0090] The position estimation unit 350 estimates the position and posture of the camera 310 in the three-dimensional point group using the two-dimensional image obtained by the camera 310 and the three-dimensional point group with attribute values ​​obtained by decoding in the decoding unit 340 , thereby estimating the position and posture of the vehicle 300 .

[0091] The control unit 360 controls the driving of the vehicle 300. Specifically, the control unit 360 performs automatic driving or driving assistance of the vehicle 300 by controlling a steering wheel for steering the wheels, a power source such as an engine or a motor for rotating and driving the wheels, and brakes for braking the wheels. For example, the control unit 360 uses the current position of the vehicle 300, the destination of the vehicle 300, and surrounding road information to determine a path indicating which road the vehicle 300 is traveling on. In addition, the control unit 360 controls the steering wheel, the power source, and the brakes so that the vehicle travels on the determined path.

[0092] Decoding unit 340, position estimation unit 350, and control unit 360 may be implemented by a processor or a dedicated circuit. In other words, decoding unit 340, position estimation unit 350, and control unit 360 may be implemented by software or hardware.

[0093] The external memory 370 may also store information required by the processor, such as programs. The external memory 370 may also store data generated by the processor during processing, such as a three-dimensional point group with attribute values ​​and a coded stream. The external memory 370 may be implemented, for example, by a non-volatile storage device such as a flash memory or a HDD (Hard Disk Drive).

[0094] Next, the operation of position estimation system 1 will be described.

[0095] Figure 4 This is a timing chart showing an example of the operation of the position estimation system. Figure 5 This is a diagram for explaining the operation of the position estimation system.

[0096] First, the vehicle 200 sends a two-dimensional image and a three-dimensional point group to the three-dimensional point group data generating device 100 via the communication network 400 (S1). The two-dimensional image is obtained by photographing a three-dimensional object in the three-dimensional space around the vehicle 200 by the camera 210, and the three-dimensional point group is obtained by sensing the three-dimensional object in the three-dimensional space around the vehicle 200 by the LiDAR 230. In addition, the detailed processing in the vehicle 200 is performed using Figure 2 This has already been explained and is therefore omitted.

[0097] Next, in the three-dimensional point cloud data generating device 100 , the communication unit 110 acquires the two-dimensional image and the three-dimensional point cloud from the vehicle 200 via the communication network 400 ( S3 ).

[0098] Then, the mapping unit 120 performs a mapping process of adding attribute values ​​detected from the two-dimensional image to a plurality of three-dimensional points constituting the three-dimensional point group, using the two-dimensional image and the three-dimensional point group acquired by the communication unit 110 ( S4 ).

[0099] Next, the details of the mapping process will be described together with the detailed structure of the mapping unit 120. Figure 6 as well as Figure 7 This will be described later.

[0100] Figure 6 This is a block diagram showing an example of the functional structure of the mapping unit. Figure 7 This is a flowchart showing an example of detailed processing of the mapping process.

[0101] like Figure 6 As shown, the mapping unit 120 includes a feature detection module 121 , a feature matching module 122 , a point group registration module 123 , a triangulation module 124 , a position and posture calculation module 125 , and a memory 126 .

[0102] In the mapping unit 120, the description is based on the premise of inputting a two-dimensional image and a three-dimensional point group sensed at a timing corresponding to the timing of capturing the two-dimensional image. That is, the mapping unit 120 maps the captured two-dimensional image and the sensed three-dimensional point group in a corresponding manner at mutually corresponding timings. In addition, the three-dimensional point group obtained at the timing of capturing the two-dimensional image can be either a three-dimensional point group sensed at a timing closest to the timing of capturing the two-dimensional image, or the latest three-dimensional point group sensed before the timing of capturing the two-dimensional image. In addition, when the timing of capturing by the camera 210 and the timing of sensing by the LiDAR 230 are synchronized, the two-dimensional image and the three-dimensional image obtained at the same timing can also be considered as the two-dimensional image and the three-dimensional image obtained at the same timing.

[0103] The feature detection module 121 detects the feature points of each of the multiple two-dimensional images obtained by the communication unit 110 (S11). The feature detection module 121 detects the feature quantity at the feature point. The feature quantity is, for example, ORB (Oriented FAST and Rotated BRIEF), SIFT, DAISY, etc. In addition, the feature quantity can also be represented by a 256-bit data column. In addition, the feature quantity can use the brightness value of each pixel in the two-dimensional image, or can use the color represented by RGB values, etc. The feature quantity of the feature point is an example of an attribute value of the two-dimensional image corresponding to the position on the two-dimensional image. As long as the feature quantity corresponds to the position on the two-dimensional image, it is not limited to the feature quantity of the feature point, as long as it is a feature quantity calculated for each of the multiple regions. The multiple regions are regions constituting the two-dimensional image, and each region can be a pixel of the two-dimensional image, or a block consisting of a collection of multiple pixels.

[0104] The position and posture calculation module 125 obtains an accurate three-dimensional map from the memory 126 or the external memory 140, and uses the obtained accurate three-dimensional map and the three-dimensional point group obtained by the communication unit 110, that is, the three-dimensional point group as the detection result of LiDAR230, to determine the position and posture of LiDAR230 in the accurate three-dimensional map (S12). The position and posture calculation module 125 uses a pattern matching algorithm such as ICP (Iterative Closest Point) to match the three-dimensional point group as the detection result of LiDAR230 with the three-dimensional point group constituting the accurate three-dimensional map. Thus, the position and posture calculation module 125 determines the position and posture of LiDAR230 in the accurate three-dimensional map. In addition, the position and posture calculation module 125 can also obtain an accurate three-dimensional map from an upper-level server outside the three-dimensional point group data generation device 100.

[0105] Next, the position and posture calculation module 125 obtains the position and posture of the LiDAR 230 in the accurate three-dimensional map and the relative position and posture of the camera 210 relative to the LiDAR 230 stored in the memory 250. Then, the position and posture calculation module 125 calculates the position and posture of the camera 210 at the timing when each of the plurality of two-dimensional images is captured, using the obtained accurate position and posture of the LiDAR 230 in the three-dimensional map and the relative position and posture of the camera 210 relative to the LiDAR 230 stored in the memory 250 (S13).

[0106] Next, the feature matching module 122 uses the feature points of each of the multiple two-dimensional images detected by the feature detection module 121 and the position and posture of the camera 210 at the time when each of the multiple two-dimensional images was captured calculated by the position and posture calculation module 125 to match the feature points of each of the pairs of the multiple two-dimensional images, that is, the two two-dimensional images (S14). Figure 5 As shown in "Matching / Triangulation", the feature matching module 122 matches the feature point P1 among the multiple feature points detected by the feature detection module 121 in the two-dimensional image I1 captured by the camera 210 at timing T, and the feature point P2 among the multiple feature points detected by the two-dimensional image I2 captured by the camera 210 at timing T+1 after timing T. For example, timing T+1 may also be the timing of the next capture of timing T. Thus, the feature matching module 122 establishes correspondence between different two-dimensional images for the multiple feature points detected for each of the multiple two-dimensional images. In this way, the feature matching module 122 matches the corresponding feature points in two of the multiple two-dimensional images by using the multiple feature points detected for each of the multiple two-dimensional images, and outputs multiple matched feature point pairs.

[0107] Next, the triangulation module 124 calculates, for each of the plurality of feature point pairs matched in the feature matching module 122, the three-dimensional position corresponding to the position of the matched feature point pair on the two-dimensional image, for example, by triangulation, using the positions of the two feature points constituting the pair on the two-dimensional image and the position and posture of the camera 210 when each of the two two-dimensional images from which the feature points of the pair are obtained (S15). Figure 5 As shown in the "Matching / Triangulation" of FIG. 1 , the triangulation module 124 calculates the three-dimensional position P10 by triangulating the feature point P1 detected in the two-dimensional image I1 and the feature point P2 detected in the two-dimensional image I2. The position and posture of the camera 210 used here are the position and posture of the camera 210 at the time of taking each of the two two-dimensional images that can obtain the feature point pair.

[0108] Next, the point group registration module 123 determines a plurality of three-dimensional points corresponding to the three-dimensional position calculated in the triangulation module 124, i.e., the first three-dimensional point, from among the plurality of three-dimensional points constituting the first three-dimensional point group as an accurate three-dimensional map (S16). The point group registration module 123 may, for example, determine as the first three-dimensional point a three-dimensional point that is closest to one three-dimensional position among the plurality of three-dimensional points constituting the first three-dimensional point group. The point group registration module 123 may, for example, determine as the first three-dimensional point one or more three-dimensional points that are included in a prescribed range based on one three-dimensional position among the plurality of three-dimensional points constituting the first three-dimensional point group. That is, the point group registration module 123 may determine as the first three-dimensional point a plurality of three-dimensional points that satisfy prescribed conditions based on one three-dimensional position as the first three-dimensional point.

[0109] Then, the point group registration module 123 generates a second three-dimensional point group consisting of a plurality of second three-dimensional point groups by adding the feature quantities of the two feature points constituting the pair to the determined plurality of first three-dimensional points for each of the plurality of pairs (S17). In addition, the point group registration module 123 can generate a second three-dimensional point group by (i) adding the feature quantity of one of the two feature points constituting the pair to the determined plurality of first three-dimensional points, or (ii) by adding one feature quantity calculated based on the two feature quantities of the two feature points constituting the pair to generate the second three-dimensional point group. That is, the point group registration module 123 generates a second three-dimensional point group by adding the feature quantities based on the two feature quantities of the plurality of pairs to the determined plurality of first three-dimensional points. In addition, the one feature quantity calculated based on the two feature quantities is, for example, an average value. In addition, the "feature quantity" mentioned here is an example of an attribute value, so in the processing in the point group registration module 123, it can also be applied to other attribute values.

[0110] In steps S16 and S17, the point group registration module 123 performs, for example, Figure 5 As shown in the "Establish Correspondence" of FIG. 1 , by determining the 3D point corresponding to the 3D position P10 obtained by the triangulation module 124, the feature values ​​of the feature points P1 and P2 corresponding to the 3D position P10 are added to the 3D point in the accurate 3D map corresponding to the 3D position P10 to establish correspondence. Thus, a 3D point corresponding to the 3D position P10 is generated. Figure 5 The three-dimensional point group C1 shown includes a plurality of three-dimensional points to which feature values ​​of feature points P1 and P2 are respectively added. That is, the three-dimensional points to which attribute values ​​are added included in the three-dimensional point group C1 are composed of three-dimensional coordinates, the illumination (brightness) in each two-dimensional image of the feature points P1 and P2 as attribute values, and the feature values. That is, it is also possible to add not only the feature values ​​of the feature points to the three-dimensional points, but also the attribute values ​​such as the illumination (brightness) in the two-dimensional image of the feature points or the color such as the RGB value.

[0111] In addition, the point group registration module 123 may also generate a second three-dimensional point by adding a plurality of attribute values ​​to a first three-dimensional point. For example, as described above, the point group registration module 123 may also add the feature values ​​of each of the plurality of feature points detected from the plurality of two-dimensional images as a plurality of attribute values ​​to the first three-dimensional point. In addition, the point group registration module 123 may also add attribute values ​​of different types as a plurality of attribute values ​​to the first three-dimensional point.

[0112] In addition, the memory 126 may also store an accurate three-dimensional map. The memory 126 may store an accurate three-dimensional map in advance or may store an accurate three-dimensional map received from a superior server by the communication unit 110. The memory 126 may be implemented by a non-volatile storage device such as a flash memory or a HDD (Hard Disk Drive).

[0113] In this way, the point group registration module 123 generates a second three-dimensional point group including a plurality of second three-dimensional points that are a plurality of three-dimensional points to which feature amounts of feature points are added, by performing the mapping process of step S4.

[0114] Back to Figure 4 After step S4, the encoding unit 130 generates a coded stream by encoding the second three-dimensional point group generated in the mapping unit 120 (S5).

[0115] Then, the communication unit 110 transmits the encoded stream generated in the encoding unit 130 to the vehicle 300 ( S6 ).

[0116] In the vehicle 300 , the communication unit 330 obtains the coded stream from the three-dimensional point cloud data generating device 100 ( S7 ).

[0117] Next, the decoding unit 340 obtains the second 3D point group by decoding the coded stream obtained by the communication unit 330 (S8). That is, the vehicle 300 obtains a 3D point group including a plurality of 3D points to which attribute values ​​of 2D images obtained by capturing 3D objects are respectively added in advance.

[0118] Next, the position estimation unit 350 uses the two-dimensional image obtained by the camera 310 and the three-dimensional point group with attribute values ​​attached obtained by decoding in the decoding unit 340 to estimate the position and posture of the camera 310 in the three-dimensional point group, thereby estimating the position and posture of the vehicle 300 (S9).

[0119] The details of the position estimation process performed by the position estimation unit 350 will be described.

[0120] like Figure 3 As shown, the position inference unit 350 has a feature detection module 351, a feature matching module 352 and a memory 353.

[0121] The feature detection module 351 obtains a second two-dimensional image of the surroundings of the vehicle 300 captured by the camera 310 of the vehicle 300. Then, the feature detection module 351 detects a plurality of second attribute values ​​from the obtained second two-dimensional image, the second attribute values ​​being attribute values ​​of the second two-dimensional image corresponding to positions on the second two-dimensional image. The processing in the feature detection module 351 is the same as the processing in the feature detection module 121 of the mapping unit 120 in the vehicle 200. In addition, the processing in the feature detection module 351 may not be the same as all the processing in the feature detection module 121 of the mapping unit 120. For example, in the case where the feature detection module 121 detects a plurality of types of first attribute values ​​as attribute values, the feature detection module 351 may detect one or more attribute values ​​included in the plurality of types as second attribute values.

[0122] Next, the feature matching module 352 determines one or more fifth three-dimensional points corresponding to the second attribute value among the plurality of second three-dimensional points for each of the plurality of second attribute values ​​detected by the feature detection module 351, thereby generating one or more combinations consisting of the second attribute value and one or more fifth three-dimensional points. Specifically, the feature matching module 352 determines one or more second three-dimensional points to which the attribute value closest to the attribute value corresponding to the two-dimensional position among the plurality of second three-dimensional points is attached as the fifth three-dimensional point for each of the plurality of two-dimensional positions on the two-dimensional image corresponding to the plurality of attribute values ​​detected by the feature detection module 351. Thus, the feature matching module 352 generates one or more combinations consisting of the two-dimensional position on the two-dimensional image and one or more fifth three-dimensional points. The feature matching module 352 obtains the position and posture of the camera 310 relative to the vehicle 300 from the memory 353. The feature matching module 352 calculates the position and posture of the vehicle 300 using the determined one or more combinations and the position and posture of the camera 310 relative to the obtained vehicle 300. Furthermore, the more combinations the feature matching module 352 generates, the more accurately the position and posture of the vehicle 300 can be calculated.

[0123] The memory 353 may store the position and posture of the camera 310 relative to the vehicle 300. The memory 353 may also store the second three-dimensional point group obtained by decoding by the decoding unit 340. The memory 353 is implemented by a nonvolatile storage device such as a flash memory or a HDD (Hard Disk Drive).

[0124] According to the three-dimensional point group data generation method of the present embodiment, the three-dimensional point group data generation device 100 generates a second three-dimensional point group including a second three-dimensional point, which is obtained by adding a feature amount of a feature point corresponding to a position on a two-dimensional image obtained by using a camera 210, among a plurality of three-dimensional points constituting the first three-dimensional point group obtained by using a distance sensor such as LiDAR 230, to the three-dimensional point corresponding to the position on the two-dimensional image. Therefore, the vehicle 300 as a position estimation device for estimating its own position can efficiently estimate its own position by comparing the feature amount of the feature point corresponding to the position of the captured two-dimensional image with the feature amount of the feature point added to each second three-dimensional point of the second three-dimensional point group obtained from the three-dimensional point group data generation device 100, even if it does not sense the three-dimensional objects around the vehicle 300.

[0125] In addition, in the method for generating three-dimensional point cloud data of the present embodiment, a feature point pair is determined by matching the feature points of each of the two two-dimensional images, and the position corresponding to each of the two-dimensional images of the feature points of the pair is used to determine the three-dimensional point corresponding to the position, and the feature amount of the two feature points constituting the pair is added to the determined three-dimensional point. Therefore, the three-dimensional point corresponding to the feature point can be determined with high accuracy.

[0126] In addition, in the three-dimensional point group data generation method of the present embodiment, in the generation of the second three-dimensional point group, the second three-dimensional point is generated by adding the feature amount of each of the plurality of feature points to the determined first three-dimensional point. In this way, since the number of feature points added to the first three-dimensional point is set to be a plurality, in the position estimation of the vehicle 300, the feature points of the two-dimensional image corresponding to the position on the two-dimensional image obtained for the position estimation and the first three-dimensional point can be established with high accuracy.

[0127] In addition, in the three-dimensional point group data generation method of the present embodiment, the multiple attribute values ​​added to a first three-dimensional point in the generation of the second three-dimensional point group are attribute values ​​detected from multiple two-dimensional images. Here, a first three-dimensional point is a point that can be photographed from multiple viewpoints. In addition, in the position estimation in the vehicle 300, the camera 310 captures two-dimensional images at different positions. Therefore, the two-dimensional image obtained by the camera 310 is taken from multiple different viewpoints. Thus, by adding multiple attribute values ​​detected from multiple two-dimensional images, in the position estimation, even if two-dimensional images captured from multiple different viewpoints are used, it is possible to easily establish a correspondence between the attribute values ​​obtained from each two-dimensional image and the first three-dimensional point. Therefore, position estimation can be easily performed.

[0128] In addition, in the method for generating three-dimensional point group data of the present embodiment, the multiple attribute values ​​added to one first three-dimensional point in generating the second three-dimensional point group are attribute values ​​of different attribute types. In this way, since the attribute values ​​added to one first three-dimensional point are attribute values ​​of different types, in the position estimation of the vehicle 300, the attribute values ​​of the two-dimensional image corresponding to the position on the two-dimensional image obtained for position estimation and the first three-dimensional point can be established with high accuracy.

[0129] In addition, according to the position inference method of the present embodiment, the attribute values ​​of the two-dimensional image obtained by photographing the three-dimensional object are pre-attached to the three-dimensional point group. Therefore, even if the three-dimensional point group is not re-sensed, as long as the second two-dimensional image around the vehicle 300 is re-photographed, the vehicle 300's own position can be efficiently inferred by comparing the second attribute value corresponding to the position of the captured second two-dimensional image and the combination of one or more fifth three-dimensional points corresponding to the second attribute value, and the first attribute value of each three-dimensional point attached to the three-dimensional point group.

[0130] (Variation Example)

[0131] (Variant 1)

[0132] In the three-dimensional point group data generating device 100 of the above-mentioned embodiment, importances can also be calculated for each of the plurality of second three-dimensional points included in the second three-dimensional point group, the calculated importances can be added to the corresponding second three-dimensional points, and a third three-dimensional point group including the plurality of third three-dimensional points obtained by adding can be generated. In addition, the three-dimensional point group data generating device 100 can also filter the number of third three-dimensional points to be sent to the vehicle 300 from the plurality of third three-dimensional points included in the third three-dimensional point group according to the added importances, and send the filtered number of third three-dimensional points to the vehicle 300.

[0133] use Figure 8 Describe the action. Figure 8 is a flowchart showing an example of the operation of the three-dimensional point group data generating device in Modification Example 1. Figure 4 The process is performed in step S5 in the timing diagram.

[0134] In the three-dimensional point group data generating device 100, the point group registration module 123 of the mapping unit 120 also calculates the importance of the second three-dimensional point based on the attribute value attached to the second three-dimensional point for each of the more than one second three-dimensional points (S21). Then, the point group registration module 123 attaches the calculated importance to the second three-dimensional point for each of the more than one second three-dimensional points, thereby generating a third three-dimensional point group, wherein the third three-dimensional point group includes more than one three-dimensional point having an attribute value and an importance. Fig. 9The importance calculation process is described in detail.

[0135] Next, the encoding unit 130 of the three-dimensional point group data generating device 100 generates a coded stream using the third three-dimensional point group (S22). Fig.10 and Fig.11 The details of the encoding stream generation process are described.

[0136] Fig. 9 This is a flowchart showing an example of detailed processing of calculating the importance.

[0137] The point group registration module 123 performs a calculation process of importance for each of the plurality of second three-dimensional points included in the generated second three-dimensional point group. The following describes the process performed on one second three-dimensional point. In the calculation process of importance, the same process is performed on all the second three-dimensional points one by one.

[0138] First, the point group registration module 123 calculates the number of two-dimensional images in which the second three-dimensional point to be processed is visible (S31). Alternatively, the point group registration module 123 calculates the number by counting the number of two-dimensional images in which the second three-dimensional point to be processed is reflected in a plurality of two-dimensional images.

[0139] In addition, the point group registration module 123 calculates a matching error in matching a plurality of feature points associated with the second three-dimensional point to be processed ( S32 ). The point group registration module 123 may obtain the matching error from the feature matching module 122 .

[0140] Furthermore, the point cloud registration module 123 calculates a matching error in matching between a second three-dimensional point to be processed (ie, a three-dimensional point in an accurate three-dimensional map) and a feature point associated with the second three-dimensional point ( S33 ).

[0141] The point group registration module 123 calculates the importance of the second three-dimensional point as the processing object using the number of two-dimensional images in which the second three-dimensional point as the processing object can be seen, the matching error between multiple feature points of the two-dimensional images, and the matching error between the three-dimensional map and the feature point calculated in steps S31 to S33 (S34). For example, the greater the number of two-dimensional images in which the second three-dimensional point as the processing object can be seen, the point group registration module 123 calculates the importance as a larger value. In addition, for example, the greater the matching error between multiple feature points of the two-dimensional image, the point group registration module 123 calculates the importance as a smaller value. In addition, for example, the greater the matching error between the three-dimensional map and the feature point, the point group registration module 123 calculates the importance as a smaller value. In this way, the importance is an indicator indicating that the larger the value, the more important it is.

[0142] The point cloud registration module 123 obtains the three-dimensional coordinates of the second three-dimensional point as the processing target ( S35 ), and obtains the feature points of the two-dimensional image added to the second three-dimensional point as the processing target ( S36 ).

[0143] Then, the point group registration module 123 generates a third three-dimensional point by adding the calculated importance and feature points to the obtained three-dimensional coordinates (S37). In addition, in steps S35 and S36, the three-dimensional coordinates and feature points are respectively obtained from the second three-dimensional point as the processing object, but not limited to this. In step S37, the third three-dimensional point can also be generated by adding the calculated importance to the second three-dimensional point as the processing object.

[0144] The point group registration module 123 executes the processing of steps S31 to S37 for all the second three-dimensional points to generate a three-dimensional point group to which importance is further added, that is, a third three-dimensional point group. Fig.10 As shown, the third three-dimensional point includes three-dimensional coordinates, feature points of each of the N two-dimensional images in which the second three-dimensional point can be seen, and importance.

[0145] Here, as long as the feature values ​​among the N feature quantities are close, they can also be unified. For example, if the difference between the values ​​of the two feature quantities F0 and F1 is below the threshold, the value can also be unified as one feature quantity of the average value of F0 and F1. In this way, the amount of data of the second three-dimensional point group can be reduced. In addition, an upper limit can be set for the value of the number N of feature quantities that can be assigned to the second three-dimensional point. For example, when the number of feature quantities is larger than N, the value of the feature quantity can also be used to select the upper N, and the selected N feature quantities can be assigned to the three-dimensional point.

[0146] Next, we will refer to Fig.11 and Fig.12 The details of the encoding process and the detailed structure of the encoding unit 130 are described.

[0147] Fig.11 This is a block diagram showing an example of the functional structure of the encoding unit. Fig.12 This is a flowchart showing an example of detailed processing of the encoding process.

[0148] like Fig.11 As shown, the encoding unit 130 has a feature sorting module 131, a feature combining module 132, a memory 133 and an entropy encoding module 134.

[0149] In the encoding process of the encoding unit 130 , a plurality of third three-dimensional points included in the third three-dimensional point group are processed.

[0150] The feature ranking module 131 arranges the plurality of third three-dimensional points included in the third three-dimensional point group in descending order of importance assigned to each of the plurality of third three-dimensional points ( S41 ).

[0151] Next, the feature combination module 132 starts loop 1 ( S42 ) of executing the following steps S43 and S44 for each of the plurality of third three-dimensional points. The feature combination module 132 executes loop 1 in descending order of importance.

[0152] The feature combination module 132 determines whether the importance added to the third three-dimensional point as the processing object exceeds a threshold value (S43). In addition, the threshold value is a value received from the vehicle 300. The threshold value can also be set to a different value according to the specifications of the vehicle 300. That is, the greater the information processing capability and / or detection capability of the vehicle 300, the greater the threshold value can be set to.

[0153] When determining that the importance assigned to the third three-dimensional point to be processed exceeds the threshold value (Yes in S43 ), the feature combination module 132 executes the process of step S44 .

[0154] On the other hand, when the feature combination module 132 determines that the importance assigned to the third three-dimensional point to be processed is less than the threshold value (No in S43), the feature combination module 132 executes the process of step S45. Thus, loop 1 ends.

[0155] In step S44, the feature combination module 132 adds the third three-dimensional point to be processed as encoding target data (S44). After step S44, the feature combination module 132 performs loop 1 by taking the next third three-dimensional point as the processing target.

[0156] In this way, the feature combination module 132 executes steps S43 and S44 to extract a plurality of fourth three-dimensional points to which importance exceeding the threshold is added from the plurality of third three-dimensional points.

[0157] In this case, the loop 1 is executed for all the third three-dimensional points, and when the determination result in step S43 is "No", the loop 1 is repeated for the next third three-dimensional point.

[0158] In step S45, the entropy coding module 134 performs entropy coding on the plurality of third three-dimensional points as the coding target data, and generates a coded stream (S45). For example, the entropy coding module 134 may generate a coded stream by arithmetic coding by representing the coding target data in an octree structure and binarizing it.

[0159] The generated coded stream is Figure 4In step S6, it is sent to the vehicle 300 that sent the threshold.

[0160] According to the method for generating three-dimensional point group data of variation 1, in the generation of the second three-dimensional point, further, (i) for each of the plurality of second three-dimensional points, the importance of the second three-dimensional point is calculated based on the attribute value attached to the second three-dimensional point, and (ii) the calculated importance is attached to the second three-dimensional point, thereby generating a third three-dimensional point group, the third three-dimensional point group including one or more third three-dimensional points each having an attribute value and an importance. Therefore, for example, the more important the third three-dimensional point is, the more preferentially it can be used, so the amount of three-dimensional point group data used in the processing can be adjusted so as not to adversely affect the accuracy of the processing.

[0161] In addition, in the three-dimensional point group data generation method of modification example 1, a threshold is also received from the vehicle 300 as a client device, a plurality of fourth three-dimensional points to which importance exceeding the received threshold is added are extracted from one or more third three-dimensional points, and a fourth three-dimensional point group including the extracted plurality of fourth three-dimensional points is sent to the client device. Therefore, the data volume of the three-dimensional point group data to be sent can be modulated according to the request of the vehicle 300.

[0162] Furthermore, the encoding unit 130 of the three-dimensional point cloud data generating device 100 in the above-mentioned modification 1 obtains the threshold value from the vehicle 300, but the number of three-dimensional points may be obtained from the vehicle 300. In this case, the encoding unit 130 may also encode the third three-dimensional point of the number of three-dimensional points obtained in descending order of importance as encoding target data.

[0163] (Variant 2)

[0164] In the above-mentioned modification 1, the 3D point group data generating device 100 receives a threshold value from the vehicle 300 as a client device, and excludes the third 3D points whose importance is below the threshold value from the encoding target data, but it is not limited to this, and the encoding stream can also be generated by encoding all the plurality of third 3D points as encoding target data. That is, the 3D point group data generating device 100 can also send the data encoded for all the plurality of third 3D points to the vehicle 300.

[0165] In this case, the vehicle 300 decodes the received coded stream to obtain a plurality of third three-dimensional points. The vehicle 300 may then use a third three-dimensional point whose importance exceeds a threshold value among the plurality of third three-dimensional points to estimate the vehicle 300's own position.

[0166] In addition, when the communication unit 330 obtains all the three-dimensional points, the decoding unit 340 of the vehicle 300 can also preferentially decode the three-dimensional points with greater importance from all the three-dimensional points. As a result, the decoding unit 340 can decode the required number of three-dimensional points, thereby reducing the processing load spent on the decoding process. In addition, in this case, if the coded stream is encoded in order from large to small importance, the decoding unit 340 decodes the coded stream, and the decoding process can be stopped at the moment when the three-dimensional points with importance below the threshold are decoded, or the decoding process can be stopped at the moment when the required number of three-dimensional points are obtained. As a result, even if the three-dimensional point group data generation device 100 does not generate a coded stream for each vehicle 300, the processing load spent on the decoding process can be reduced. Therefore, the processing load spent on the process of generating the coded stream by the three-dimensional point group data generation device 100 can be reduced.

[0167] In addition, the vehicle 300 may not use the plurality of third three-dimensional points for the process of estimating its own position, but may use them for the process of reconstructing the three-dimensional image of the three-dimensional map. That is, the vehicle 300 as a client device may also switch the threshold value according to the purpose implemented by the vehicle 300. For example, when the vehicle 300 estimates its own position, it is determined that only three-dimensional points with high importance are needed, and the threshold value is set to the first threshold value. When the client draws a map, it is determined that even three-dimensional points with low importance are needed, and the threshold value is set to the second threshold value which is smaller than the first threshold value.

[0168] (other)

[0169] The mapping unit 120 may add the three-dimensional position calculated by matching the feature points in the two two-dimensional images to the accurate three-dimensional map as a three-dimensional point having the feature amount of the feature point.

[0170] The mapping unit 120 may also match feature points in one two-dimensional image with three-dimensional points of an accurate three-dimensional map, and add feature quantities of the feature points to the three-dimensional points that match the feature points. In this case, the feature points may be back-projected into a three-dimensional space or projected into a two-dimensional image using the posture of the camera 210, thereby matching the feature points with three-dimensional points of an accurate three-dimensional map. That is, the mapping unit 120 determines a three-dimensional point that is close to a three-dimensional position determined by matching a pair of feature points of two two-dimensional images in a plurality of two-dimensional images, thereby matching the feature points with the three-dimensional points, but is not limited thereto, and feature points of one two-dimensional image may also be matched with the three-dimensional points. In other words, the mapping unit 120 may also, for each of the one or more detected attribute values, (i) determine one or more first three-dimensional points corresponding to the position on the two-dimensional image of the attribute value among the multiple three-dimensional points constituting the first three-dimensional point group, and (ii) add the attribute value to the determined one or more first three-dimensional points, thereby generating a second three-dimensional point group, wherein the second three-dimensional point group includes one or more second three-dimensional points respectively having the attribute value.

[0171] In addition, the coding of the three-dimensional points is not limited to entropy coding, and any coding may be applied. For example, the three-dimensional point group as the coding target data may be coded in an octree structure (octree coding).

[0172] The three-dimensional point group data generating device 100 is a server different from the vehicle 200, but may be provided in the vehicle 200. That is, the processing of the three-dimensional point group data generating device 100 may also be performed by the vehicle 200. In this case, the second three-dimensional point group or the third three-dimensional point group obtained by the vehicle 200 may also be sent to the upper server, and the upper server may collect the second three-dimensional point group or the third three-dimensional point group of various places from a plurality of vehicles 200.

[0173] In addition, in each of the above embodiments, each component is composed of dedicated hardware, but it can also be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the three-dimensional point group data generation method or position estimation method of each of the above embodiments is the following program.

[0174] That is, the program causes a computer to execute a three-dimensional point group generation method for generating a three-dimensional point group including a plurality of three-dimensional points using a processor, in which a two-dimensional image obtained by photographing a three-dimensional object using a camera and a first three-dimensional point group obtained by sensing the three-dimensional object using a distance sensor are obtained, and one or more attribute values ​​of the two-dimensional image corresponding to positions on the two-dimensional image are detected from the obtained two-dimensional image, and for each of the one or more attribute values ​​detected, (i) one or more first three-dimensional points corresponding to the positions of the attribute values ​​on the two-dimensional image among the plurality of three-dimensional points constituting the first three-dimensional point group are determined, and (ii) the attribute values ​​are added to the one or more determined first three-dimensional points, thereby executing a three-dimensional point group data generation method, which generates a second three-dimensional point group including one or more second three-dimensional points each having the attribute values.

[0175] In addition, the program causes a computer to execute a position inference method for inferring the current position of a moving body using a processor, in which a three-dimensional point group including a plurality of three-dimensional points to which first attribute values ​​as attribute values ​​of a first two-dimensional image obtained by photographing a three-dimensional object are respectively pre-attached is obtained, a second two-dimensional image of the surroundings of the moving body photographed by a camera possessed by the moving body is obtained, and from the obtained second two-dimensional image, one or more second attribute values ​​as attribute values ​​of the second two-dimensional image corresponding to a position on the second two-dimensional image are detected, and for each of the one or more second attribute values ​​detected, one or more fifth three-dimensional points corresponding to the second attribute value are determined from the plurality of three-dimensional points, thereby generating one or more combinations consisting of the second attribute value and the one or more fifth three-dimensional points, the position and posture of the camera relative to the moving body are obtained from a storage device, and the position and posture of the moving body are calculated using the one or more generated combinations and the obtained position and posture of the camera.

[0176] In the above, one or more forms of the three-dimensional point group data generation method, position inference method, three-dimensional point group data generation device, and position inference device of the present invention are described based on the embodiments, but the present disclosure is not limited to the embodiments. As long as it does not deviate from the main purpose of the present disclosure, various deformation methods that can be thought of by those skilled in the art are implemented in the present embodiment, and the method of combining the constituent elements in different embodiments to construct the method can also be included in the scope of the present disclosure.

[0177] Industrial Applicability

[0178] The present disclosure is useful as a three-dimensional point group data generation method, a position estimation method, a three-dimensional point group data generation device, and a position estimation device that can achieve further improvements.

[0179] Description of the attached figure

[0180] 1. Position estimation system

[0181] 100 3D point group data generation device

[0182] 110 Communications Department

[0183] 120 Mapping Department

[0184] 121, 351 feature detection module

[0185] 122, 352 feature matching module

[0186] 123 point group registration module

[0187] 124 Triangulation Module

[0188] 125 position and posture calculation module

[0189] 126, 133, 250, 353 memory

[0190] 130 Coding Department

[0191] 131 Feature sorting module

[0192] 132 Feature combination module

[0193] 134 Entropy Coding Module

[0194] 140, 370 external memory

[0195] 200, 300 vehicles

[0196] 210, 310 cameras

[0197] 220, 240, 320 acquisition department

[0198] 230LiDAR

[0199] 260, 330 Communications Department

[0200] 340 Decoding Department

[0201] 350 Position Estimation Unit

[0202] 360 Control Department

[0203] 400 Communication Network

[0204] 410 Base Station

Claims

1. A method for generating point group data, executed using a processor, wherein: The point group data generation method comprises: Obtaining sensing results from the distance sensor; and The sensing result is used to generate second point group data, the second point group data including three-dimensional coordinates and one or more attribute values ​​of each of the plurality of points.

2. The point group data generation method according to claim 1, wherein: The one or more attribute values ​​include a value indicating an importance for selecting or for reordering a plurality of points to be encoded.

3. The point group data generation method according to claim 1, wherein: The one or more attribute values ​​include a value representing an importance for selecting a point to be transmitted.

4. The point group data generation method according to claim 1, wherein: Also includes: Acquire first image data from an image sensor; as well as The first feature quantity of each of the plurality of points is calculated based on the first image data, wherein: The one or more attribute values ​​include a plurality of the first feature quantities.

5. The point group data generation method according to claim 4, wherein: Also includes: acquiring second image data from the image sensor; as well as The second feature quantity of each of the plurality of points is calculated based on the second image data, wherein: The one or more attribute values ​​include a plurality of the second feature quantities.

6. The point group data generation method according to claim 1, wherein: Also includes: Acquire first image data from an image sensor; as well as Using stored information indicating relative positions and directions of the distance sensor and the image sensor, a three-dimensional position corresponding to the two-dimensional position in the first image data is calculated.

7. A point group data generating device, comprising: A distance sensor and a processor, wherein the processor is configured as follows: Acquire a sensing result from the distance sensor; and The sensing result is used to generate second point group data, the second point group data including three-dimensional coordinates and one or more attribute values ​​of each of the plurality of points.

8. The point cloud data generating device according to claim 7, wherein: The one or more attribute values ​​include a value indicating an importance for selecting or for reordering a plurality of points to be encoded.

9. The point cloud data generating device according to claim 7, wherein: The one or more attribute values ​​include a value representing an importance for selecting a point to be transmitted.

10. The point group data generating device according to claim 7, further comprising an image sensor, wherein: The processor is further configured as: acquiring first image data from the image sensor; and The first feature quantity of each of the plurality of points is calculated based on the first image data, wherein: The one or more attribute values ​​include a plurality of the first feature quantities.

11. The point cloud data generating device according to claim 10, wherein: The processor is further configured as: acquiring second image data from the image sensor; and The second feature quantity of each of the plurality of points is calculated based on the second image data, wherein: The one or more attribute values ​​include a plurality of the second feature quantities.

12. The point group data generating device according to claim 7, further comprising an image sensor, wherein: The processor is further configured as: acquiring first image data from the image sensor; and Using stored information indicating relative positions and directions of the distance sensor and the image sensor, a three-dimensional position corresponding to the two-dimensional position in the first image data is calculated.

Images