Synthesis method for point group data and device thereof

Abstract

To provide a synthesis method for point group data and its device that accurately extracts and synthesizes first point group data and second point group data for the same road environment.SOLUTION: A point group data synthesis method uses a point group data synthesizer comprising a point group data acquisition part, an image data comparison part, and a point group data synthesis part to synthesize point group data obtained by the point group data acquisition part of a measurement vehicle comprising a point group data acquisition part and an image data acquisition part. The point group data extraction part extracts first point group data and second point group data that are synthesis objects, and the image data comparison part compares first image data acquired in synchronization with the acquisition of the first point group data with second image data acquired in synchronization with the acquisition of the second point group data and the point group data synthesis part synthesizes the first point group data and the second point group data based on the comparison results of the first image data and the second image data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method and device for synthesizing point cloud data. [Background technology]

[0002] Efforts are underway to acquire three-dimensional data of road environments for use in autonomous driving and other applications. To obtain 3D data of the road environment, a measurement vehicle equipped with a 3D laser scanner is generally used. This measurement vehicle travels along the road to be measured, operates the 3D laser scanner to measure the road surface and the roadside environment in 3D, and converts the measurement results into point cloud data.

[0003] When acquiring point cloud data while driving a measuring vehicle, positional deviations generally occur if the driving history (acquisition time) is different, even if the data is acquired at the same location. That is, different coordinates are assigned to point data corresponding to the same fixed feature in the road environment. In other words, the same coordinates may be assigned to point data corresponding to different fixed features in the road environment. Such positional deviations are particularly likely to occur in areas with unstable GNSS reception environments, such as urban areas and mountainous areas.

[0004] Therefore, a method has been proposed for aligning point cloud data acquired at the same location. The aligned point cloud data are then synthesized. Please refer to Patent Documents 1 and 2 as prior art documents related to this invention. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2021-170049 A [Patent Document 2] JP 2012-242318 A DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0006] The alignment of point cloud data is premised on the fact that the target point cloud data is acquired at the same point, but in an environment where positional deviation is likely to occur as described above, the same coordinates may be assigned to point data constituting point cloud data acquired at different points. For example, if a first road and a second road exist in spatially close positions, when comparing the first point cloud data acquired at the first point on the first road with the second point cloud data acquired at the second point on the second road (close to the first point), the same coordinates may be assigned to the point data constituting both. Point cloud data having point data with the same coordinates can be combined after alignment processing, but combining point cloud data acquired on different roads as described above does not make sense and should be avoided. [Means for solving the problem]

[0007] The present invention has been made to solve the above problems, and a first aspect thereof is defined as follows. A method for synthesizing point cloud data, comprising: using a point cloud data synthesizing device including a point cloud data extracting unit, an image data comparing unit, and a point cloud data synthesizing unit; synthesizing point cloud data obtained by a point cloud data acquiring unit of a measuring vehicle including a point cloud data acquiring unit and an image data acquiring unit, causing the point cloud data extraction unit to extract first point cloud data and second point cloud data to be combined, wherein the first point cloud data is acquired in a first travel history of the measurement vehicle, the second point cloud data is acquired in a second travel history of the measurement vehicle, and the first point cloud data and the second point cloud data have point data with the same coordinates; The image data comparison unit is caused to compare first image data acquired in synchronization with acquisition of the first point cloud data with second image data acquired in synchronization with acquisition of the second point cloud data, wherein the first image data and the second image data are acquired by the image data acquisition unit. causing the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data based on a comparison result between the first image data and the second image data; Methods for merging point cloud data.

[0008] According to the first aspect of the point cloud data synthesis method defined in this manner, even if the first point cloud data and the second point cloud data are extracted as candidates for synthesis because they have point data with the same coordinates, if the first image acquired in synchronization with the first point cloud data and the second image acquired in synchronization with the second point cloud data are different, the image will be removed from the candidates for synthesis as it has been acquired using a driving history on a different road (although close to each other) than the first point cloud data and the second point cloud data.

[0009] In addition, the coordinates (X, Y, Z) assigned to each point data constituting the point cloud data include not only physically identical coordinates, but also closely spaced coordinates that are the subject of point cloud coordinate alignment. If the first image data and the second image data are identical or similar, the first point cloud data and the second point cloud data are combined by the point cloud data combination unit as being obtained in a driving history of traveling the same road.

[0010] A second aspect of the present invention is defined as follows. In the synthesis method defined in the first aspect, the image data comparison unit includes an AI processing unit and a classified scene comparison unit, and the AI ​​processing unit has a function of classifying the image data acquired by the image data acquisition unit into a plurality of predetermined classified scenes; The first image data is input to the AI ​​processing unit to generate a first classified scene, and the second image data is input to the AI ​​processing unit to generate a second classified scene; The classified scene comparison unit causes the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data when the first classified scene and the second classified scene are in the same classification.

[0011] According to the second aspect of the point cloud data synthesis method defined in this manner, by performing AI processing on the first image data and the second image data, it is possible to more accurately determine the differences and similarities between the two scenes. In this second phase, the accuracy of AI processing is ensured by classifying the acquired image data into predetermined classification scenes. The burden on AI processing can also be reduced by limiting the number of classification scenes.

[0012] Examples of the classified scenes include (1) between buildings, (2) under an overpass (crossing), (3) under an overpass (parallel), (4) few feature points (highway), and (5) tunnel. These classified scenes can be selected arbitrarily. That is, multiple images (acquired by the image data acquisition device of the measurement vehicle) representative of each classified scene are prepared, and the AI ​​processing unit learns from these images as teacher data. By inputting image data actually acquired on the road to be measured to the AI ​​processing unit, the input image is assigned to one or more classified scenes.

[0013] Since image data captured at the same coordinates on the same road should essentially be classified into the same classification scene, if the first classification scene and the second classification scene are the same classification scene, it is presumed that the first image data and the second image data captured the same road environment, and thus it is possible to allow the first point cloud data and the second point cloud data to be combined.

[0014] A third aspect of the present invention is an apparatus for executing the point cloud data synthesis method defined in the first aspect, and is defined as follows. A point cloud data synthesis device including a point cloud data extraction unit, an image data comparison unit, and a point cloud data synthesis unit, and a point cloud data synthesis method is carried out to synthesize point cloud data obtained by the point cloud data acquisition unit of a measuring vehicle including a point cloud data acquisition unit and an image data acquisition unit, causing the point cloud data extraction unit to extract first point cloud data and second point cloud data to be combined, wherein the first point cloud data is acquired in a first travel history of the measurement vehicle, the second point cloud data is acquired in a second travel history of the measurement vehicle, and the first point cloud data and the second point cloud data have point data with the same coordinates; The image data comparison unit is caused to compare first image data acquired in synchronization with acquisition of the first point cloud data with second image data acquired in synchronization with acquisition of the second point cloud data, wherein the first image data and the second image data are acquired by the image data acquisition unit. causing the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data based on a comparison result between the first image data and the second image data; Point cloud data synthesis device.

[0015] A fourth aspect of the present invention is a program applied to a computer device of the point cloud data synthesis device defined in the third aspect, and is defined as follows. A program for executing point cloud data synthesis, which is applied to a computer device of a point cloud data synthesis device comprising a point cloud data extraction unit, an image data comparison unit, and a point cloud data synthesis unit, and which synthesizes point cloud data obtained by the point cloud data acquisition unit of a measuring vehicle comprising a point cloud data acquisition unit and an image data acquisition unit, causing the point cloud data extraction unit to extract first point cloud data and second point cloud data to be combined, wherein the first point cloud data is acquired in a first travel history of the measurement vehicle, the second point cloud data is acquired in a second travel history of the measurement vehicle, and the first point cloud data and the second point cloud data have point data with the same coordinates; The image data comparison unit is caused to compare first image data acquired in synchronization with acquisition of the first point cloud data with second image data acquired in synchronization with acquisition of the second point cloud data, wherein the first image data and the second image data are acquired by the image data acquisition unit. causing the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data based on a comparison result between the first image data and the second image data; A program for a computer device of a point cloud data synthesis device. [Brief description of the drawings]

[0016] [Figure 1]FIG. 1 is a block diagram showing the configuration of a point cloud data synthesis device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flow chart illustrating the operation of the device of FIG. [Diagram 3] FIG. 3 is a block diagram showing a configuration of an image data comparison unit according to another embodiment. [Figure 4] FIG. 4 shows the training image data of the classification scene. [Diagram 5] FIG. 5 is a block diagram showing the hardware configuration of a computer device applied to the synthesizing device of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration of a point cloud data synthesis device 1 according to an embodiment of the present invention. This point cloud data synthesis device 1 comprises a measuring vehicle data storage unit 20, a point cloud data extraction unit 30, a point cloud data acquisition coordinate specification unit 35, an image data comparison unit 40, and a point cloud data synthesis unit 50.

[0018] The measurement vehicle data storage unit 20 includes a point cloud data storage unit 21, a coordinate data storage unit 23, and an image data storage unit 25. The point cloud data storage unit 21 stores point cloud data acquired by the point cloud data acquisition unit 11 of the measurement vehicle 10. The coordinate data storage unit 23 stores coordinates identified by the coordinate identification unit 13 of the measurement vehicle 10 together with the time of the coordinates. The image data storage unit 25 stores image data acquired by the image data acquisition unit 15 of the measurement vehicle 10.

[0019] The measurement vehicle 10 is equipped with a general-purpose point cloud data acquisition unit 11, a coordinate identification unit 13, and an image data acquisition unit 15. A 3D laser scanner is used for the point cloud data acquisition unit 11. The coordinate identification unit 13 identifies the position coordinates of the measurement vehicle 10 together with the time using a GPS or the like. A wide-angle visible light camera is used for the image data acquisition unit 15, and it is preferable that the image data acquisition unit 15 and the point cloud data acquisition unit 11 are arranged so as to acquire the road environment in the same direction (for example, the traveling direction of the measurement vehicle 10). The acquisition of point cloud data by the point cloud data acquisition unit 11 and the acquisition of image data by the image data acquisition unit 15 are synchronized. That is, at substantially the same time and at the same coordinates, point cloud data and image data are acquired for the road environment in the same direction.

[0020] At this time, the time when the point cloud data was acquired is linked to the coordinates specified by the coordinate specifying unit 13. Similarly, the time when the image data was acquired is also linked to the coordinates specified by the coordinate specification unit 13 . The coordinate data storage unit 23 stores the coordinates of the measuring vehicle 10 together with the time. Based on the coordinate data of the measuring vehicle stored in the coordinate data storage unit 23, a travel history corresponding to the road links is formed by a well-known method.

[0021] The point cloud data extraction unit 30 extracts point cloud data to be synthesized from the point cloud data stored in the point cloud data storage unit 21. The first point cloud data and the second point cloud data to be synthesized are extracted from different driving histories. When the measurement vehicle is driven on the road to be measured (first driving history), if there are no obstacles (oncoming vehicles, parked vehicles, people, etc.) in the path of the laser beam, point cloud data that accurately represents the road environment can be obtained. However, when an obstacle is present, for example, the road edge (ditch) or road signs may not be reflected in the point cloud data. Therefore, it is necessary to generate point cloud data in a state where no obstacles are present by running the measurement vehicle again (or multiple times) on the road to be measured (second driving history) and combining the later acquired point cloud data with the earlier acquired point cloud data.

[0022] Therefore, the first point cloud data obtained from the first travel history is determined, and the second point cloud data to be combined with the first point cloud data is extracted from the second travel history. Any method for extracting the second point cloud data may be used, but for example, the point cloud data stored in the point cloud data extracting unit 30 may be extracted based on the maximum proportion of point data with the same or similar coordinates when each point data constituting the first point cloud data is compared.

[0023] The point cloud data acquisition coordinate specifying unit 35 refers to the data in the coordinate data storage unit 23 and specifies the acquisition coordinates and acquisition time of the extracted first point cloud data and the acquisition coordinates and acquisition time of the extracted second point cloud data. The image data comparison unit 40 extracts the first image data and the second image data corresponding to the acquisition coordinates and acquisition time of the first point cloud data and the acquisition coordinates and acquisition time of the second point cloud data specified above from the data in the image data storage unit 25. The extracted first image data and second image data are compared by a general-purpose image comparison device to measure the degree of match between them. When the degree of coincidence between the first image data and the second image data is equal to or greater than a predetermined threshold, it is presumed that the first image data and the second image data are images of the same road environment, and it is presumed that the first point cloud data and the second point cloud data acquired synchronously therewith also reflect the same road environment.

[0024] On the other hand, when the degree of coincidence between the first image data and the second image data does not reach the predetermined threshold value, it is estimated that the first image data and the second image data have captured different road environments. The extracted first point cloud data and second point cloud data are found to have identical coordinates assigned to each point data, but positional deviations may occur in the acquired point cloud data, resulting in many coordinates of point data constituting each point cloud data being the same in the first point cloud data acquired at a first point on the first road and the second point cloud data acquired at a second point (close to the first point) on a second road (close to the first road) different from the first road. On the other hand, the first image data acquired in synchronization with the first point cloud data captures the environment of the first road, and the second image data acquired in synchronization with the second point cloud data captures the environment of the second road, so there is a clear difference between the two image data. In this way, by comparing the first image data acquired in synchronization with the acquisition of the first point cloud data with the second image data acquired in synchronization with the acquisition of the second point cloud data, it is possible to prevent the synthesis of point cloud data corresponding to different road environments, even though the coordinates assigned to the point data constituting the data match.

[0025] When the degree of coincidence between the first image data and the second image data exceeds a predetermined threshold and it is estimated that both data represent the same road environment, the point cloud data synthesis unit 50 aligns the first point cloud data with the second point cloud data using a general-purpose method and synthesizes the two. Since the first travel history to which the first point cloud data belongs and the second travel history to which the second point cloud data belongs correspond to the same road (link), the series of first point cloud data acquired in the first travel history is synthesized with the series of second point cloud data acquired in the second travel history. At this time, comparison between image data can be omitted. Point cloud data to be synthesized can be one that has the same or closest coordinates.

[0026] Next, the operation of the point cloud data synthesizing device 1 will be described with reference to the flowchart of FIG. In step 1, the data (point cloud data, coordinate data, and image data) acquired from the measurement vehicle 10 is stored in each storage section (point cloud data storage section 21, coordinate data storage section 23, and image data storage section 25) of the measurement vehicle data storage unit 20. In step 3, for a road link selected based on an instruction from an operator or based on a predetermined program, a first point cloud data is identified from a series of point cloud data belonging to the first driving history.

[0027] In step 5, second point cloud data candidates (plural) to be combined with the first point cloud data are extracted. In this example, point cloud data acquired at coordinates within a predetermined range (e.g., a radius of 30 m) from the acquisition coordinates of the first point cloud data in a driving history different from the first driving history is set as the second point cloud data candidates. The matching rate between the coordinates assigned to the point data constituting each point cloud data of the second point cloud data candidates and the coordinates assigned to the point data of the first point cloud data is compared, and the one with the highest matching rate is identified as the second point cloud data. At this time, the point cloud data having the second to nth matching ratios can also be extracted as next candidates and stored in the buffer memory.

[0028] In step 9, the image data comparison unit 40 extracts the first image data acquired in synchronization with the first point cloud data from the image data storage unit 25. Similarly, the image data comparison unit 40 extracts the second image data acquired in synchronization with the second point cloud data from the image data storage unit 25. In step S11, the image data comparison unit 40 compares the first image data with the second image data, and if the degree of match between the two image data is equal to or greater than a predetermined threshold (step S11: Y), the first point cloud data and the second point cloud data are synthesized by the point cloud data synthesis unit 50. The synthesis method is a well-known method.

[0029] In step S11, if the degree of match between the two image data does not reach a predetermined threshold value (step S11: N), the process proceeds to step S15. In step 15, the second to n-th point cloud data extracted in step 7 are set as new second point cloud data in descending order of matching rate, and the processes in steps 9 and 11 are repeated.

[0030] The first driving history to which the first point cloud data to be combined belongs and the second driving history to which the second point cloud data to be combined belong were obtained by driving on the same road (link). Therefore, the series of first point cloud data belonging to the first driving history and the series of second point cloud data belonging to the second driving history are combined to obtain point cloud data that reflects the road environment of the road (link).

[0031] FIG. 3 shows another embodiment of the image data comparison section 40. In FIG. The image data comparison unit 40 used in this embodiment is shown in detail. The image data comparison unit 40 includes an AI processing unit 41 and a classified scene comparison unit 43 . In this example, the first image data is compared with the second image data using AI. The image data acquired by the image data acquisition unit is classified into five classified scenes, and when the first image data and the second image data are classified into the same classified scene, the two image data are deemed to be identical or similar (step 11:Y in FIG. 2), and the first point cloud data and the second point cloud data are combined. When the two image data are classified into different classified scenes, the two image data are deemed to be different (step 11:N in FIG. 2), and the first point cloud data and the second point cloud data are not combined.

[0032] Examples of five classified scenes are shown in Figure 4. In order to train the AI ​​processing unit 41, for each classification scene, corresponding training data and non-corresponding training data are read into the calculation unit 413 in units of several hundreds of images. Thereafter, the first image data and the second image data that have actually been acquired are read into the calculation unit 413 via the image data input unit 411. As a result, the calculation unit 413 performs AI processing and causes the classification scene output unit 415 to output the classification scene of the input image data. By identifying the relationship between the first image data and the second image data based on the classified scenes obtained through AI processing, it is possible to accurately and quickly determine the differences and similarities between the first image data and the second image data.

[0033] FIG. 5 shows a hardware configuration of the point cloud data synthesis device 1 shown in FIG. The calculation unit 300 includes a CPU 301, a ROM 303, and a RAM 305, and controls the entire system. It also controls the calculation parts of the point cloud data extraction unit 30, the point cloud data acquisition coordinate specification unit 35, the image data comparison unit 40, and the point cloud data synthesis unit 50. The ROM 303 is a non-volatile memory in which a control program for controlling the calculation unit 300 and the like are stored. The RAM 305 provides a working area for the CPU 301. The control program for controlling the calculation unit 300 may be stored not only in the ROM 303, but also in the RAM 305 or the first storage device 340. Various data are output via the output device 320.

[0034] The first storage device 340 functions as the measurement vehicle data storage unit 20 . It is preferable that the first storage device 340 utilizes a partial area of ​​a memory device of the server system, such as a hard memory or a flash memory. A part of the RAM of the calculation unit can be used as a buffer memory for temporarily storing data. A communication interface 460 is connected to an external communication network. The devices constituting the computer are connected via a system bus 370 .

[0035] The present invention is not limited to the above-mentioned embodiment and examples. Various modifications within the scope of the claims and within the scope that can be easily conceived by a person skilled in the art are also included in the present invention.

[0036] 1 point cloud data synthesis device, 10 measuring vehicle, 11 point cloud data acquisition unit, 13 coordinate specification unit, 15 image data acquisition unit, 21 point cloud data storage unit, 23 coordinate data storage unit, 25 image data storage unit, 30 point cloud data extraction unit, 35 point cloud data acquisition coordinate specification unit, 40 image data comparison unit, 50 point cloud data synthesis unit

Claims

1. A method for synthesizing point cloud data, comprising: using a point cloud data synthesizing device including a point cloud data extracting unit, an image data comparing unit, and a point cloud data synthesizing unit; synthesizing point cloud data obtained by a point cloud data acquiring unit of a measuring vehicle including a point cloud data acquiring unit and an image data acquiring unit, causing the point cloud data extraction unit to extract first point cloud data and second point cloud data to be combined, wherein the first point cloud data is acquired in a first travel history of the measurement vehicle, the second point cloud data is acquired in a second travel history of the measurement vehicle, and the first point cloud data and the second point cloud data have point data with the same coordinates; The image data comparison unit is caused to compare first image data acquired in synchronization with acquisition of the first point cloud data with second image data acquired in synchronization with acquisition of the second point cloud data, wherein the first image data and the second image data are acquired by the image data acquisition unit. causing the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data based on a comparison result between the first image data and the second image data; Methods for merging point cloud data.

2. The image data comparison unit includes an AI processing unit and a classification scene comparison unit, and the AI ​​processing unit has a function of classifying the image data acquired by the image data acquisition unit into a plurality of predetermined classification scenes, The first image data is input to the AI ​​processing unit to generate a first classified scene, and the second image data is input to the AI ​​processing unit to generate a second classified scene; The synthesis method according to claim 1 , wherein the classified scene comparison unit causes the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data when the first classified scene and the second classified scene are in the same classification.

3. A point cloud data synthesis device including a point cloud data extraction unit, an image data comparison unit, and a point cloud data synthesis unit, and a point cloud data synthesis method is carried out to synthesize point cloud data obtained by the point cloud data acquisition unit of a measuring vehicle including a point cloud data acquisition unit and an image data acquisition unit, causing the point cloud data extraction unit to extract first point cloud data and second point cloud data to be combined, wherein the first point cloud data is acquired in a first travel history of the measurement vehicle, the second point cloud data is acquired in a second travel history of the measurement vehicle, and the first point cloud data and the second point cloud data have point data with the same coordinates; The image data comparison unit is caused to compare first image data acquired in synchronization with acquisition of the first point cloud data with second image data acquired in synchronization with acquisition of the second point cloud data, wherein the first image data and the second image data are acquired by the image data acquisition unit. causing the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data based on a comparison result between the first image data and the second image data; Point cloud data synthesis device.

4. A program for executing point cloud data synthesis, which is applied to a computer device of a point cloud data synthesis device comprising a point cloud data extraction unit, an image data comparison unit, and a point cloud data synthesis unit, and which synthesizes point cloud data obtained by the point cloud data acquisition unit of a measuring vehicle comprising a point cloud data acquisition unit and an image data acquisition unit, causing the point cloud data extraction unit to extract first point cloud data and second point cloud data to be combined, wherein the first point cloud data is acquired in a first travel history of the measurement vehicle, the second point cloud data is acquired in a second travel history of the measurement vehicle, and the first point cloud data and the second point cloud data have point data with the same coordinates; The image data comparison unit is caused to compare first image data acquired in synchronization with acquisition of the first point cloud data with second image data acquired in synchronization with acquisition of the second point cloud data, wherein the first image data and the second image data are acquired by the image data acquisition unit. causing the point cloud data synthesis unit to synthesize the first point cloud data and the second point cloud data based on a comparison result between the first image data and the second image data; A program for a computer device of a point cloud data synthesis device.

Images