Point cloud processing device, point cloud processing method, and point cloud processing program

Abstract

This point group processing device includes: an input unit that receives a combination of a 3D point group which is on the surface of an object that was measured, and a measured posture in which the 3D point group was measured; a foreground extraction unit that extracts, on the basis of the measured posture, a foreground point group which is the point group of a foreground portion, from the 3D point group; and a point group integration unit that integrates the foreground point group and a background point group which is the point group of a background portion and which was found in advance.

Description

[Technical Field]

[0001] The disclosed technology relates to a point cloud processing device, a point cloud processing method, and a point cloud processing program. [Background technology]

[0002] A measuring device called LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) uses the reflection of light such as lasers to measure the distance to objects in space, and by combining this with location information obtained from GPS (Global Positioning System) or other devices, it obtains point cloud data, which is a collection of points with three-dimensional coordinates.

[0003] Due to the nature of LIDAR, which measures one point at a time, the number of points that can be measured per unit time is smaller than with camera images, so long-term measurement is necessary to obtain a high-density point cloud. However, when measuring the point cloud of a moving object, the measurement must be completed in a short time, making it difficult to measure a high-density point cloud with LIDAR.

[0004] A technique for increasing the density of a low-density point cloud can be, for example, the technique shown in Patent Document 1. The point cloud is converted into a depth image from the camera's viewpoint, and the densified depth image is then converted back into a point cloud, thereby increasing the density of the point cloud in the range captured in the camera image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-174406 Summary of the Invention [Problem to be solved by the invention]

[0006] While a typical lidar can measure a point cloud in all directions (360 degrees horizontally), the angle of view of the camera images used in the technology described in Patent Document 1 is limited, and in order to densify a point cloud over a wide area, it is necessary to prepare a large number of cameras and take images in all directions. In that case, it takes time to apply densification processing to all camera images, making it unusable for applications requiring real-time performance.

[0007] The disclosed technology has been developed in consideration of the above points, and aims to provide a point cloud processing device, a point cloud processing method, and a point cloud processing program that can measure three-dimensional point clouds over a wide area, including moving objects, in a short period of time. [Means for solving the problem]

[0008] A first aspect of the present disclosure is a point cloud processing device that includes an input unit that accepts a combination of a three-dimensional point cloud on a measured surface of an object and a measurement orientation at which the three-dimensional point cloud was measured, a foreground extraction unit that extracts a foreground point cloud, which is a point cloud of a foreground portion, from the three-dimensional point cloud based on the measurement orientation, and a point cloud integration unit that integrates the foreground point cloud with a background point cloud, which is a point cloud of a background portion that has been determined in advance.

[0009] A second aspect of the present disclosure is a point cloud processing method in which a computer receives a combination of a three-dimensional point cloud on a measured surface of an object and a measurement orientation at which the three-dimensional point cloud was measured, extracts a foreground point cloud, which is a point cloud of the foreground part, from the three-dimensional point cloud based on the measurement orientation, and integrates the foreground point cloud with a background point cloud, which is a point cloud of the background part that has been determined in advance.

[0010] A third aspect of the present disclosure is a point cloud processing program for causing a computer to function as the point cloud processing device of the first aspect. [Effects of the Invention]

[0011] According to the disclosed technology, a three-dimensional point cloud over a wide area including a moving object can be measured in a short time. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic block diagram of an example of a computer that functions as a point cloud processing device according to the first, second, third, and fourth embodiments. [Figure 2] FIG. 1 is a block diagram showing a configuration of a point cloud processing device according to a first embodiment. [Figure 3] FIG. 10 is a diagram for explaining a method for extracting a group of foreground points. [Figure 4] 4 is a flowchart showing a point cloud processing routine of the point cloud processing device of the first embodiment. [Figure 5] 4 is a flowchart showing a point cloud processing routine of the point cloud processing device of the first embodiment. [Figure 6] FIG. 1 is a diagram showing an arrangement of multiple lidars and multiple cameras. [Figure 7] FIG. 10 is a block diagram showing the configuration of a point cloud processing device according to a second embodiment. [Figure 8] 10 is a flowchart showing a point cloud processing routine of the point cloud processing device of the second embodiment. [Figure 9] FIG. 10 is a block diagram showing the configuration of a point cloud processing device according to a third embodiment. [Figure 10] 10 is a flowchart showing a point cloud processing routine of the point cloud processing device of the third embodiment. [Figure 11] FIG. 10 is a block diagram showing the configuration of a point cloud processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same reference numerals are used to designate identical or equivalent components and parts in each drawing. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0014] <Outline of this embodiment> In this embodiment, a foreground point cloud representing objects including moving objects is extracted from a 3D point cloud measured by a lidar, and the foreground point cloud is then integrated with a background point cloud representing a stationary background that has been separately measured over time. In this way, by measuring the background point cloud representing the stationary background in advance, a high-density 3D point cloud can be measured over time. Furthermore, by measuring the 3D point cloud by narrowing the range to only the area representing the moving object, the amount of data to be transmitted and processed can be reduced, enabling highly real-time measurement. Therefore, point cloud data with higher measurement accuracy can be output faster than conventional methods. Furthermore, because point cloud data can be measured quickly and in a short time, it is possible to measure a three-dimensional space including a moving object in a short time or to measure it in real time as a moving image.

[0015] [First embodiment] <Configuration of the point cloud processing device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of a point cloud processing device 10 according to this embodiment.

[0016] 1, the point cloud processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0017] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores a point cloud processing program for measuring a three-dimensional point cloud. The point cloud processing program may be a single program, or may be a group of programs composed of multiple programs or modules.

[0018] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0019] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various data, including a combination of a 3D point cloud on the surface of an object and the measurement orientation at which the 3D point cloud was measured, and a combination of a camera image capturing an area corresponding to the 3D point cloud and the shooting orientation at which the camera image was captured. Any device, such as a lidar, depth camera, stereo camera, or sonar, can be used to measure the 3D point cloud. For example, a 3D point cloud measured by a lidar and a camera image capturing an area corresponding to the 3D point cloud are input. While the measurement direction of the 3D point cloud and the shooting direction of the camera image do not need to coincide, the closer the two are, the more effective it is at minimizing blind spots and coloring errors during correspondence. The input camera image is captured by a camera, and each pixel has brightness and / or color information, e.g., a pixel value expressed as RGB.

[0020] Here, the measured orientation of the 3D point cloud can be expressed, for example, as a combination of a rotation matrix, a scaling matrix, and a translation matrix for converting the 3D point cloud into the same absolute coordinate system as the background point cloud. Note that if the input 3D point cloud and background point cloud are expressed in the same absolute coordinate system, if the measured orientation is estimated by the foreground point cloud integration unit 204 (described later), or if the measured orientation is estimated by the point cloud integration unit 104, input of the measured orientation of the 3D point cloud is not necessary. The shooting orientation of the camera image can be expressed, for example, as a combination of the camera's internal parameters, external parameters, and distortion coefficients. Note that the camera's external parameters may be expressed as a combination of a rotation matrix and a translation vector relative to the absolute coordinate system, or as a combination of a rotation matrix and a translation vector relative to the lidar coordinate system. The 3D point cloud may also be expressed in the form of a depth map recorded as an image having pixel-by-pixel depth values ​​from a certain viewpoint.

[0021] The display unit 16 is, for example, a liquid crystal display, and displays various information including the measured three-dimensional point cloud. The display unit 16 may employ a touch panel system and function as the input unit 15.

[0022] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0023] Next, a description will be given of the functional configuration of the point cloud processing device 10. FIG.

[0024] As shown in FIG. 2, the point cloud processing device 10 functionally includes a correspondence unit 101, a coloring unit 102, a foreground extraction unit 103, a point cloud integration unit 104, and an output unit 105.

[0025] The correspondence unit 101 identifies the correspondence between each point of the 3D point cloud and each pixel of the camera image by using the 3D point cloud, the camera image, and either or both of the measurement attitude and the shooting attitude. Note that there may be pixels with no corresponding points or points with no corresponding pixels.

[0026] The coloring unit 102 assigns color information of each pixel in the camera image to the corresponding point in the 3D point cloud based on the correspondence relationship identified by the correspondence unit 101, and outputs the colored 3D point cloud. At this time, the coloring unit 102 may estimate the 3D coordinates corresponding to each pixel using the camera image and the shooting posture, and create and output a 3D point cloud to which color information has been assigned based on the color information of each pixel and the estimated 3D coordinates.

[0027] The foreground extraction unit 103 extracts a foreground point cloud, which is a point cloud of the foreground portion, from the three-dimensional point cloud.

[0028] Specifically, the foreground extraction unit 103 receives a three-dimensional point cloud as input, extracts only points corresponding to the object being photographed, and outputs them as a foreground point cloud. For foreground extraction, a range of three-dimensional coordinates occupied by the object may be determined in advance, and only points within that range may be extracted. Alternatively, a reference point cloud that does not include the foreground may be measured in advance, and only points in the input point cloud that are not near the reference point cloud may be considered as the foreground point cloud (see FIG. 3).

[0029] Figure 3 shows an example in which the reference point cloud is converted into a depth map from the camera's viewpoint, the input three-dimensional point cloud is converted into a depth map from the camera's viewpoint, and the foreground is extracted by thresholding the difference between each pixel to obtain the foreground point cloud.

[0030] Alternatively, image processing such as image recognition may be applied to the camera image to determine a two-dimensional region of the target object, and corresponding points may be extracted based on the correspondence between pixels and points identified by the correspondence unit 101.

[0031] The point cloud integration unit 104 integrates the foreground point cloud with a background point cloud, which is a point cloud of the background portion that has been obtained in advance.

[0032] Specifically, the point cloud integration unit 104 receives the background point cloud, the foreground point cloud, and the position information of the foreground point cloud, and outputs a three-dimensional point cloud in which all point clouds are superimposed. If the position information of the foreground point cloud is not input, the foreground point cloud may be displayed at a predetermined position, or separately calculated position information may be used as the position information of the foreground point cloud. Note that if the background point cloud and the foreground point cloud are expressed in the same absolute coordinate system, it is not necessary to input the position information of the foreground point cloud.

[0033] The output unit 105 outputs the three-dimensional point cloud output from the point cloud integration unit 104 as data. For example, the integrated three-dimensional point cloud may be output as a file. The three-dimensional point cloud may also be output in the form of an image or video taken from a certain viewpoint. The output method may include saving the data in a file, transmitting the data to another system, or displaying the data directly on a screen.

[0034] <Action of the point cloud processing device according to this embodiment> Next, the operation of the point cloud processing device 10 will be described.

[0035] 4 and 5 are flowcharts showing the flow of point cloud processing by the point cloud processing device 10. The CPU 11 reads a point cloud processing program from the ROM 12 or storage 14, expands it into the RAM 13, and executes it, thereby performing point cloud processing. It is assumed that data input to the point cloud processing device 10 is acquired using an equipment configuration including a camera and a LIDAR, and the camera and the LIDAR are installed so that the object to be measured is within the camera's shooting range and the LIDAR's measurement range. It is assumed that a background point cloud calculated in advance is input to the point cloud processing device 10. The point cloud processing is an example of a point cloud processing method. If the installation positions of the camera and the LIDAR can be fixed, calibration can be performed in advance to determine the measurement orientation of the 3D point cloud and the shooting orientation of the camera, thereby eliminating the need to calculate the coordinate conversion formula each time.

[0036] First, in step S101, the CPU 11 acquires a reference point group, which is a three-dimensional point group measured by a lidar in a state where there is no object to be photographed, and a measured attitude.

[0037] In step S102, the CPU 11 acquires a combination of a three-dimensional point cloud measured by a rider and the measured posture at which the three-dimensional point cloud was measured, and a combination of a camera image taken by a camera and the shooting posture at which the camera image was taken.

[0038] In step S103, the CPU 11, as the correspondence unit 101, uses the three-dimensional point cloud, the camera image, and either or both of the measurement attitude and the shooting attitude to derive a coordinate transformation formula between the three-dimensional point cloud and the camera image as a correspondence relationship between the three-dimensional point cloud and the camera image.

[0039] Next, steps S104 and S105 are repeatedly executed for each pixel of the camera image.

[0040] In step S104, the CPU 11, functioning as the coloring unit 102, estimates the depth value of the pixel in the camera image based on the transformation formula derived in step S103, estimates the three-dimensional coordinates, and generates a corresponding three-dimensional point, thereby densifying the point cloud. As an example, a densified depth map is obtained using the technique described in Patent Document 1. Specifically, the 3D point cloud is converted into a depth map, pixels without depth values ​​are interpolated with the depth values ​​of surrounding pixels, the depth map is corrected so as to minimize inconsistencies with the camera image, and the corrected depth map is converted back into a three-dimensional point cloud, thereby obtaining a densified three-dimensional point cloud.

[0041] In step S105, the CPU 11 functions as the coloring unit 102 to add color information of the pixel to the three-dimensional point generated in step S104.

[0042] In step S106, the CPU 11, functioning as the foreground extraction unit 103, converts the reference point group into a depth map viewed from the same viewpoint as the camera image, based on the conversion formula derived in step S103, and also converts the densified 3D point group into a depth map viewed from the same viewpoint as the camera image. Note that if a camera image from a camera viewpoint corresponding to the reference point group can be acquired, densification may also be performed on the depth map of the reference point group. If a camera image corresponding to the reference point group cannot be acquired, the depth values ​​of pixels with no corresponding points may be interpolated using nearest neighbor interpolation or the like.

[0043] Then, the process of step S107 is executed for each pixel in the depth map.

[0044] In step S107, the CPU 11, functioning as the foreground extraction unit 103, deletes the depth value of the pixel in question in accordance with the difference in the depth value of the pixel in question in the depth map. b , the depth value of the same pixel in the depth map of the input 3D point cloud is D i Then, the depth values ​​of pixels that satisfy the following formula are deleted.

[0045] D b -D i <d

[0046] Note that D is a positive value proportional to the distance from the camera, and is expressed so that the farther it is from the camera, the larger the value becomes. d is a parameter representing the threshold value.

[0047] In step S108, the CPU 11, functioning as the foreground extraction unit 103, acquires a foreground point cloud by converting the depth map after the processing of step S107 back into a point cloud. Specifically, the CPU 11 converts each pixel of the depth map into three-dimensional coordinates using a conversion formula derived using camera internal parameters including the focal length of the camera lens, thereby converting the depth map back into a three-dimensional point cloud. Note that the foreground extraction unit 103, rather than the coloring unit 102, may color the point cloud by assigning color information of pixels in the camera image corresponding to pixels in the depth map to corresponding points.

[0048] Next, in step S109, the CPU 11, functioning as the point cloud integration unit 104, acquires point cloud position information of the foreground point cloud in order to integrate the foreground point cloud with the background point cloud. Specifically, in order to display the foreground point cloud at an arbitrary position, predetermined position information may be acquired, or point cloud position information input from an external system may be acquired.

[0049] In step S109, the CPU 11 functions as the point cloud integration unit 104 to integrate the foreground point cloud and the background point cloud based on the point cloud position information of the foreground point cloud and the point cloud position information of the background point cloud that has been calculated in advance. Here, the point cloud position information of the foreground point cloud and the point cloud position information of the background point cloud are position information in the same coordinate system.

[0050] In step S110, the CPU 11 functions as the output unit 105 to output, as data, the three-dimensional point cloud output from the point cloud integration unit 104. Specifically, the integrated point cloud is output to a file or another system, or is displayed on the display unit 16.

[0051] In step S111, the CPU 11 determines whether to continue measuring the three-dimensional point cloud. If the three-dimensional point cloud measurement is to be continued, the process returns to step S102, and a new combination of the three-dimensional point cloud measured by the lidar and the measurement attitude at which the three-dimensional point cloud was measured, and a new combination of the camera image taken by the camera and the shooting attitude at which the camera image was taken, are acquired. On the other hand, if the three-dimensional point cloud measurement is not to be continued, the point cloud processing routine is terminated.

[0052] As described above, the point cloud processing device according to this embodiment receives a combination of a 3D point cloud on the surface of a measured object and the measured orientation at which the 3D point cloud was measured, extracts a foreground point cloud from the 3D point cloud, and integrates the foreground point cloud with a previously determined background point cloud, which is a point cloud of the background. This makes it possible to measure a 3D point cloud over a wide area, including a moving object, in a short time.

[0053] In addition, by increasing the point density in addition to adding color information to the point cloud, a higher density point cloud can be obtained even in a short measurement time.

[0054] In addition, the input three-dimensional point cloud is converted into a depth map from the camera's viewpoint, and compared with a reference depth map created from a reference point cloud measured in advance.Only points corresponding to pixels whose difference with the reference depth map is greater than a threshold are extracted as foreground point clouds, thereby enabling high-speed extraction of only points corresponding to moving objects.

[0055] Furthermore, with conventional technology, when measuring a 3D point cloud representing a large space including a moving object at high resolution, it was difficult to measure because the state of the moving object changes during measurement of one frame. On the other hand, with this embodiment, a background point cloud without a moving object is measured in advance, and for the area representing the moving object, a sparse point cloud is measured in a short time, and then densified using high-resolution technology, thereby achieving measurement in a short time.

[0056] [Second embodiment] Next, a point cloud processing apparatus according to a second embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

[0057] The second embodiment differs from the first embodiment in that two or more combinations of lidar and camera are used to measure and integrate foreground point clouds.

[0058] <Configuration of the point cloud processing device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of a point cloud processing device 210 according to this embodiment.

[0059] As shown in FIG. 1, the point cloud processing device 210 includes a CPU 11, a ROM 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface 17.

[0060] The input unit 15 is used to perform various inputs, including two or more sets, where one set is a combination of a measured three-dimensional point cloud and the measurement attitude at which the three-dimensional point cloud was measured, and another set is a combination of a camera image of an area corresponding to the three-dimensional point cloud and the shooting attitude at which the camera image was taken. The following describes the process of performing input of two sets.

[0061] Specifically, the data input to the input unit 15 can be acquired by an equipment configuration with two cameras 20A, 20B and two LIDARs 30A, 30B as shown in Fig. 6. A plurality of cameras 20A, 20B and a plurality of LIDARs 30A, 30B are installed so that the object to be measured falls within the shooting range of the cameras 20A, 20B and the measurement range of the LIDARs 30A, 30B.

[0062] For example, the input unit 15 accepts a combination of a three-dimensional point cloud measured by the rider 30A and the measured attitude of the rider 30A who measured the three-dimensional point cloud, and a combination of a camera image of an area corresponding to the three-dimensional point cloud taken by the camera 20A and the shooting attitude of the camera 20A who took the camera image. The input unit 15 also accepts a combination of a three-dimensional point cloud measured by the rider 30B and the measured attitude of the rider 30B who measured the three-dimensional point cloud, and a combination of a camera image of an area corresponding to the three-dimensional point cloud taken by the camera 20B and the shooting attitude of the camera 20B who took the camera image.

[0063] Next, a description will be given of the functional configuration of the point cloud processing device 210. FIG.

[0064] As shown in FIG. 7, the point cloud processing device 210 functionally includes correspondence units 201A and 201B, coloring units 202A and 202B, foreground extraction units 203A and 203B, a foreground point cloud integration unit 204, a point cloud integration unit 205, and an output unit 206.

[0065] Similar to the correspondence unit 101 described in the first embodiment above, the correspondence unit 201A uses the three-dimensional point cloud measured by the rider 30A, the camera image captured by the camera 20A, and either or both of the measurement attitude of the rider 30A and the shooting attitude of the camera 20A to determine the correspondence between each point of the three-dimensional point cloud measured by the rider 30A and each pixel of the camera image captured by the camera 20A.

[0066] Similarly to the correspondence unit 201A, the correspondence unit 201B uses the three-dimensional point cloud measured by the rider 30B, the camera image captured by the camera 20B, and either or both of the measurement attitude of the rider 30B and the shooting attitude of the camera 20B to determine the correspondence between each point of the three-dimensional point cloud measured by the rider 30B and each pixel of the camera image captured by the camera 20B.

[0067] Similar to the coloring unit 102 described in the first embodiment above, the coloring unit 202A assigns color information of each pixel of the camera image captured by the camera 20A to the corresponding point of the three-dimensional point cloud measured by the lidar 30A based on the correspondence relationship identified by the correspondence unit 201A, and outputs the colored three-dimensional point cloud.

[0068] Similar to the coloring unit 202A, the coloring unit 202B assigns color information of each pixel of the camera image captured by the camera 20B to the corresponding point of the three-dimensional point cloud measured by the lidar 30B based on the correspondence relationship identified by the correspondence unit 201B, and outputs the colored three-dimensional point cloud.

[0069] The foreground extraction unit 203A, like the foreground extraction unit 103 described in the first embodiment, extracts a foreground point cloud, which is a point cloud of the foreground portion, from the 3D point cloud colored by the coloring unit 202A.

[0070] Similar to the foreground extraction unit 203A, the foreground extraction unit 203B extracts a foreground point cloud, which is a point cloud of the foreground portion, from the three-dimensional point cloud colored by the coloring unit 202B.

[0071] The foreground point cloud integration unit 204 converts the foreground point clouds extracted by each of the foreground extraction units 203A and 203B into the same coordinate system and outputs a foreground point cloud in which multiple foreground point clouds are superimposed. For example, each foreground point cloud may be converted into the same absolute coordinate system based on the measured orientation corresponding to each foreground point cloud, and then superimposed. The foreground point cloud integration unit 204 may also integrate the foreground point clouds extracted by each of the foreground extraction units 203A and 203B and align them to minimize misalignment at the joints. In other words, the foreground point cloud integration unit 204 corrects the positions of the foreground point clouds to minimize the difference in the area common to the two foreground point clouds extracted by each of the foreground extraction units 203A and 203B, and then integrates the foreground point clouds.

[0072] The point cloud integration unit 205 integrates the integrated foreground point cloud and background point cloud, similarly to the point cloud integration unit 104 described in the first embodiment.

[0073] The output unit 206 outputs the three-dimensional point cloud output from the point cloud integration unit 205 as data, similar to the output unit 105 described in the first embodiment.

[0074] <Action of the point cloud processing device according to this embodiment> Next, the operation of the point cloud processing device 210 will be described.

[0075] 8 is a flowchart showing the flow of point cloud processing by the point cloud processing device 210. Point cloud processing is performed by the CPU 11 reading out a point cloud processing program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. Furthermore, it is assumed that the data input to the point cloud processing device 210 is acquired by an equipment configuration including cameras 20A and 20B and LIDARs 30A and 30B, and that the cameras 20A and 20B and LIDARs 30A and 30B are installed so that the object to be photographed to be measured falls within the photographing range and measurement range. It is assumed that a background point cloud obtained in advance has been input to the point cloud processing device 210.

[0076] First, in step S201, the CPU 11 acquires a reference point group, which is a three-dimensional point group measured by the LIDAR 30A in a state where there is no object to be photographed, and a measured attitude.

[0077] In parallel with step S201, in step S203, the CPU 11 acquires a reference point group, which is a three-dimensional point group measured by the RIDER 30B in a state where there is no object to be photographed, and a measured attitude.

[0078] Then, in step S202, the CPU 11 extracts a foreground point cloud from the three-dimensional point cloud measured by the LIDAR 30A.

[0079] This step S202 is realized by the same processing as steps S102 to S108 described in the first embodiment, and involves obtaining a three-dimensional point cloud, increasing density, coloring the point cloud, and extracting a foreground point cloud.

[0080] In parallel with step S202, in step S204, the CPU 11 extracts a foreground point cloud from the three-dimensional point cloud measured by the LIDAR 30B.

[0081] This step S204 is realized by the same processing as steps S102 to S108 described in the first embodiment, and involves obtaining a three-dimensional point cloud, increasing density, coloring the point cloud, and extracting a foreground point cloud.

[0082] Then, in step S205, the CPU 11 functions as the foreground point group integrating unit 204 to integrate the foreground point groups extracted by the foreground extraction units 203A and 203B, and aligns the points so as to minimize the misalignment at the joints.

[0083] In this case, the foreground point clouds can be integrated into a single point cloud by aligning their positions by rotating and translating the foreground point clouds based on their respective measured orientations. However, there are cases where perfect alignment cannot be achieved due to differences in the time at which each point cloud is measured. Therefore, for example, by using the technology described in Non-Patent Document 1, it is possible to calculate rotation and translation so that areas common to multiple point clouds overlap as much as possible.

[0084] [Non-patent document 1]: RUSINKIEWICZ, Szymon; LEVOY, Marc. Efficient variants of the ICP algorithm. In: Proceedings third international conference on 3-D digital imaging and modeling. IEEE, 2001. p. 145-152.

[0085] Next, in step S206, the CPU 11, functioning as the point cloud integration unit 205, acquires point cloud position information for integrating the foreground point cloud with the background point cloud. Specifically, when integrating the foreground point clouds, if one of the foreground point clouds is used as a reference and the other foreground point clouds are integrated by calculating rotation and translation to align them with the reference foreground point cloud, the measured orientation of the reference foreground point cloud may be used as the point cloud position information. Furthermore, in order to reduce the positional deviation before and after integrating multiple foreground point clouds, the foreground point cloud position information may be determined so that the center of gravity does not change before and after integration. Furthermore, in order to display the foreground point cloud at an arbitrary position, predetermined position information or point cloud position information input from an external system may be used.

[0086] In step S207, the CPU 11, functioning as the point cloud integration unit 205, integrates the foreground point cloud and the background point cloud based on the point cloud position information of the foreground point cloud and the point cloud position information of the background point cloud.

[0087] In step S208, the CPU 11 functions as the output unit 206 to output the three-dimensional point cloud output from the point cloud integration unit 205 as data.

[0088] In step S209, CPU 11 determines whether to continue measuring the three-dimensional point cloud. If the measurement of the three-dimensional point cloud is to be continued, the process returns to steps S202 and S204, and a new combination of the three-dimensional point cloud measured by Riders 30A and 30B and the measurement attitude at which the three-dimensional point cloud was measured, and a new combination of the camera images taken by cameras 20A and 20B and the shooting attitude at which the camera images were taken, are acquired. On the other hand, if the measurement of the three-dimensional point cloud is not to be continued, the point cloud processing routine is terminated.

[0089] As described above, the point cloud processing device of the second embodiment can measure multiple moving objects and perform measurements with few blind spots by measuring and integrating foreground point clouds using multiple combinations of lidar and cameras.

[0090] In addition, by correcting and integrating the positions of the foreground point clouds so that the difference in areas common to two or more foreground point clouds is minimized, it is possible to reduce the visual discomfort even when there is a difference in measurement time between measuring instruments and simple integration would result in a discrepancy.

[0091] Furthermore, when a lidar measures a 3D point cloud horizontally, the ground is captured in a fan-shaped image, and ground points remain in the point cloud output from the coloring unit. If the foreground point cloud were to be integrated in this state, the fan-shaped portions would coincide, resulting in a failed alignment. Therefore, it is necessary to delete the point clouds outside the foreground in advance. While threshold processing based on the height coordinates of the point clouds is conceivable, in this embodiment, the foreground point cloud is extracted based on the difference from the reference point cloud measured initially. This makes it possible to handle cases where slopes, walls, obstacles, etc. are present.

[0092] [Third embodiment] Next, a point cloud processing device according to a third embodiment will be described. Note that parts having the same configuration as those in the first and second embodiments will be given the same reference numerals and descriptions thereof will be omitted.

[0093] The third embodiment differs from the second embodiment in that coloring is performed after integrating a plurality of foreground point groups.

[0094] <Configuration of the point cloud processing device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of a point cloud processing device 310 according to this embodiment.

[0095] As shown in FIG. 1, the point cloud processing device 310 includes a CPU 11, a ROM 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface 17.

[0096] As in the second embodiment, the input unit 15 is used to perform various inputs including two or more sets, where one set is a combination of a measured three-dimensional point cloud and the measurement attitude at which the three-dimensional point cloud was measured, and another set is a combination of a camera image capturing an area corresponding to the three-dimensional point cloud and the shooting attitude at which the camera image was captured. The following describes the process of performing input of two sets.

[0097] Specifically, the data input to the input unit 15 can be acquired by an equipment configuration including two cameras 20A and 20B and two lidars 30A and 30B shown in FIG.

[0098] Next, a description will be given of the functional configuration of the point cloud processing device 310. FIG.

[0099] As shown in FIG. 9, the point cloud processing device 310 functionally includes correspondence units 301A and 301B, foreground extraction units 302A and 302B, a foreground point cloud integration unit 303, a coloring unit 304, a point cloud integration unit 305, and an output unit 306.

[0100] Similar to the correspondence unit 201A described in the second embodiment above, the correspondence unit 301A uses the three-dimensional point cloud measured by the rider 30A, the camera image captured by the camera 20A, and either or both of the measurement attitude of the rider 30A and the shooting attitude of the camera 20A to determine the correspondence between each point of the three-dimensional point cloud measured by the rider 30A and each pixel of the camera image captured by the camera 20A.

[0101] Furthermore, similar to the correspondence unit 201B described in the second embodiment above, the correspondence unit 301B uses the three-dimensional point cloud measured by the rider 30B, the camera image captured by the camera 20B, and either or both of the measurement attitude of the rider 30B and the shooting attitude of the camera 20B to determine the correspondence between each point of the three-dimensional point cloud measured by the rider 30B and each pixel of the camera image captured by the camera 20B.

[0102] The foreground extraction unit 302A extracts a foreground point cloud, which is a point cloud of the foreground portion, from the three-dimensional point cloud measured by the LIDAR 30A.

[0103] Specifically, similar to the foreground extraction unit 103 described in the first embodiment above, the foreground extraction unit 302A receives as input a three-dimensional point cloud measured by the LIDAR 30A, extracts only points corresponding to the object to be photographed, and outputs them as a foreground point cloud.

[0104] Similar to the foreground extraction unit 302A, the foreground extraction unit 302B extracts a foreground point cloud, which is a point cloud of the foreground portion, from the three-dimensional point cloud measured by the LIDAR 30B.

[0105] Similar to foreground point cloud integration unit 204 described in the second embodiment, foreground point cloud integration unit 303 converts the foreground point clouds extracted by foreground extraction units 302A and 302B into the same coordinate system and outputs a foreground point cloud in which multiple foreground point clouds are superimposed. Furthermore, foreground point cloud integration unit 303 may integrate the foreground point clouds extracted by foreground extraction units 302A and 302B, and align them so that misalignment at the joints is minimized.

[0106] Based on the correspondence relationships identified by correspondence units 301A and 301B, coloring unit 304 assigns to each point of the integrated foreground point group a color determined based on color information of the corresponding pixel in the camera images captured by cameras 20A and 20B. The color to be assigned may be, for example, the median value for each dimension of RGB, or a representative value of the largest cluster obtained by clustering.

[0107] Specifically, based on the correspondence relationship identified by correspondence unit 301A, color information of each pixel in the camera image captured by camera 20A is assigned to the corresponding point in the integrated foreground point cloud, and based on the correspondence relationship identified by correspondence unit 301B, color information of each pixel in the camera image captured by camera 20B is assigned to the corresponding point in the integrated foreground point cloud. Then, the color information to be assigned to each point in the integrated foreground point cloud is determined by majority vote, and the colored 3D point cloud is output.

[0108] The point cloud integration unit 305 integrates the integrated foreground point cloud and background point cloud, similarly to the point cloud integration unit 205 described in the second embodiment.

[0109] The output unit 306 outputs the three-dimensional point cloud output from the point cloud integration unit 305 as data, similar to the output unit 206 described in the second embodiment.

[0110] <Action of the point cloud processing device according to this embodiment> Next, the operation of the point cloud processing device 310 will be described.

[0111] 10 is a flowchart showing the flow of point cloud processing by the point cloud processing device 310. Point cloud processing is performed by the CPU 11 reading out a point cloud processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Furthermore, it is assumed that the data input to the point cloud processing device 310 is acquired by an equipment configuration including cameras 20A and 20B and LIDARs 30A and 30B, and that the cameras 20A and 20B and LIDARs 30A and 30B are installed so that the object to be photographed to be measured falls within the photographing range and measurement range. It is assumed that a background point cloud obtained in advance has been input to the point cloud processing device 310.

[0112] First, in step S301, the CPU 11 acquires a reference point group, which is a three-dimensional point group measured by the LIDAR 30A in a state where there is no object to be photographed, and a measured attitude.

[0113] In parallel with step S301, in step S303, the CPU 11 acquires a reference point group, which is a three-dimensional point group measured by the Rider 30B in a state where there is no object to be photographed, and a measured attitude.

[0114] Then, in step S302, the CPU 11 extracts a foreground point cloud from the three-dimensional point cloud measured by the LIDAR 30A.

[0115] This step S302 is realized by the same processes as steps S102, S103, and S106 to S108 described in the first embodiment, and a three-dimensional point cloud is acquired and a foreground point cloud is extracted.

[0116] In parallel with step S302, in step S304, the CPU 11 extracts a foreground point cloud from the three-dimensional point cloud measured by the LIDAR 30B.

[0117] This step S304 is realized by the same processes as steps S102, S103, and S106 to S108 described in the first embodiment, and a three-dimensional point cloud is acquired and a foreground point cloud is extracted.

[0118] Then, in step S305, the CPU 11 functions as the foreground point group integrating unit 303 to integrate the foreground point groups extracted by the foreground extraction units 302A and 302B, and aligns the points so as to minimize the misalignment at the joints.

[0119] In step S306, the CPU 11, functioning as the coloring unit 304, assigns a color determined from the color information of the pixels in the camera image to each point in the integrated foreground point cloud based on the correspondence relationship identified by the correspondence units 301A and 301B. The CPU 11 may also increase the density of the point cloud by estimating the depth value of each pixel in the camera image based on a conversion formula, estimating the three-dimensional coordinates, and generating corresponding three-dimensional points.

[0120] Next, in step S307, the CPU 11 functions as the point cloud integration unit 305 to acquire point cloud position information for integrating the foreground point cloud with the background point cloud.

[0121] In step S308, the CPU 11, functioning as the point cloud integration unit 305, integrates the foreground point cloud and the background point cloud based on the point cloud position information of the foreground point cloud and the point cloud position information of the background point cloud.

[0122] In step S309, the CPU 11 functions as the output unit 306 to output the three-dimensional point cloud output from the point cloud integration unit 205 as data.

[0123] In step S310, CPU 11 determines whether to continue measuring the three-dimensional point cloud. If the three-dimensional point cloud measurement is to be continued, the process returns to steps S302 and S304, and a new combination of the three-dimensional point cloud measured by Riders 30A and 30B and the measurement attitude at which the three-dimensional point cloud was measured, and a new combination of the camera images captured by cameras 20A and 20B and the shooting attitude at which the camera images were captured are acquired. On the other hand, if the three-dimensional point cloud measurement is not to be continued, the point cloud processing routine is terminated.

[0124] As described above, the point cloud processing device of the third embodiment can determine the color to be used based on multiple camera images by coloring the point cloud after integrating the foreground point cloud, thereby reducing color unevenness at the boundary positions of multiple point clouds.

[0125] When coloring a 3D point cloud, the color of each pixel in the camera images captured by the camera is referenced for the 3D point cloud measured by the LIDAR, and the color of each point is determined. When the LIDAR and the camera are separate devices, parallax inevitably occurs, and calibration errors also occur. In this embodiment, by determining the color of each point based on multiple camera images, blind spots caused by parallax can be compensated for and coloring errors caused by errors can be reduced by majority vote.

[0126] Furthermore, when densifying the 3D point cloud using the coloring unit, points that did not exist in the original point cloud are added, which may result in points being generated in positions different from the actual positions, which may result in poor integration of the foreground point cloud. In this embodiment, by integrating the foreground point cloud before densifying the point cloud, more accurate alignment can be achieved.

[0127] [Fourth embodiment] Next, a point cloud processing device according to a fourth embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

[0128] The fourth embodiment differs from the first embodiment in that the timing of measuring a three-dimensional point cloud and capturing a camera image is controlled so that the measurement and capture data are performed simultaneously, and the measured and captured data are input continuously.

[0129] <Configuration of the point cloud processing device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of a point cloud processing device 410 according to this embodiment.

[0130] As shown in FIG. 1, the point cloud processing device 410 includes a CPU 11, a ROM 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface 17.

[0131] Next, a description will be given of the functional configuration of the point cloud processing device 410. Fig. 11 is a block diagram showing an example of the functional configuration of the point cloud processing device 410.

[0132] As shown in FIG. 11, the point cloud processing device 410 functionally includes a shooting control unit 400, a correspondence unit 101, a coloring unit 102, a foreground extraction unit 103, a point cloud integration unit 104, and an output unit 105.

[0133] The imaging control unit 400 repeatedly performs control so that the timing of measuring the three-dimensional point cloud by the lidar and the timing of capturing the camera image by the camera correspond to each other, and inputs the results to the correspondence unit 101.

[0134] Specifically, the imaging control unit 400 minimizes the time lag between the 3D point cloud measured by the LIDAR and the camera images captured by the camera, and controls the measurements and images to be taken simultaneously. For example, the time settings may be synchronized in advance to perform measurements and images periodically, or an external system may send measurement and image capture signals to multiple measuring devices simultaneously. Alternatively, measurements may be taken continuously by the LIDAR and images taken by the camera, and the measured 3D point cloud and captured camera images may be input sequentially to the corresponding unit 101.

[0135] The other configurations and operations of the point cloud processing device 410 according to the fourth embodiment are the same as those of the first embodiment, and therefore the description thereof will be omitted.

[0136] As described above, the point cloud processing device of the fourth embodiment controls the measurement of three-dimensional point clouds and the capture of camera images, thereby suppressing coloring of point clouds due to timing differences between measurement and capture, and misalignment when integrating foreground point clouds, and by capturing images continuously, it is possible to capture a time series of scenes including moving objects as three-dimensional information.

[0137] When measuring a 3D point cloud representing an object, including a moving object, in real time, if the point cloud is measured and images are captured at a fixed frame rate, subsequent processing (densification, point cloud integration) may not be able to keep up. On the other hand, if the next frame is processed after all processing is completed, it will take the same amount of time as sequential processing, even if each process can be parallelized. Therefore, in this embodiment, by timing the completion of each process and issuing instructions to measure the point cloud and capture images to the LIDAR and camera in advance, it is possible to achieve measurement at the highest possible frame rate.

[0138] The imaging control unit 400 may be configured by a plurality of PCs that are time-synchronized in advance or a plurality of PCs that are connected via a network and operate in cooperation with each other.

[0139] Furthermore, the control by the imaging control unit 400 may be applied to the point cloud processing devices of the second and third embodiments.

[0140] <Modification> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0141] For example, in the above embodiment, the number of LIDARs and cameras is the same, but this is not limited to this. As long as the imaging range of any camera covers the measurement range of the LIDAR, the number of LIDARs and cameras does not have to be the same.

[0142] Furthermore, although the example has been described in which there is one background point group obtained in advance, the present invention is not limited to this. Even if there are multiple background point groups obtained in advance, they may be integrated in advance.

[0143] Although the example has been described in which a camera image is input, the present invention is not limited to this. It is also possible for no camera image to be input. In this case, color information is not added to the 3D point cloud.

[0144] Although the above description has been given using an example in which the densification of a three-dimensional point cloud is performed using the method described in Patent Document 1, the present invention is not limited to this example, and other densification methods may also be used to perform the densification of a three-dimensional point cloud.

[0145] Although the above description is based on an example in which the reference point cloud is measured at the beginning of point cloud processing, the present invention is not limited to this. A depth map may be obtained from a background point cloud that has been calculated in advance and used as the depth map for the reference point cloud.

[0146] The reference point cloud may also be measured by a different LIDAR from the 3D point cloud used to extract the foreground point cloud. In this case, the reference point cloud measured by the different LIDAR can be converted into a depth map viewed from the same viewpoint as the camera image.

[0147] Furthermore, various processes executed by the CPU after reading software (programs) in the above embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. Point cloud processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0148] In addition, in each of the above embodiments, the point cloud processing program is described as being pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0149] The following additional notes are provided regarding the above-described embodiments.

[0150] (Additional note 1) A point cloud processing device, Memory and at least one processor coupled to said memory; Including, The processor: receiving a combination of a measured 3D point cloud on the surface of the object and a measurement orientation at which the 3D point cloud was measured; extracting a foreground point cloud, which is a point cloud of a foreground portion, from the three-dimensional point cloud based on the measured orientation; The foreground point group is integrated with a background point group, which is a point group of the background portion, which has been obtained in advance. The point cloud processing device is configured as follows.

[0151] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform point cloud processing, The point cloud processing includes: receiving a combination of a measured 3D point cloud on the surface of the object and a measurement orientation at which the 3D point cloud was measured; extracting a foreground point cloud, which is a point cloud of a foreground portion, from the three-dimensional point cloud based on the measured orientation; The foreground point group is integrated with a background point group, which is a point group of the background portion, which has been obtained in advance. Non-transitory storage medium. [Explanation of symbols]

[0152] 10, 210, 310, 410 Point Cloud Processing Device 11 CPU 14. Storage 15 Input section 16 Display section 20A, 20B Camera 30A, 30B riders 101 Response Department 102, 304 Colored section 103 Foreground extraction part 104, 205, 305 Point Cloud Integration Unit 105, 206, 306 Output section 201A, 201B, 301A, 301B corresponding parts 202A, 202B Colored section 203A, 203B, 302A, 302B Foreground extraction section 204, 303 Foreground point cloud integration part 400 Imaging control unit

Claims

1. an input unit that receives a combination of a measured 3D point cloud on the surface of an object and a measured orientation at which the 3D point cloud is measured; a foreground extraction unit that extracts a foreground point cloud, which is a point cloud of a foreground portion, from the three-dimensional point cloud based on the measured orientation; a point cloud integration unit that integrates the foreground point cloud with a background point cloud that is a point cloud of a background portion that has been obtained in advance; A point cloud processing device comprising:

2. Further comprising a foreground point cloud integration unit; the input unit receives two or more combinations of the three-dimensional point cloud and the measurement orientation; the foreground extraction unit extracts the foreground point cloud from each set of the three-dimensional point cloud; the foreground point cloud integration unit integrates the foreground point clouds extracted from each set of the three-dimensional point clouds; The point cloud processing device according to claim 1 , wherein the point cloud integration unit integrates the integrated foreground point cloud and the integrated background point cloud.

3. The corresponding part, and a coloring portion, the input unit further receives a combination of a camera image and a shooting attitude at which the camera image was captured, the combination corresponding to the combination of the three-dimensional point cloud and the measurement attitude; the correspondence unit specifies a correspondence relationship between each point of the three-dimensional point cloud and each pixel of the camera image based on at least one of the measurement attitude and the photographing attitude; The point cloud processing device according to claim 1 , wherein the coloring unit assigns color information of each pixel of the camera image to a corresponding point in the three-dimensional point cloud based on the correspondence relationship.

4. Further comprising a foreground point cloud integration unit; the input unit receives two or more combinations of the three-dimensional point cloud and the measurement orientation, and receives two or more combinations of camera images and shooting orientations at which the camera images were captured; the foreground extraction unit extracts the foreground point cloud from each set of the three-dimensional point cloud; the foreground point cloud integration unit integrates the foreground point clouds extracted from each set of the three-dimensional point clouds; the correspondence unit specifies, for each pair, a correspondence relationship between each point of the three-dimensional point cloud and each pixel of the camera image based on the measurement attitude and the shooting attitude; The point cloud processing device according to claim 3 , wherein the coloring unit assigns color information of a corresponding pixel in the camera image to each point in the integrated foreground point cloud based on the correspondence relationship identified for each pair.

5. The point cloud processing apparatus according to claim 3 , wherein the coloring unit further performs densification of the three-dimensional point cloud.

6. 2. The point cloud processing device according to claim 1, wherein the foreground extraction unit converts the received three-dimensional point cloud into a depth map from a specific viewpoint, in which each pixel has a pixel value corresponding to a depth value of an object, compares the converted depth map with a predetermined reference depth map, and extracts, as the foreground point cloud, a point cloud consisting of points corresponding to pixels whose depth values ​​differ from those of corresponding pixels in the reference depth map by a threshold or more.

7. The point cloud processing device according to claim 2 , wherein the foreground point cloud integration unit corrects and integrates the positions of the foreground point clouds so as to minimize a difference in an area common to two or more foreground point clouds.

8. The point cloud processing device according to claim 3, further comprising an imaging control unit that repeatedly controls the measurement timing of the three-dimensional point cloud and the shooting timing of the camera image to correspond to each other, and repeatedly inputs the three-dimensional point cloud and the camera image to the correspondence unit.

9. The computer receiving a combination of a measured 3D point cloud on the surface of the object and a measurement orientation at which the 3D point cloud was measured; extracting a foreground point cloud, which is a point cloud of a foreground portion, from the three-dimensional point cloud based on the measured orientation; The foreground point group is integrated with a background point group, which is a point group of the background portion, which has been obtained in advance. Point cloud processing methods.

10. A point cloud processing program for causing a computer to function as the point cloud processing device according to any one of claims 1 to 8.

Images