3D point cloud processing device, 3D point cloud processing method, and 3D point cloud processing program

Abstract

To accurately up-sample a three-dimensional point cloud obtained through measurement.SOLUTION: An input processing unit 20 receives a first image and a second image for which at least relationship of imaging positions has been determined in advance, and a three-dimensional point cloud on a surface of an object for which at least a relationship between an imaging position and a measurement position has been determined in advance, to determine pixel positions on the first image corresponding to each of three-dimensional points of the three-dimensional point cloud. A vicinity selection unit 24 selects a depth value, based on the pixel positions on the first image corresponding to each of the three-dimensional points of three-dimensional point cloud, for each of the pixel positions on the first image, from depth values of three-dimensional points corresponding to neighboring pixel positions, so that consistency between the first image and the second image may be higher and a distance from the neighboring pixel positions may be smaller.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The disclosed technology relates to a three-dimensional point cloud processing device, a three-dimensional point cloud processing method, and a three-dimensional point cloud processing program. [Background technology]

[0002] Non-Patent Document 1 discloses a technology that uses LiDAR (Light Detection and Ranging) and stereo images to output a depth image by upsampling a sparse point cloud obtained by LiDAR based on the disparity estimation results of the stereo images.

[0003] Non-Patent Document 2 discloses a technology that outputs a depth image by upsampling a LiDAR point cloud from LiDAR and stereo images using supervised deep learning.

[0004] Non-Patent Document 3 discloses a technology that outputs a depth image by upsampling a LiDAR point cloud from LiDAR and stereo images using semi-supervised deep learning. The technology disclosed in Non-Patent Document 3 makes it possible to perform inference by removing the influence of projection errors even if they exist. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Maddern, Will, and Paul Newman. "Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo." 2016 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016. [Non-Patent Document 2] Park, Kihong, Seungryong Kim, and Kwanghoon Sohn. "High-precision depth estimation with the 3d lidar and stereo fusion." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018. [Non-Patent Document 3] Cheng, Xuelian, et al. "Noise-aware unsupervised deep lidar-stereo fusion." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] The problem with Non-Patent Document 1 is that the projected LiDAR points are processed as correct depth values, and the case where a projection error occurs is not taken into consideration.

[0007] The problems with Non-Patent Document 2, like Non-Patent Document 1, are that it does not take into account that projection errors exist in the data, and that it requires high-density depth images for learning, but it is not easy to obtain such data.

[0008] The problem with Non-Patent Document 3 is that a large number of pairs of LiDAR measurements and stereo images that are correctly registered during learning are required, and obtaining such data is not easy.

[0009] As mentioned above, the problem with Non-Patent Documents 1 and 2 is that they do not take into account the projection error of LiDAR data. Also, the problem with Non-Patent Document 3 is that it requires a large amount of training data.

[0010] Furthermore, all of Non-Patent Documents 1 to 3 use the process of "creating upsampled data based on the results of disparity estimation." Disparity estimation using stereo reduces distance measurement accuracy over long distances. In contrast, LiDAR has high distance measurement accuracy even over long distances, but this characteristic is not utilized.

[0011] The disclosed technology has been made in consideration of the above-mentioned points, and aims to provide a 3D point cloud processing device, a 3D point cloud processing method, and a 3D point cloud processing program that can accurately upsample 3D point clouds obtained by measurement. [Means for solving the problem]

[0012] A first aspect of the present disclosure is a three-dimensional point cloud processing device that includes an input processing unit that receives a first image and a second image, for which the relationship between at least the shooting positions is determined in advance, and a three-dimensional point cloud on a surface of an object, for which the relationship between at least the shooting positions and the measurement positions is determined in advance, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, and a neighborhood selection unit that selects, for each pixel position on the first image, based on the pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, from depth values ​​of three-dimensional points corresponding to a plurality of nearby pixel positions, so as to increase the consistency between the first image and the second image.

[0013] A second aspect of the present disclosure is a three-dimensional point cloud processing method, in which an input processing unit receives a first image and a second image, for which at least the relationship between the shooting positions is determined in advance, and a three-dimensional point cloud on a surface of an object, for which the relationship between at least the shooting positions and the measurement positions is determined in advance, determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, and a neighborhood selection unit selects, for each pixel position on the first image, based on the pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, from depth values ​​of three-dimensional points corresponding to multiple nearby pixel positions, so as to achieve high consistency between the first image and the second image.

[0014] A third aspect of the present disclosure is a three-dimensional point cloud processing program for causing a computer to function as the three-dimensional point cloud processing device of the first aspect. Effect of the Invention

[0015] According to the disclosed technology, a 3D point cloud obtained by measurement can be upsampled with high accuracy. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a schematic block diagram of an example of a computer that functions as the three-dimensional point cloud processing device of the present embodiment. [Diagram 2] 1 is a diagram illustrating an example of measurement points by a LiDAR sensor and scenes captured by a first camera and a second camera. FIG. [Diagram 3] 1 is a block diagram showing a configuration of a three-dimensional point cloud processing device according to an embodiment of the present invention. [Figure 4] FIG. 2A is a diagram showing an example of a first image, FIG. 2B is a diagram showing an example of a second image, and FIG. 2C is a diagram showing an example of the result of projecting a three-dimensional point cloud onto the first image. [Diagram 5] FIG. 13 is a diagram for explaining a method for extracting a neighborhood set. [Figure 6] 4 is a flowchart showing a three-dimensional point cloud processing routine of the three-dimensional point cloud processing device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0018] <Outline of this embodiment> In this embodiment, a sparse 3D point cloud obtained by measurement using LiDAR is upsampled using a first image and a second image captured by a camera as clues.

[0019] Specifically, instead of assuming that the LiDAR projection is correct, the optimal depth value is selected from the 3D points projected around the pixel based on an energy function consisting of a cost function and a smoothing term. This method does not require training data and correctly upsamples the 3D point cloud measured by LiDAR even if there is an error in the LiDAR projection information.

[0020] Furthermore, this embodiment performs upsampling by assigning the 3D point cloud measured by LiDAR to pixels, which means that measurement accuracy is maintained over long distances, unlike methods that perform disparity estimation.

[0021] As described above, this embodiment is characterized by the fact that it does not use learning data, that it is possible to perform correct upsampling even when there is a projection error due to occlusion caused by vehicle movement or differences in sensor position between the LiDAR and the camera, and that it is possible to suppress loss of accuracy over long distances by upsampling the 3D point cloud measured by the LiDAR without estimating distance based on image parallax.

[0022] <Configuration of the 3D point cloud processing device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of a three-dimensional point cloud processing device 10 according to this embodiment.

[0023] 1, the 3D point cloud processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0024] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a three-dimensional point cloud processing program for upsampling a three-dimensional point cloud. The three-dimensional point cloud processing program may be one program, or may be a group of programs composed of multiple programs or modules.

[0025] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0026] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs including a first image and a second image for which the relationship between at least the shooting positions is previously obtained, and a three-dimensional point group P on the surface of an object for which the relationship between at least the shooting positions and the measurement positions is previously obtained. For example, the three-dimensional point group P measured by the LiDAR sensor 50, a first image (see FIG. 4(A)) photographed by the first camera 52, and a second image (see FIG. 4(B)) photographed by the second camera 54 as shown in FIG. 2 are input to the input unit 15. The relationship between the shooting positions of the first camera 52 and the second camera 54 is previously obtained, and the relationship between the shooting position of the first camera 52 and the measurement position of the LiDAR sensor 50 is previously obtained.

[0027] The first image is a distortion-corrected RGB or grayscale image, and the second image is a distortion-corrected RGB or grayscale image. The three-dimensional point cloud P is a collection of three-dimensional points measured by the LiDAR sensor 50. Each three-dimensional point is a three-dimensional vector, and when the three-dimensional point cloud P includes N three-dimensional points, the three-dimensional point cloud P is a collection of three-dimensional vectors having N elements.

[0028] In addition, the input unit 15 receives as input the internal parameters K_1 of the first camera 52, the internal parameters K_2 of the second camera 54, the rotation matrix R_C between the first camera 52 and the second camera 54, the translation vector T_C between the first camera 52 and the second camera 54, the projection matrix R_L between the first camera 52 and the LiDAR sensor 50, and the translation vector T_L between the first camera 52 and the LiDAR sensor 50.

[0029] The internal parameters K_1 and K_2 of the first camera 52 and the second camera 54 are 3×3 camera internal parameter matrices. The rotation matrix R_C between the first camera 52 and the second camera 54 is a 3×3 rotation matrix. The translation vector T_C between the first camera 52 and the second camera 54 is a three-dimensional vector. The projection matrix R_L between the first camera 52 and the LiDAR sensor 50 is a 3×3 rotation matrix. The translation vector T_L between the first camera 52 and the LiDAR sensor 50 is a three-dimensional vector.

[0030] The display unit 16 is, for example, a liquid crystal display, and displays various information including the results of upsampling the three-dimensional point cloud P measured by the LiDAR sensor 50. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0031] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0032] Next, a description will be given of the functional configuration of the three-dimensional point cloud processing device 10. FIG.

[0033] The three-dimensional point cloud processing device 10 functionally comprises an input processing unit 20, a neighborhood extraction unit 22, a neighborhood selection unit 24, and a smoothing unit 26, as shown in FIG.

[0034] Based on the first image, the second image, and the three-dimensional point group P received by the input unit 15, the input processing unit 20 projects each three-dimensional point of the three-dimensional point group P onto the first image, and obtains a pixel position on the first image corresponding to each three-dimensional point (see FIG. 4(C)). FIG. 4(C) shows an example in which one of the three-dimensional points on the surface of an object represented by dots is erroneously projected onto a pixel position within an area representing a white object on the first image.

[0035] Specifically, each 3D point of the 3D point group P is projected onto the first image, and the 3D points projected outside the area of ​​the first image are removed from the 3D point group P. A set Q_1 is obtained that is made up of elements that are combinations of pixel positions and depth values ​​on the first image corresponding to each 3D point of the 3D point group P after the removal. The points indicated by each element of set Q_1 are projected onto the second image, and the elements indicating the points projected outside the area of ​​the second image are removed from set Q_1, and the set is updated to the set Q_1 after the removal.

[0036] For example, for each of the three-dimensional points included in the three-dimensional point group P, the projection matrix R_L and the translation vector T_L are applied according to the following formula to obtain the three-dimensional point group P_1. The three-dimensional point group P_1 is a set of N three-dimensional vectors.

[0037] P_1=R_L P+T_L

[0038] The 3D point group P_1 is projected onto the first image of the first camera 52 using the internal parameter K_1, and those projected outside the region of the first image are removed to obtain a set Q_1 of elements consisting of (x coordinate in the first image, y coordinate in the first image, depth value of the 3D point). For example, (x, y, d) calculated according to the following formula for each 3D point p (∈P_1) is set as an element of the set Q_1. (X,Y,d)=K_1 p x=X / d y=Y / d The number of points projected within the region of the first image is denoted as M_1. Note that the correspondence between each element of the set Q_1 and each 3D point of the 3D point group P before the transformation is maintained.

[0039] Then, for each of the 3D points included in the 3D point group P_1, the projection matrix R_C and the translation vector T_C are applied according to the following formula to obtain the 3D point group P_2. The 3D point group P_2 is a set of N 3D vectors.

[0040] P_2=R_C P_1+T_C

[0041] The 3D point group P_2 is projected onto the second image of the second camera 54 using the internal parameter K_2, and points projected outside the area of ​​the second image are removed from the 3D point group P_2 to obtain a set Q_2 of elements consisting of (x coordinate in the second image, y coordinate in the second image, depth value of the 3D point). The number of points projected within the area of ​​the second image is denoted as M_2. Note that the correspondence between each element of the set Q_2 and each element of the 3D point group P before conversion is maintained.

[0042] Elements of set Q_1 that are also included in set Q_2 are extracted using correspondence with the three-dimensional point group P, and set Q_1 is updated so that it becomes a set consisting of elements that are included in both set Q_1 and set Q_2.

[0043] Based on the pixel positions on the first image corresponding to each of the three-dimensional points in the three-dimensional point group P, the neighborhood extraction unit 22 extracts, for each pixel position on the first image, depth values ​​of three-dimensional points corresponding to multiple neighboring pixel positions.

[0044] Specifically, for pixels in the first image, elements having nearby pixel positions are extracted from among the elements contained in set Q_1. As a result, a neighborhood set Q_1_xy is extracted for each pixel position (x, y). The neighborhood set Q_1_xy for each pixel position (x, y) is a subset of set Q_1. Furthermore, for pixel positions (x, y) for which no elements having nearby pixel positions have been extracted, the neighborhood set Q_1_xy is set as an empty set.

[0045] The definition of neighborhood is arbitrary, but for example, neighborhood may be within a certain radius from a pixel position (FIG. 5), or K neighborhoods from a pixel position. FIG. 5 shows an example in which elements having pixel positions that exist within a search radius from a pixel position of interest are extracted as a neighborhood set Q_1_xy.

[0046] In the belief propagation unit 32 described later, the pixel position (x, y) is assigned the depth value of one of the elements included in the neighborhood set Q_1_xy.

[0047] The neighborhood selection unit 24 selects a depth value from the depth values ​​of the elements of the neighborhood set Q_1_xy for each pixel position (x, y) in the first image such that there is high consistency between the first image and the second image and the distance between the pixel position (x, y) and the pixel position of the element is short.

[0048] Specifically, the neighborhood selector 24 includes a cost calculator 30 and a belief propagation unit 32 .

[0049] The cost calculation unit 30 projects a point represented by using the pixel position (x, y) and the depth value of each element included in the neighborhood set Q_1_xy for each pixel position (x, y) on the first image onto the second image, and obtains the distance between the pixel value of the pixel position (x, y) on the first image and the pixel value of the pixel position where the point is projected on the second image. The cost calculation unit 30 calculates a cost function represented by using the distance between the pixel position (x, y) on the first image and the pixel position of the element, and the distance between the pixel value of the pixel position (x, y) on the first image obtained above and the pixel value of the pixel position where the point is projected on the second image for each element included in the neighborhood set Q_1_xy for each pixel position (x, y) on the first image.

[0050] Specifically, the cost calculation unit 30 calculates a cost function for each element q=(q_x, q_y, d) included in the neighborhood set Q_1_xy. The cost function evaluates the likelihood that a pixel position (x, y) has a depth value d of the element q.

[0051] First, it is derived where on the second image the pixel position (x, y) of the first image is projected when it has a depth value of d. This can be obtained by inversely projecting a point (x, y, d) expressed using the pixel position (x, y) and depth value d of the first image by the inverse matrix of the internal parameter K_1, applying the projection matrix R_C and the translation vector T_C to perform coordinate transformation into the coordinate system of the second image of the second camera 54, and then projecting onto the second image plane by the internal parameter K_2. The pixel position on the second image obtained in this way is assumed to be (x', y'). For example, the coordinates (x', y') of the pixel position (x, y) projected onto the second image using the depth value d are calculated according to the following formula. (X',Y',d')=K_2(RC K_1 -1 (d×x,d×y,d)+T_C) x'=X' / d' y'=Y' / d' As a result, the cost function c_xyq when the pixel position (x, y) has a depth value of q can be calculated as follows.

[0052] c_xyq=w_S Stereo(I_1,I_2,x,y,x',y')+w_D Distance(x,y,q_x,q_y)

[0053] Stereo is a function that evaluates the difference in pixel values ​​at pixel position (x, y) in the first image and pixel position (x', y') in the second image. There are methods that take the difference in pixel values, the Hamming distance between images that have been census transformed, and the difference in the gradient of pixel values ​​(see Non-Patent Document 3), and the weighting and window size are arbitrary.

[0054] Distance is the distance between the pixel position (x, y) of the first image and the pixel position (q_x, q_y) of the element, and is the l1 distance, l2 distance, Huber distance, or the like, and a truncated value of the distance may also be used.

[0055] w_S and w_D are the respective weights. c_xyq means that when a depth value d is assigned to a pixel position (x,y), the first and second images are highly consistent, and it is easy to assign the depth value of a 3D point projected near the pixel position (x,y).

[0056] The belief propagation unit 32 selects a depth value of a 3D point for each pixel position (x, y) on the first image based on a cost function and an energy function expressed using the difference from the depth values ​​selected at adjacent pixels.

[0057] Specifically, the belief propagation unit 32 defines an energy function that takes into account a smoothing term in addition to the cost function described above, and selects the depth value d(x, y) that minimizes the energy function for each pixel position (x, y) of the first image by belief propagation. The energy function E is defined by the following equation.

[0058] JPEG0007613690000001.jpg1985

[0059] norm(∇d(x,y)) means the norm of the difference between adjacent depth values ​​d(x,y) (such as l1 distance, l2 distance, or Huber distance), and a truncated norm may be used. The smaller the difference in depth values ​​between adjacent pixels, the smaller the norm(∇d(x,y)) becomes. For example, norm(∇d(x,y)) is the sum of the norms of the selected depth value differences (such as l1 distance, l2 distance, or Huber distance) calculated for the pixel positions adjacent to the right and below the pixel position of interest. Here is a specific example of how to calculate the difference with the pixel position adjacent to the right. The calculation of the difference with the pixel position adjacent below is similar. At the pixel position (x, y) of interest in the first image, a depth value d is selected from Q_1_xy, and at the adjacent pixel position (x+1, y), a depth value d' is selected from Q_1_x+1_y. Then, the difference between them (d-d') is derived and its norm is calculated.

[0060] By minimizing the energy function E, the optimal depth value d(x,y) can be determined for each pixel position (x,y) taking into account the stereo error, the closeness of the projection position, and the smoothness of the depth value at each pixel position.

[0061] The energy function E can be minimized by a technique called belief propagation, more specifically, loopy belief propagation.

[0062] As described above, the belief propagation unit 32 determines a combination of depth values ​​d(x, y) selected from the neighborhood set Q_1_xy for each of all pixel positions (x, y) by minimizing the energy function E according to the following equation.

[0063] d(x,y)=argmin E

[0064] The smoothing unit 26 smoothes the depth value d(x, y) selected for each pixel position (x, y) on the first image.

[0065] Here, the depth value d(x, y) is derived by assigning the depth value of a 3D point projected to a nearby pixel position by the belief propagation unit 32, and is therefore a discontinuous map. Therefore, the smoothing unit 26 according to this embodiment performs smoothing using a variational method, which is a smoothing method for continuous functions, generates a smoothed depth value d_v(x, y), and outputs it to the display unit 16 as a depth image.

[0066] Specifically, the following energy function E_V is minimized using the variational method.

[0067] JPEG0007613690000002.jpg14118

[0068] norm1 and norm2 are distances such as the l1 distance, the l2 distance, or the Huber distance. G is the ADT (Anisotropic Diffusion Tensor) or B-ADT (Binary Anisotropic Diffusion Tensor) described in Non-Patent Document 4, and weights the smoothing term norm2(∇d_V(x,y)) for each position in the image.

[0069] The minimization of the energy function E_V can be performed by the first order primal dual algorithm as described in Non-Patent Document 4. This allows a combination of depth values ​​d_V(x,y) for all pixel positions (x,y) after smoothing to be obtained according to the following formula:

[0070] d_V(x,y)=argmin E_V

[0071] [Non-Patent Document 4] Yao, Yasuhiro, et al. "Discontinuous and Smooth Depth Completion with Binary Anisotropic Diffusion Tensor." IEEE Robotics and Automation Letters 5.4 (2020): 5128-5135.

[0072] <Action of the 3D point cloud processing device according to this embodiment> Next, the operation of the three-dimensional point cloud processing device 10 will be described.

[0073] FIG. 6 is a flowchart showing the flow of 3D point cloud processing by the 3D point cloud processing device 10. The CPU 11 reads out a 3D point cloud processing program from the ROM 12 or the storage 14, expands it in the RAM 13, and executes it to perform 3D point cloud processing. In addition, the 3D point cloud P measured by the LiDAR sensor 50, the first image taken by the first camera 52, and the second image taken by the second camera 54 are input to the 3D point cloud processing device 10. In addition, it is assumed that the internal parameters K_1 of the first camera 52, the internal parameters K_2 of the second camera 54, the rotation matrix R_C between the first camera 52 and the second camera 54, the translation vector T_C between the first camera 52 and the second camera 54, the projection matrix R_L between the first camera 52 and the LiDAR sensor 50, and the translation vector T_L between the first camera 52 and the LiDAR sensor 50 are input to the 3D point cloud processing device 10.

[0074] In step S100, the CPU 11 functioning as the input processing unit 20 acquires the first image, the second image, and the three-dimensional point group received by the input unit 15.

[0075] In step S102, the CPU 11, functioning as the input processing unit 20, calculates sets Q_1 and Q_2 based on the first and second images and the three-dimensional point group.

[0076] In step S104, the CPU 11, functioning as the neighborhood extraction unit 22, extracts a neighborhood set Q_1_xy for each pixel position (x, y) on the first image based on the pixel position on the first image corresponding to each 3D point of the 3D point group P.

[0077] In step S106, the CPU 11, functioning as the cost calculation unit 30, calculates a cost function for each element q=(q_x, q_y, d) included in the neighborhood set Q_1_xy for each pixel position (x, y) on the first image.

[0078] In step S108, the CPU 11, as the belief propagation unit 32, selects, for each pixel position (x, y) on the first image, one of the depth values ​​of each element included in the neighborhood set Q_1_xy based on a cost function and an energy function expressed using the difference from the depth value selected in an adjacent pixel.

[0079] In step S110, the CPU 11, as the smoothing unit 26, smoothes the selected depth values ​​for each pixel position on the first image, displays the smoothed depth values ​​as a depth image on the display unit 16, and terminates the 3D point cloud processing routine.

[0080] As described above, the 3D point cloud processing device according to the present embodiment selects a depth value for each pixel position on the first image based on the pixel position on the first image onto which each 3D point of the 3D point cloud measured by the LiDAR sensor is projected, from the depth values ​​of 3D points corresponding to a plurality of neighboring pixel positions, so that the consistency between the first image and the second image is high and the distance to the neighboring pixel positions is short. This allows the 3D point cloud obtained by measurement to be upsampled with high accuracy.

[0081] <Modification> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0082] For example, although the case where a 3D point cloud is acquired by measurement using a LiDAR sensor has been described as an example, the present invention is not limited to this. A sensor other than a LiDAR sensor may be used to measure the 3D point cloud.

[0083] In addition, although the first image and the second image are taken by different cameras in the above example, the present invention is not limited to this example. As long as the relationship between the photographing positions is determined in advance, the first image and the second image may be taken by the same camera.

[0084] In addition, various types of processing that the CPU reads and executes software (programs) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electric circuits that are processors having circuit configurations designed exclusively for executing specific processing, such as GPUs (Graphics Processing Units), PLDs (Programmable Logic Devices) whose circuit configurations can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits). In addition, the 3D point cloud processing may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0085] In addition, in each of the above embodiments, the three-dimensional point cloud processing program is pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0086] The following supplementary notes are further disclosed regarding the above embodiment.

[0087] (Additional note 1) A three-dimensional point cloud processing device, comprising: Memory, at least one processor coupled to the memory; Including, The processor, receiving a first image and a second image, the relationship between at least the shooting positions of which is previously determined, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is previously determined, and determining pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud; Based on pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, the depth value is selected from depth values ​​of three-dimensional points corresponding to a plurality of neighboring pixel positions for each of the pixel positions on the first image such that consistency between the first image and the second image is high. A 3D point cloud processing device configured as follows.

[0088] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform 3D point cloud processing, The three-dimensional point cloud processing includes: receiving a first image and a second image, the relationship between at least the shooting positions of which is previously determined, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is previously determined, and determining pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud; Based on pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, the depth value is selected from depth values ​​of three-dimensional points corresponding to a plurality of neighboring pixel positions for each of the pixel positions on the first image such that consistency between the first image and the second image is high. Non-transitory storage medium. [Explanation of symbols]

[0089] 10 3D point cloud processing device 14. Storage 15 Input section 16 Display 20 Input Processing Section 22 Neighborhood Extraction Unit 24 Neighborhood Selection Section 26 Smoothing section 30 Cost Calculation Department 32 Belief Propagation Department 50 LiDAR sensors 52 Camera No. 1 54 Second Camera

Claims

1. an input processing unit that receives a first image and a second image, the relationship between at least the shooting positions of which is determined in advance, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is determined in advance, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud; a neighborhood selection unit that selects, for each pixel position on the first image, a depth value from depth values ​​of three-dimensional points corresponding to a plurality of neighboring pixel positions based on a pixel position on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, such that consistency between the first image and the second image is high; A three-dimensional point cloud processing device comprising:

2. The neighborhood selection unit is For each pixel location on the first image, For each of the three-dimensional points corresponding to the plurality of neighboring pixel positions, a point consisting of the pixel position on the first image and a depth value of the three-dimensional point is projected onto the second image, and a distance between a pixel value of the pixel position on the first image and a pixel value of the pixel position of the point projected onto the second image is calculated; 2. The three-dimensional point cloud processing device according to claim 1, further comprising: a cost function expressed using a distance between a pixel value of a pixel position on the first image and a pixel value of a pixel position of a point projected onto the second image, the cost function being expressed using a distance between a pixel value of a pixel position on the first image and a pixel value of a pixel position of a point projected onto the second image, the cost function being expressed using a cost function ...

3. The neighborhood selection unit is For each pixel location on the first image, 3. The three-dimensional point cloud processing device according to claim 2, wherein the depth value is selected based on the cost function and an energy function expressed using a difference between a depth value selected at a pixel position of interest on the first image and a depth value selected at an adjacent pixel position.

4. The three-dimensional point cloud processing device according to any one of claims 1 to 3, further comprising a smoothing unit that smoothes the depth value selected for each pixel position on the first image.

5. A 3D point cloud processing device as described in any one of claims 1 to 4, wherein the neighborhood selection unit selects, for each pixel position on the first image based on a pixel position on the first image corresponding to each of the 3D points of the 3D point cloud, the depth value from the depth values ​​of 3D points corresponding to multiple nearby pixel positions, so that there is high consistency between the first image and the second image and the distance to the nearby pixel positions is short.

6. The input processing unit: projecting each of the three-dimensional points of the three-dimensional point cloud onto the first image, removing three-dimensional points projected outside the area of ​​the first image, and obtaining a set of elements that are combinations of pixel positions and depth values ​​on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud after the removal; projecting the points of each element of the set onto the second image, removing elements projected outside the region of the second image, and updating the set to the removed elements; The neighborhood selection unit is Based on pixel positions of the elements of the set on the first image, a depth value of the element of the set having a plurality of neighboring pixel positions is selected for each pixel position on the first image from depth values ​​of the elements of the set such that consistency between the first image and the second image is high and the distance to the neighboring pixel positions is short. The three-dimensional point cloud processing device according to any one of claims 1 to 5.

7. an input processing unit receives a first image and a second image, the relationship between at least the shooting positions of which is determined in advance, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is determined in advance, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud; A neighborhood selection unit selects, for each pixel position on the first image, a depth value from depth values ​​of three-dimensional points corresponding to a plurality of neighboring pixel positions based on a pixel position on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, such that consistency between the first image and the second image is high. 3D point cloud processing method.

8. A three-dimensional point cloud processing program for causing a computer to function as the three-dimensional point cloud processing device according to any one of claims 1 to 6.

Images