Three-dimensional point cloud segmentation device, three-dimensional point cloud segmentation method, and three-dimensional point cloud segmentation program

Abstract

A nearby point group for each point included in a first three-dimensional point group in which each point has three-dimensional coordinates is searched for by a search unit (24) from a second three-dimensional point group in which the density of the included points is higher than that of the first three-dimensional point group and each point has three-dimensional coordinates and information derived from an image. An object corresponding to each point included in the first three-dimensional point group is inferred by an inference unit (26) on the basis of a feature extracted from the nearby point groups searched for by the search unit (24).

Description

[Technical Field]

[0001] The disclosed technology relates to a three-dimensional point cloud segmentation device, a three-dimensional point cloud segmentation method, and a three-dimensional point cloud segmentation program. [Background technology]

[0002] Many methods have been proposed for performing semantic segmentation on 3D point clouds using deep learning. For example, PointNet++ and KPConv have been reported as segmentation methods in recent years (Non-Patent Documents 1 and 2).

[0003] Generally, accurate measurement of three-dimensional points is performed using a TOF (Time Of Flight) distance sensor such as LiDAR. With LiDAR, a laser pulse is emitted into the surrounding area, and the distance to the target is obtained from the time it takes for the laser pulse to be reflected back to the LiDAR. In addition, since the direction of the laser emission is known, the three-dimensional coordinates of each point can be obtained based on the distance and direction information.

[0004] However, in the case of LiDAR, 3D points can only be measured within the number of pulses emitted, so the resulting 3D point cloud may be low density. In particular, lower-cost LiDAR tends to emit fewer pulses in a given period, resulting in a low-density measured point cloud.

[0005] Furthermore, with regard to the density of point clouds, a method has been proposed for increasing the density of low-density point clouds using images (Patent Document 1 and Non-Patent Document 3). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-174406 [Non-patent literature]

[0007] [Non-Patent Document 1] Qi, CR, Yi, L., Su, H., & Guibas, LJ, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space", Advances in Neural Information Processing Systems, 30, 2017 [Non-patent document 2] Thomas, H., Qi, CR, Deschaud, JE, Marcotegui, B., Goulette, F., & Guibas, LJ, "Kpconv: Flexible and deformable convolution for point clouds", In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6411-6420), 2019 [Non-patent document 3] Yao, Y., Ishikawa, R., Ando, ​​S., Kurata, K., Ito, N., Shimamura, J., & Oishi, T., "Non-Learning Stereo-Aided Depth Completion Under Mis-Projection via Selective Stereo Matching", IEEE Access, 9, 136674-136686, 2021 Summary of the Invention [Problem to be solved by the invention]

[0008] In conventional techniques such as those described in Non-Patent Documents 1 and 2, segmentation is performed by extracting features only from the target 3D point cloud. Therefore, if the target 3D point cloud has a low density, the shape features of the object cannot be captured, resulting in a loss of information and a failure of segmentation.

[0009] Furthermore, even if conventional techniques such as those described in Patent Document 1 and Non-Patent Document 3 are simply applied to densify a low-density three-dimensional point cloud, it may not be possible to obtain segmentation results with sufficient accuracy.

[0010] The present disclosure has been made in view of the above points, and aims to accurately perform segmentation of a three-dimensional point cloud. [Means for solving the problem]

[0011] A first aspect of the present disclosure is a three-dimensional point cloud segmentation device including: a search unit that searches for a neighboring point cloud for each point included in a first three-dimensional point cloud, each point having three-dimensional coordinates, from a second three-dimensional point cloud, each point having three-dimensional coordinates and information derived from an image, and having a higher density of points included in the first three-dimensional point cloud; and an inference unit that infers objects corresponding to each point included in the first three-dimensional point cloud based on features extracted from the neighboring point cloud searched by the search unit.

[0012] A second aspect of the present disclosure is a three-dimensional point cloud segmentation method, in which a search unit searches for a neighboring point cloud for each point included in a first three-dimensional point cloud, each point having three-dimensional coordinates, from a second three-dimensional point cloud, each point having three-dimensional coordinates and information derived from an image, and the density of points included is higher than that of the first three-dimensional point cloud, and an inference unit infers an object corresponding to each point included in the first three-dimensional point cloud based on features extracted from the neighboring point cloud searched by the search unit.

[0013] A third aspect of the present disclosure is a three-dimensional point cloud segmentation program for causing a computer to function as each unit of the above-described three-dimensional point cloud segmentation device. [Effects of the Invention]

[0014] The disclosed technique allows accurate segmentation of 3D point clouds. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a three-dimensional point cloud segmentation device. [Figure 2] FIG. 1 is a functional block diagram of a three-dimensional point cloud segmentation device. [Figure 3] FIG. 10 is a diagram illustrating an example of input data. [Figure 4] FIG. 10 is a diagram for explaining an example of densification. [Figure 5] FIG. 10 is a diagram illustrating an example of searching for a neighborhood point group. [Figure 6] FIG. 10 is a diagram illustrating an example of an inference model. [Figure 7] 10 is a flowchart illustrating an example of a learning process. [Figure 8] 10 is a flowchart illustrating an example of an inference process. [Figure 9] FIG. 10 is a diagram for explaining an example of an experimental result. [Figure 10] FIG. 10 is a diagram illustrating an example of a segmentation result. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0017] First, an outline of this embodiment will be described.

[0018] The 3D point cloud segmentation device of this embodiment is a device that performs semantic segmentation learning and inference on a low-density 3D point cloud (hereinafter referred to as "3D point cloud A") measured using a low-resolution LiDAR or the like.

[0019] The 3D point cloud segmentation device according to this embodiment receives a 3D point cloud A as input and generates a point cloud (hereinafter referred to as "densified point cloud B") by densifying the 3D point cloud A using an image. That is, the density of points contained in each point cloud is higher in densified point cloud B than in 3D point cloud A. Therefore, in this embodiment, when comparing 3D point cloud A and densified point cloud B, the density of 3D point cloud A is defined as low density, and the density of densified point cloud B is defined as high density.

[0020] The 3D point cloud segmentation device also learns parameters of an inference model for performing segmentation, and performs inference, i.e., segmentation of the 3D point cloud, using the inference model to which the learned parameters are applied. Furthermore, the 3D point cloud segmentation device also updates parameters used when densifying the 3D point cloud A using the aforementioned learned labels for segmentation (details will be described later).

[0021] Next, the configuration of the 3D point cloud segmentation device according to this embodiment will be described.

[0022] Fig. 1 is a block diagram showing the hardware configuration of a 3D point cloud segmentation device 10 according to this embodiment. As shown in Fig. 1, the 3D point cloud segmentation device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0023] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each component and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a three-dimensional point cloud segmentation program for executing the learning process and inference process described below.

[0024] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs and data as a working area. The storage 14 is composed of storage devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data, including the operating system.

[0025] The input unit 15 includes, for example, a pointing device such as a mouse and a keyboard, and is used to perform various inputs. The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0026] The communication I / F 17 is an interface for communicating with other devices, and the communication uses wireless communication standards such as 4G, 5G, or Wi-Fi (registered trademark).

[0027] Next, the functional configuration of the three-dimensional point cloud segmentation device 10 according to this embodiment will be described.

[0028] FIG. 2 is a block diagram showing an example of the functional configuration of a 3D point cloud segmentation device 10. As shown in FIG. 2, the 3D point cloud segmentation device 10 includes, as its functional configuration, a densification unit 22, a search unit 24, an inference unit 26, a learning unit 28, and an update unit 30. The inference unit 26 further includes a neighboring point cloud feature extraction unit 26a, a total point cloud feature extraction unit 26b, and a classification unit 26c. The 3D point cloud segmentation device 10 also manages various information using databases. The databases include, for example, an image parameter DB (database) 32, a 3D point cloud DB 34, a learning label DB 36, a densification parameter DB 38, and a densification point cloud DB 40. The databases also include, for example, a DNN (Deep Neural Network) parameter DB 42 and an inference label DB 44. Each functional configuration is realized by the CPU 11 reading out a three-dimensional point cloud segmentation program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0029] The densification unit 22 generates a densified point group B by densifying the three-dimensional point group A based on the correspondence between the three-dimensional point group A and an image A obtained by capturing a space including the three-dimensional point group A. Specifically, the densification unit 22 acquires, as input data, the three-dimensional point group A, a densification parameter Pδ, the image A, the internal parameters of the camera, and the external parameters of the camera from a corresponding database.

[0030] The three-dimensional point cloud A is stored in the three-dimensional point cloud DB 34. The left side of Figure 3 shows a schematic diagram of the three-dimensional point cloud A. The three-dimensional point cloud A is point cloud data in which each point has three-dimensional coordinates. The three-dimensional point cloud A is a low-density point cloud acquired by measurements such as LiDAR, but it is a three-dimensional point cloud with accurate position information and little noise. The number of points contained in the three-dimensional point cloud A is the number (L) of points whose labels are inferred at one time by the inference model described below. If the number of points contained in the three-dimensional point cloud is greater than L, the three-dimensional point cloud A is obtained by processing the input points in advance so that the number of input points becomes L.

[0031] Image A is stored in the image parameter DB 32. An example of image A is shown in the center of Figure 3. Image A is an image captured of a space including 3D point cloud A, i.e., the location where 3D point cloud A was measured. The position and orientation of the camera that captured image A are known from the extrinsic parameters (rotation and translation in three-dimensional space) relative to the origin of the coordinate system of 3D point cloud A. Three-dimensional coordinates are converted to two-dimensional coordinates of the image using the position and orientation of the camera specified by the extrinsic parameters and the camera's intrinsic parameters. The camera's extrinsic and intrinsic parameters are stored in the image parameter DB 32 in association with image A. Furthermore, for one 3D point cloud A, there are K images A, the number of which depends on the densification method described below, and each of the K images A has its own extrinsic and intrinsic camera parameters.

[0032] The densification parameter Pδ is a set of multiple parameters according to a densification method described later, and is stored in the densification parameter DB 38. In the initial stage of learning, a default value determined in advance as the densification parameter Pδ is stored in the densification parameter DB 38. Furthermore, during inference, the final densification parameter Pδ updated during learning is stored in the densification parameter DB 38.

[0033] Specifically, as shown in Fig. 4, the densification unit 22 associates the three-dimensional point group A with the image A, applies a densification parameter Pδ, and densifies the three-dimensional point group A using the image A as a clue to generate a densified point group B. The densification unit 22 may use, for example, the method described in Patent Document 1 or Non-Patent Document 3 as a densification method. An example of the densification method will be described below.

[0034] The densification unit 22 converts the three-dimensional point cloud A into a depth map a using the external and internal parameters of the camera. Since the depth map a is created by projecting each point of the low-density three-dimensional point cloud A onto an image, most of the pixels have no value. The densification unit 22 inputs the depth map a and image A and performs a densification process to generate a depth map b in which all pixels have depth values. The densification unit 22 converts the depth map b back into a three-dimensional point cloud using the external and internal parameters of the camera to generate the three-dimensional point cloud b. The densification unit 22 generates a three-dimensional point cloud b for each of the K images A, and generates a densified point cloud B by combining the K three-dimensional point clouds b.

[0035] The densification unit 22 stores the generated densified point cloud B in the densified point cloud DB 40. Note that the densified point cloud B has a larger error in position information than a point cloud measured by a TOF distance sensor, but is a three-dimensional point cloud with a higher density. In addition, each pixel of image A has color information, and the densified point cloud B generated using image A also holds information derived from such an image (hereinafter referred to as "image-derived information").

[0036] The search unit 24 acquires the densified point cloud B from the densified point cloud DB 40, searches for neighbors for each point of the three-dimensional point cloud A from the densified point cloud B, and acquires a group of neighboring points (hereinafter referred to as "neighboring point cloud C"). If the three-dimensional point cloud A contains L points and N neighboring points are acquired for each point of the three-dimensional point cloud A, the neighboring point cloud C contains L points × N points. Furthermore, if each point of the densified point cloud B contains m-dimensional information, the neighboring point cloud C will be L points × N × m-dimensional information.

[0037] 5, the search unit 24 overlaps the densified point cloud B with the three-dimensional point cloud A, and samples a predetermined number (here, N points) of points in the densified point cloud B that are included within a radius r (right side of FIG. 5) of a point a included in the three-dimensional point cloud A. The search unit 24 acquires a set of the sampled points as a neighborhood point cloud C of the point a.

[0038] More specifically, the search process is performed in the following order, for example. 1. Construct a KD tree (T_B) from the densified point cloud B. 2. For a point a in the three-dimensional point cloud A, use T_B to obtain a subset (B_a) of the densified point cloud B that is within a radius r of point a. 3. For the elements of B_a, perform coordinate transformation by parallel translation to relative coordinates with point a as the origin. 4. If the number of elements in B_a is greater than N, randomly sample N points from B_a. 5. If the number of elements in B_a is smaller than N, add an element with all elements set to zero to B_a, and set the number of elements to N. 6. Perform steps 2 to 5 for all points (L points) included in the three-dimensional point cloud A to obtain a neighborhood point cloud C (L points x N x m dimensions).

[0039] The inference unit 26 includes a nearby point group feature extraction unit 26a, a total point group feature extraction unit 26b, and a classification unit 26c, and infers objects corresponding to each point included in the three-dimensional point group A based on features extracted from the nearby point group C searched by the search unit 24. Specifically, the inference unit 26 inputs the nearby point group C to an inference model for inferring labels indicating objects corresponding to each point in the three-dimensional point group A, and acquires inference labels output from the inference model. During learning, the inference unit 26 passes the acquired inference labels to the learning unit 28 and the update unit 30, and during inference, stores the acquired inference labels in the inference label DB 44.

[0040] In this embodiment, for example, as shown in FIG. 6, an inference model including three types of DNNs, DNNα, DNNβ, and DNNγ, is used. A DNN parameter Pα of DNNα, a DNN parameter Pβ of DNNβ, and a DNN parameter Pγ of DNNγ are stored in the DNN parameter DB 42. Each of the DNN parameters Pα, Pβ, and Pγ is a set of multiple parameters, specifically, the weights and bias values ​​of the edges of the DNN. In the initial stage of learning, the DNN parameter DB 42 stores the DNN parameters Pα, Pβ, and Pγ initialized by random numbers. Furthermore, during inference, the final DNN parameters Pα, Pβ, and Pγ updated during learning are stored in the DNN parameter DB 42.

[0041] Below, details of the neighboring point cloud feature extractor 26a, the total point cloud feature extractor 26b, and the classifier 26c, as well as details of DNNα, DNNβ, and DNNγ will be described.

[0042] The neighboring point cloud feature extraction unit 26a uses DNNα independently for each point in the three-dimensional point cloud A to extract features of the neighboring point cloud C searched for each point. Specifically, the neighboring point cloud feature extraction unit 26a receives the neighboring point cloud C from the search unit 24 as input data and acquires DNN parameters Pα from the DNN parameter DB 42. The neighboring point cloud feature extraction unit 26a inputs the neighboring point cloud C for each of the L points in the three-dimensional point cloud A to L DNNαs, each having the same DNN parameter Pα. The DNNα independently applies CNN (Convolutional Neural Network) processing to the input of the N×m-dimensional neighboring point cloud C. More specifically, the DNNα applies convolution, activation, batch normalization, and dropout processing across multiple layers. As a result, the DNNα outputs S-dimensional features for each neighboring point cloud C. By having L DNNα perform the above processing on L points, the neighborhood point cloud feature extraction unit 26a derives neighborhood point cloud features F_C of L points×S dimensions.

[0043] The total point cloud feature extraction unit 26b uses one DNNβ for the three-dimensional point cloud A to extract features for classifying objects corresponding to each point included in the three-dimensional point cloud A from the neighboring point cloud features F_C extracted by the neighboring point cloud feature extraction unit 26a. Specifically, the total point cloud feature extraction unit 26b receives the neighboring point cloud features F_C from the neighboring point cloud feature extraction unit 26a as input data and acquires DNN parameters Pβ from the DNN parameter DB 42. The total point cloud feature extraction unit 26b inputs the L-point×S-dimensional neighboring point cloud features F_C to the DNNβ in which the DNN parameters Pβ are set, and derives the L-point×T-dimensional total point cloud features F. The DNNβ may be, for example, a known three-dimensional point cloud segmentation module such as Pointnet++ or KPConv.

[0044] The classification unit 26c classifies objects corresponding to each point included in the 3D point cloud A based on the features extracted by the total point cloud feature extraction unit 26b using a DNNγ for classifying objects corresponding to each point. Specifically, the classification unit 26c receives the total point cloud feature F from the total point cloud feature extraction unit 26b as input data, and acquires the DNN parameter Pγ from the DNN parameter DB 42. The classification unit 26c inputs the total point cloud feature F of L points × T dimensions to the DNNγ in which the DNN parameter Pγ is set. The DNNγ estimates labels independently for the L points and outputs inferred labels of L points × U dimensions. The DNNγ may be composed of, for example, a multi-layer Perceptron and a Softmax layer, and may output inferred labels subjected to One Hot Encoding.

[0045] 3D point clouds measured by LiDAR or other sensors generally do not capture the color of objects, so color information often cannot be utilized, which can lead to segmentation errors. Furthermore, with LiDAR, 3D points can only be measured within the number of emitted pulses, resulting in a low-density 3D point cloud. In this embodiment, as described above, a low-density 3D point cloud A measured by LiDAR or other sensors is densified using an image, and features of a nearby point cloud C containing image-derived information are extracted. These features include color information and texture information based on the image-derived information. Furthermore, because the densification process has been performed, surface information that could not be captured in the low-density 3D point cloud A is also included in the features. In this embodiment, by performing segmentation using these features, segmentation can be performed more accurately than when segmentation is performed using only the position information contained in the 3D point cloud.

[0046] The learning unit 28 uses a training 3D point cloud A in which the correct answers for the objects corresponding to each point are known, and learns the parameters of the inference model so as to minimize the error between the inference result by the inference model and the correct answer. Specifically, the learning unit 28 uses the correct class label (hereinafter referred to as the "trained label") for each point of the 3D point cloud A as the correct answer. For example, the class here refers to the type of object, and in the case of an outdoor point cloud, it would be road, building, utility pole, ground, etc. If there are U classes, the 3D point cloud A with L points may be one-hot encoded and used as an L-point × U-dimensional training label. The training labels are stored in the training label DB 36. The training labels are conceptually shown on the right side of Figure 3. In the example of Figure 3, the class indicated by the training label is represented by the pattern of the points corresponding to each point of the 3D point cloud A.

[0047] More specifically, the learning unit 28 receives inference labels from the inference unit 26 as input data and acquires learning labels for the 3D point cloud A that is the subject of inference from the learning label DB 36. The learning unit 28 then updates the DNN parameters Pα, Pβ, and Pγ by backpropagation based on a loss function calculated from the inference labels (L points × U dimensions) and the learning labels (L points × U dimensions). The learning unit 28 evaluates the error (loss function) by, for example, cross entropy. The learning unit 28 terminates learning when the error between the inference labels and the learning labels no longer becomes smaller compared to all iterations, or when the updating of the DNN parameters Pα, Pβ, and Pγ has been repeated a predetermined number of times.

[0048] The update unit 30 updates the densification parameter Pδ applied by the densification unit 22 when densifying the three-dimensional point cloud A so as to increase the accuracy of the positions of each point included in the densified point cloud B. Specifically, the update unit 30 generates multiple parameter sets around the currently set densification parameter Pδ, for example, by incrementing or decrementing the value of the currently set densification parameter Pδ by a predetermined value. The update unit 30 causes the densification unit 22, the search unit 24, and the inference unit 26 to perform a series of processes using the parameter sets of each generated pattern and the DNN parameters Pα, Pβ, and Pγ obtained by the learning unit 28. The update unit 30 then updates the densification parameter Pδ with the parameter set of the pattern that minimizes the error between the inferred label and the learned label. The update unit 30 terminates the update when the error between the inferred label and the learned label no longer becomes smaller than in the previous iteration, or when the update of the densification parameter Pδ has been repeated a predetermined number of times. This updates the densification parameter Pδ.

[0049] In point cloud densification methods such as those described in Patent Document 1 and Non-Patent Document 3, parameters are updated so that the result of densifying a low-density point cloud is closer to the correct high-density point cloud. However, this method requires a correct high-density point cloud measured in the same area as the low-density point cloud to update the parameters, which requires a device that measures a high-density 3D point cloud. Therefore, parameter updating is not easy. In this embodiment, if there are learning labels for a low-density 3D point cloud prepared to train the parameters of a DNN for segmentation, the densification parameters can be updated without requiring a correct high-density point cloud.

[0050] Next, the operation of the 3D point cloud segmentation device 10 according to this embodiment will be described.

[0051] 7 is a flowchart showing the flow of the learning process by the 3D point cloud segmentation device 10. The CPU 11 reads out a 3D point cloud segmentation program from the ROM 12 or storage 14, expands it in the RAM 13, and executes it, thereby performing the learning process.

[0052] First, in step S101, the CPU 11, as the densification unit 22, applies a densification parameter Pδ based on the correspondence between a low-density three-dimensional point group A and an image A captured of a space including the three-dimensional point group A, to generate a densified point group B by densifying the three-dimensional point group A.

[0053] Next, in step S102, the CPU 11 functions as the search unit 24 to search the densified point group B for a neighborhood point group C for each point of the three-dimensional point group A.

[0054] Next, in step S103, the CPU 11, as the neighboring point group feature extraction unit 26a, uses DNNα independently for each point of the three-dimensional point group A to extract neighboring point group features F_C, which are features of the neighboring point group C searched for each point.

[0055] Next, in step S104, the CPU 11, as the total point cloud feature extraction unit 26b, uses one DNNβ for the three-dimensional point cloud A to extract a total point cloud feature F, which is a feature for classifying objects corresponding to each point included in the three-dimensional point cloud A, from the nearby point cloud features F_C.

[0056] Next, in step S105, the CPU 11, as the classification unit 26c, uses DNNγ to classify the objects corresponding to each point, and obtains, from the total point group features F, an inference label which is the classification result of the objects corresponding to each point included in the three-dimensional point group A.

[0057] Next, in step S106, the CPU 11, as the learning unit 28, updates the values ​​of the DNN parameters Pα, Pβ, and Pγ, which are parameters of the inference model, so as to minimize the error between the inference label and the learning label for the three-dimensional point group A to be inferred.

[0058] Next, in step S107, the CPU 11, as the learning unit 28, determines whether or not to terminate learning of the parameters of the inference model. For example, it may determine that learning is to terminate when the error between the inference label and the learning label has not become smaller compared to the previous iteration, or when parameter updates have been repeated a predetermined number of times. If learning is to terminate, proceed to step S108; if not, return to step S102.

[0059] Next, in step S108, the CPU 11, functioning as the update unit 30, generates a plurality of parameter sets around the currently set densification parameter Pδ. The CPU 11, functioning as the update unit 30, also causes the densification unit 22, the search unit 24, and the inference unit 26 to perform a series of processes using the parameter sets of each generated pattern and the DNN parameters Pα, Pβ, and Pγ obtained by the learning unit 28. The CPU 11, functioning as the update unit 30, then updates the densification parameter Pδ with the parameter set of the pattern that minimizes the error between the inferred label and the learned label.

[0060] Next, in step S109, the CPU 11, functioning as the update unit 30, determines whether or not to end the update of the densification parameter Pδ. For example, it may be determined that the update is to be ended when the error between the inferred label and the learned label is no longer smaller than that in the previous iteration, or when the parameter update has been repeated a predetermined number of times. If the update is not to be ended, the process returns to step S101; if it is to be ended, the learning process ends.

[0061] Fig. 8 is a flowchart showing the flow of the inference process executed by the CPU 11 of the three-dimensional point cloud segmentation device 10. The CPU 11 reads out the three-dimensional point cloud segmentation program from the storage device 12, expands it in the memory 13, and executes it, whereby the CPU 11 functions as each functional component of the three-dimensional point cloud segmentation device 10, and executes the inference process shown in Fig. 8. Note that the inference process is executed in a state where the learned DNN parameters Pα, Pβ, and Pγ and the densification parameter Pδ are stored in the DNN parameter DB 42 and the densification parameter DB 38, respectively, as a result of the execution of the above-mentioned learning process.

[0062] In steps S201 to S205, the CPU 11 functions as the densification unit 22, the search unit 24, the neighboring point cloud feature extraction unit 26a, the total point cloud feature extraction unit 26b, and the classification unit 26c to execute the same processes as steps S101 to S105 of the above-described learning process (FIG. 7). As a result, an inference label is acquired for each point of the three-dimensional point cloud A that is the inference target. In step S205, the CPU 11 functions as the classification unit 26c to store the acquired inference label in the inference label DB 44, and the inference process ends.

[0063] As described above, the 3D point cloud segmentation device according to this embodiment generates a densified point cloud containing image-derived information by densifying a sparse 3D point cloud using an image. The 3D point cloud segmentation device also searches for neighboring points of each point in the generated densified point cloud, extracts their features, and uses the features to obtain an inferred label, which is a classification result of the object corresponding to each point in the 3D point cloud. This allows for more accurate segmentation of the 3D point cloud than when segmenting a 3D point cloud containing only positional information.

[0064] Here, experimental results using the 3D point cloud segmentation device according to this embodiment will be described with reference to FIG.

[0065] FIG. 9 shows a comparison of the accuracy of segmentation results between a comparative method and the method of this embodiment (hereinafter referred to as "this method"). The comparative method performs segmentation using only a low-density 3D point cloud measured by a low-resolution LiDAR. Furthermore, as an index of accuracy, Intersection over Union (hereinafter referred to as "IOU") is used, which indicates the degree of agreement between the segmentation result and the correct answer (learned label). Furthermore, FIG. 9 compares the IOU for each class for the segmentation results of both methods.

[0066] 9, the IOU values ​​of the present method are improved in many classes compared to the comparative method. In other words, it can be seen that the 3D point cloud segmentation device according to this embodiment can perform more accurate segmentation.

[0067] Figure 10 shows an example of a 3D point cloud (LiDAR point cloud) measured with a low-resolution LiDAR, a densified point cloud, and a segmentation result. In the densified point cloud, each point actually has color information. The segmentation result is also a result of assigning a different color to each point in the LiDAR point cloud for each inferred class. Each object is assigned the color of the class that indicates that object, which shows that more accurate segmentation can be performed.

[0068] In the above embodiment, the processing unit is a single point cloud including L points, but it may also be processed as a batch. In this case, if the batch size is B, B three-dimensional point clouds each consisting of L points are processed together.

[0069] Furthermore, the learning process and inference process executed by the CPU after reading the software (program) in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The learning process and inference process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0070] In the above embodiment, the 3D point cloud segmentation program is pre-stored (installed) in a ROM or storage device, but the present invention is not limited to this. The program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0071] The following additional notes are provided regarding the above-described embodiments.

[0072] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: searching for a neighboring point group for each point included in a first three-dimensional point group, each point having three-dimensional coordinates, from a second three-dimensional point group, each point having three-dimensional coordinates and information derived from an image, and the density of points included is higher than that of the first three-dimensional point group; inferring an object corresponding to each point included in the first three-dimensional point cloud based on features extracted from the neighboring point cloud searched by the search unit; The three-dimensional point cloud segmentation device is configured as follows.

[0073] (Additional note 2) A non-transitory recording medium storing a program executable by a computer to perform a three-dimensional point cloud segmentation process, The three-dimensional point cloud segmentation process includes: searching for a neighboring point group for each point included in a first three-dimensional point group, each point having three-dimensional coordinates, from a second three-dimensional point group, each point having three-dimensional coordinates and information derived from an image, and the density of points included is higher than that of the first three-dimensional point group; inferring an object corresponding to each point included in the first three-dimensional point cloud based on features extracted from the neighboring point cloud searched by the search unit; Non-transitory recording media, including [Explanation of symbols]

[0074] 10. 3D point cloud segmentation device 11 CPU 12 ROM 13 RAM 14. Storage 15 Input section 16 Output section 17 Communication I / F 19 Bus 22 High density section 24 Exploration Department 26 Reasoning part 26a Nearby point cloud feature extraction unit 26b All point cloud feature extraction section 26c Classification Department 28 Learning Department 30 Update section 32 Image and parameter database 34 3D point cloud DB 36 Learning Label DB 38 Densification parameter DB 40 High-density point cloud DB 42 DNN parameter database 44 Inference Label DB

Claims

1. a search unit that searches for a neighborhood point group for each point included in a first three-dimensional point cloud, each point having three-dimensional coordinates, from points in a second three-dimensional point cloud, each point having three-dimensional coordinates and information derived from an image captured of a space including the first three-dimensional point cloud, and having a higher point density than the first three-dimensional point cloud, for points included within a predetermined distance from each point included in the first three-dimensional point cloud; an inference unit that infers an object corresponding to each point included in the first three-dimensional point cloud based on features extracted from the neighboring point cloud searched by the search unit; A three-dimensional point cloud segmentation device comprising:

2. The 3D point cloud segmentation device according to claim 1 , further comprising a densification unit that generates the second 3D point cloud by densifying the first 3D point cloud based on the correspondence between the first 3D point cloud and the image.

3. The three-dimensional coordinates of each point included in the second three-dimensional point cloud have lower positional accuracy than the three-dimensional coordinates of each point included in the first three-dimensional point cloud. The three-dimensional point cloud segmentation device according to claim 1 or 2.

4. A three-dimensional point cloud segmentation device as described in any one of claims 1 to 3, further comprising a learning unit that uses a first three-dimensional point cloud for learning, in which the correct answer of the object corresponding to each point is known, to learn the parameters of the inference model used to infer the object corresponding to each point so as to minimize the error between the inference result by the inference model and the correct answer.

5. 3. The three-dimensional point cloud segmentation device of claim 2, further comprising an update unit that generates multiple patterns of parameter sets to be applied when densifying the first three-dimensional point cloud by the densification unit, and updates the parameter sets of the densification unit with a parameter set of a pattern that minimizes the error between the inference result of an inference model used to infer the object corresponding to each point when using the second three-dimensional point cloud that has been densified by applying each of the parameter sets of the multiple patterns to a first three-dimensional point cloud for training in which the correct answer for the object corresponding to each point is known.

6. The inference unit a neighboring point cloud feature extraction unit that extracts features of a neighboring point cloud searched for each point included in the first three-dimensional point cloud using an independent first feature extractor for each point included in the first three-dimensional point cloud; a total point cloud feature extraction unit that uses one second feature extractor for the first three-dimensional point cloud to extract features for classifying objects corresponding to each point included in the first three-dimensional point cloud from the features of all the neighboring point clouds extracted by the neighboring point cloud feature extraction unit; and a classification unit that classifies objects corresponding to each point included in the first three-dimensional point cloud based on the features extracted by the total point cloud feature extraction unit using a classifier for classifying objects corresponding to each point; Including, The three-dimensional point cloud segmentation device according to any one of claims 1 to 5.

7. a search unit searches for a neighborhood point group for each point included in a first three-dimensional point cloud, each point having three-dimensional coordinates, from points included within a predetermined distance from each point included in the first three-dimensional point cloud, among points in a second three-dimensional point cloud, each point having three-dimensional coordinates and information derived from an image captured of a space including the first three-dimensional point cloud, and having a higher density of points than the first three-dimensional point cloud; an inference unit inferring an object corresponding to each point included in the first three-dimensional point cloud based on features extracted from the neighboring point cloud searched by the search unit; A three-dimensional point cloud segmentation method.

8. A three-dimensional point cloud segmentation program for causing a computer to function as each part of the three-dimensional point cloud segmentation device according to any one of claims 1 to 6.

Images