3D point cloud target detection method for automatic driving, medium and system

Abstract

The invention discloses a 3D point cloud target detection method for automatic driving, a medium and a system, and the method comprises the steps: S01, obtaining multi-view point cloud data in an automatic driving scene, reconstructing the 3D geometric information and color information of the scene through a neural radiation field, generating extreme scene point cloud data, and carrying out the reconstruction of the extreme scene point cloud data; combining the generated extreme scene point cloud data with real point cloud data to construct a training data set for automatic driving; s02, converting the point cloud data into grids, detecting holes in the point cloud data, repairing the holes to generate point cloud after hole repairing, and combining the point cloud with the original point cloud data to form complete three-dimensional point cloud; and S03, pre-processing the three-dimensional point cloud, inputting the pre-processed three-dimensional point cloud into a local dynamic convolutional neural network, and outputting a point cloud segmentation result, thereby realizing identification and positioning of the component. The method has the advantages of accurate target detection, high detection efficiency and the like.

Description

Technical Field

[0001] The present invention mainly relates to the field of image processing technology, and specifically to a 3D point cloud target detection method, medium and system for autonomous driving. Background Art

[0002] As a new data format, 3D point cloud can accurately record the 3D morphology, geometric features, spatial coordinates and other information of the object surface, and has advantages that 2D images cannot match.

[0003] However, due to equipment occlusion, light reflection, complex geographical environment and equipment accuracy limitations, the collected point cloud data often has incomplete parts, forming holes. The existence of these holes will cause the point cloud model to be discontinuous, affecting the subsequent 3D reconstruction, component detection and analysis tasks. Traditional hole repair methods are mostly based on triangular meshes, and the sparsity and unstructured characteristics of point cloud data pose challenges to the direct application of these methods. Although some geometric reconstruction and surface fitting methods can handle some hole repair problems, these methods often rely on regular geometric features or strong assumptions about the overall structure, and it is difficult to handle other more complex scenarios. In addition, most point cloud-based anomaly detection algorithms require prior information on the location of components, so they need to be pre-positioned by manual annotation. This requires the project to spend a lot of manpower and material resources to pre-annotate the data, greatly weakening the flexibility of the project.

[0004] In the field of deep learning-based autonomous driving, the diversity of training samples is crucial to the performance of artificial intelligence systems. However, some extreme scenarios, such as pedestrians crossing tracks, are unlikely to occur in reality, so it is difficult to collect enough samples. This is a major flaw for autonomous driving algorithms that require a large number of diverse samples for training. How to generate negative samples based on the existing large number of conventional data samples is a hot topic in the field of autonomous driving.

[0005] In summary, in application scenarios such as point cloud-based mapping and positioning, 3D reconstruction, target recognition, and anomaly detection, the scarcity of point cloud holes and negative samples increases the difficulty and complexity of data preprocessing, sample annotation, and algorithm processing, which greatly limits the efficiency and performance of application development and promotion. Summary of the invention

[0006] In view of the technical problems existing in the prior art, the present invention provides a 3D point cloud target detection method, medium and system for autonomous driving with accurate target detection.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A 3D point cloud target detection method for autonomous driving, comprising the steps of:

[0009] S01. Obtain multi-view point cloud data in an autonomous driving scenario, reconstruct the 3D geometric information and color information of the scene using the neural radiation field, generate extreme scene point cloud data, and then combine the generated extreme scene point cloud data with the real point cloud data to construct a training data set for autonomous driving;

[0010] S02, converting the point cloud data into a grid, detecting holes in the point cloud data and performing repair operations on the holes to generate a point cloud after the holes are repaired, and then merging the point cloud data with the original point cloud data to form a complete three-dimensional point cloud;

[0011] S03. After preprocessing the three-dimensional point cloud, it is input into a pre-trained local dynamic convolutional neural network, and the result of point cloud segmentation is output. The background points are removed and the target point cloud is clustered to obtain the point cloud of the inspection area, thereby realizing the recognition and positioning of the components.

[0012] Preferably, the specific process of step S01 is:

[0013] S1.1. Obtain multi-view point cloud data in the autonomous driving scenario;

[0014] S1.2. Reconstruct the 3D geometry and color information of the scene using neural radiation fields based on multi-view point cloud data.

[0015] S1.3, generate a depth map of each ray and calculate the 3D point cloud position using volume rendering of the neural radiance field;

[0016] S1.4, generating LiDAR point cloud data based on 3D point cloud information;

[0017] S1.5. Calculate semantic labels and generate final training data.

[0018] Preferably, the specific process of step S1.4 is:

[0019] S1.4.1: Based on the 3D geometric information generated in step S1.3, add extreme scene dynamic objects; specifically, model the motion of the dynamic objects and set their motion trajectory M(t), where t is a time parameter; simulate the pedestrian's moving path M(t) = {x(t), y(t), z(t)} through a known pedestrian gait model, and synthesize the path with the 3D reconstruction data of the track scene;

[0020] S1.4.2: For each frame of LiDAR data, use the 3D geometry information reconstructed from the neural radiance field to generate the depth value D(r) and color information c(r) corresponding to each ray r;

[0021] S1.4.3: According to the parameters of the LiDAR sensor, set the starting point o and direction d of the virtual light. For each light ray r, calculate the position p of the 3D point according to the depth value D(r); the generated point cloud is represented by the symbol P = {p1, p2, ..., p n}, where n is the number of points in each frame of LiDAR point cloud;

[0022] S1.4.4: Synthesize the full point cloud in the scene using the multi-frame point cloud data generated in step S1.4.3; for each frame f i , there is corresponding point cloud data P i ; Through the time step Δt, multiple frame point clouds at different time points are synthesized into dynamic scene point cloud P total :

[0023]

[0024] Where N is the number of frames in the time step.

[0025] Preferably, the specific process of step S1.5 is:

[0026] S1.5.1: Generate semantic labels for each point through volume rendering; for each ray r, calculate the corresponding semantic label S(r):

[0027]

[0028] Among them, s i is the semantic category of each sampling point, T i is the cumulative transmittance, σ i is the density of sampling points, δ i is the distance between sampling points;

[0029] S1.5.2: Use the pre-trained semantic segmentation model to perform semantic segmentation on the multi-view RGB images in the scene and generate weak labels S weak ; This label is combined with the point cloud in the scene to generate preliminary semantic annotations:

[0030] S weak ={track, pedestrian, vehicle, obstacle}

[0031] S1.5.3: Use the multi-view geometric consistency of the data to correct the weak labels generated by the neural radiation field model; align the semantic labels generated by the neural radiation field with the geometric information of the multi-view RGB image, correct the label points with large errors, and obtain more accurate semantic labels S final ;

[0032] S1.5.4: The semantic label S of each point obtained in step S1.5.3 final With LiDAR point cloud datatotal Combined to form the final training data set; each point p i Corresponding to a semantic category label s i , forming a semantic point cloud dataset D semantic :

[0033] D semantic ={(p1,s1),(p2,s2),...,(p n ,s n )}

[0034] Where n is the number of points in the point cloud, s i For point p i The corresponding semantic category;

[0035] S1.5.5: Combine the generated extreme scenario point cloud data with the real point cloud data to construct a training dataset for autonomous driving.

[0036] Preferably, the specific process of step S02 is:

[0037] S2.1, import point cloud and select the point cloud of target area;

[0038] S2.2, locally triangulating the point cloud of the selected target area;

[0039] S2.3, identify holes and repair them;

[0040] S2.4, generating a point cloud on the triangular mesh within the patching area, and generating a new point cloud by sampling the barycentric coordinates;

[0041] S2.5. Merge the newly generated point cloud with the original point cloud P.

[0042] Preferably, the specific process of step S2.2 is:

[0043] S2.2.1: For the area P selected by the user s Perform local triangulation and use the Delaunay triangulation method to generate a triangular mesh, denoted as T = {Δ1, Δ2, ..., Δ m}, where each triangle consists of three points Δ i =(p a , p b , p c ). Delaunay triangulation is done by max(min(∠ abc , ∠ bca , ∠ cab ))Maximize the minimum interior angle of the triangle to avoid generating long and narrow triangles;

[0044] S2.2.2: For complex geometric areas, the Poisson surface reconstruction algorithm is used to generate a smooth triangular mesh T′ based on the solution of the Poisson equation:

[0045]

[0046] Where N(p) is the normal vector field of the point and p(p) is the density function of the point.

[0047] Preferably, the specific process of step S2.3 is:

[0048] S2.3.1: Identify the hole boundary point set B, and reconstruct the hole area geometrically based on the boundary points, and interpolate to generate new points p i , the interpolation formula is:

[0049] p i =αb1+βb2+(1-α-β)b3

[0050] Among them, α, β∈[0,1] are weight parameters in the barycentric coordinate system;

[0051] S2.3.2: Generate filled triangles using minimum area triangulation, optimizing triangle area A:

[0052]

[0053] S2.3.3: Perform Laplace smoothing on the repaired triangular mesh T′, and the smoothed point cloud coordinates p′ i for:

[0054]

[0055] Where N(i) is the point p i The neighborhood of , λ is the smoothing coefficient.

[0056] Preferably, the specific process of step S2.5 is:

[0057] Step 2.5.1: The newly generated point cloud P new Merge with the original point cloud p:

[0058] P final =P∪P new

[0059] Step 2.5.2: Use the KD-tree structure to organize the merged point cloud data and remove duplicate points;

[0060] Step 2.5.3: The user selects whether the hole filling is completed. If yes, the hole filling function is exited; otherwise, the process goes to step S2.1 to continue the hole filling.

[0061] Preferably, the specific process of step S03 is:

[0062] S3.1, perform data preprocessing on all point clouds;

[0063] S3.2, use the farthest point sampling algorithm to perform farthest point sampling on all point clouds to determine the sampling center point, and determine the neighborhood represented by each center point to construct an empty set S1;

[0064] S3.3, for each point in the L1 spherical local areas obtained in step S3.2, a directed neighborhood graph is constructed using the KNN algorithm, edge features of the edges in each directed graph are calculated using a multi-layer perceptron, and then the features of the sampling center points in the M1 directed graphs are obtained using a maximum pooling operation;

[0065] S3.4, use the farthest point sampling algorithm to perform farthest point sampling on all point clouds to determine the sampling center point, and determine the neighborhood represented by each center point to construct an empty set S2;

[0066] S3.5, for each point in the L2 spherical local areas obtained in step S3.3, a directed neighborhood graph is constructed using the KNN algorithm, edge features of the edges in each directed graph are calculated using a multi-layer perceptron, and then the features of the sampling center points in the M2 directed graphs are obtained using a maximum pooling operation;

[0067] S3.6, by recording the coordinates of each point, using the distance reverse interpolation algorithm to recover the features of the points lost by downsampling from the few points in the previous layer;

[0068] S3.7. By obtaining the category label of each point, after removing the background point cloud, the point cloud with the category label as the component is obtained, and then the point cloud of the component is clustered using a clustering algorithm to obtain the point cloud belonging to each component.

[0069] Preferably, the specific process of step S3.1 is:

[0070] S3.1.1: Use the remove_non_finite_points and remove_radius_outliers functions in open3d to remove invalid points in the point cloud;

[0071] S3.1.2: Based on step S3.1.1, use uniform_down_sample in open3d to uniformly downsample the point cloud, and select a multiple of 30;

[0072] S3.1.3: Based on step S3.1.2, the point cloud is uniformly downsampled using the uniform_down_sample function in open3d, with a multiple of 30;

[0073] S3.1.4: Based on step S3.1.3, the normal vector of the point cloud is calculated using the estimate_normals function in open3d, the dimension of the point cloud is expanded from 3 dimensions to 6 dimensions, and the normal vector of the point cloud is normalized.

[0074] Preferably, the specific process of step S3.2 is:

[0075] S3.2.1: Randomly select a point p r ∈{p1,p2,.....p n}, added to set S1;

[0076] S3.2.2: Calculate {p1,p2,.....p n The distance between each point in} and the set S1;

[0077] S3.2.3: According to the point with the largest value among the results calculated in step S3.2.2, add it to set S1;

[0078] S3.2.4: Determine whether the number of points in set S1 is greater than a threshold L1. If not, return to step S3.2.2 and repeat. Otherwise, proceed to the next step.

[0079] S3.2.5: Select the points in set S1 in the order of arrangement;

[0080] S3.2.6: Based on the point selected in step S3.2.5 as the center of the sphere, the number of points contained within the radius R1 is I1, M1 points are randomly selected from the I1 points to obtain a spherical local area containing M1 points and with a radius of R1, with the center of the sphere being the point selected in step S3.2.5;

[0081] S3.2.7: Determine whether the points in set S1 have been completely selected. If not, return to step S3.2.5 and repeat. Otherwise, L1 spherical local areas {B1, B2, .....B L}, the center of each spherical local area is a point in set S1, and the process goes to step S3.3.

[0082] Preferably, the specific process of step S3.3 is:

[0083] S3.3.1: For the L1 spherical regions B obtained in step S3.2.7 i , i∈[1,L1], select the points in order as the sampling center point, use the distance calculation formula in step S3.2.2 to calculate the distances from M1-1 other points to this point, and arrange them in ascending order according to the distance size;

[0084] S3.3.2: Take the first k1 points calculated in step S3.3.1 as the nearest neighboring points of the sampling center point, and construct a directed graph G = (V, E); V = {1, 2, ..., k1} represents the nodes, and the edges formed by the points are E = V × V;

[0085] S3.3.3: Determine whether the points in the spherical local area have been completely traversed. If not, repeat step S3.3.1. If they have been completely traversed, proceed to step S3.3.4.

[0086] S3.3.4: Select the central node p i , i∈[1,M1] and other points in its directed graph Directed edge ((i, j i1 ),.......(i,j k1 )), define the edge feature as e ij =h θ (p i , p j )=h θ (p i , p j -p i ), where the function h is a nonlinear function containing a series of learnable parameters θ;

[0087] S3.3.5: Select the central node p i and select other points in the directed graph in order Construct a directed edge between two points, use the three-dimensional coordinates, and use a multi-layer perceptron to calculate the edge feature vector of this edge. The output dimension of the MLP is C11; and perform this operation on all other points. Repeat the above to obtain k1 edge features;

[0088] S3.3.6: Input the obtained k1 edge features into the maximum pooling layer, and obtain the features of each point in the M1 directed graph as the central node, p′ i =max j：(i，j)∈e h θ (p i , p j ); concatenate the input domain M1 features of step S3.3 as the input of the next step;

[0089] S3.3.7: Jump to step S3.3.1, take the feature vector obtained in the previous step as input, repeat n times, and in each repeated operation, the output dimensions of the multilayer perceptron in step S3.3.5 are C12, C13, C14, C15, and C16 respectively. The final feature vector is extracted using a multilayer perceptron with an output dimension of C17, and then the feature vector is pooled through a pooling layer to obtain a feature vector of size L1×C17.

[0090] Preferably, the specific process of step S3.4 is:

[0091] S3.4.1: Randomly select a point pr∈{p1, p2, .....p L1}, added to set S2;

[0092] S3.4.2: Calculate {p1, p2, .....p L1 The distance from each point in} to set S2, where the definition of the distance from the point to the set is the same as step S3.2.2,

[0093] S3.4.3: According to the point with the largest value among the results calculated in step S3.4.2, add it to set S2;

[0094] S3.4.4: If the number of points in set S2 is greater than a certain threshold L2, then return to step S3.4.2 and repeat the execution; otherwise, proceed to the next step;

[0095] S3.4.5: Select the points in set S2 in the order of arrangement;

[0096] S3.4.6: Based on the point selected in step S3.4.5 as the center of the sphere, the number of points contained within the radius R2 is I2, and M2 points are randomly selected from the I2 points to obtain a spherical local area with a radius of R2 containing M2 points, and the center of the sphere is the point selected in step S3.4.5;

[0097] S3.4.7: Determine whether the points in set S2 have been completely selected. If not, return to step S3.4.5 and repeat. Otherwise, L2 spherical local areas {B1, B2, .....B L2}, the center of each spherical local area is a point in set S2, and the process goes to step S3.5.

[0098] Preferably, the specific process of step S3.5 is:

[0099] S3.5.1: For the L2 spherical regions B obtained in step S3.4.7 i, i∈[1, L2], select one of the points as the sampling center point in order, use the distance calculation formula in step S3.2.2 to calculate the distance from M2-1 other points to this point, and arrange them in ascending order according to the distance size;

[0100] S3.5.2: Take the first k2 points calculated in step S3.5.1 as the nearest neighboring points of the sampling center point, and construct a directed graph G = (V, E); V = {1, 2, ..., k2} represents the nodes, and the edges formed by the points are E = V × V;

[0101] S3.5.3: Determine whether the points in the spherical local area have been completely traversed. If not, repeat step S3.5.1. If they have been completely traversed, proceed to step S3.5.4.

[0102] S3.5.4: Select the central node p i , i∈[1,M2] and other points in its directed graph Directed edge ((i, j i1 ),.......(i,j k2 ), and the edge feature e ij =h θ (p i , p j )=h θ (p i , p j -p i ), the definition of function h is the same as in step S3.3.4;

[0103] S3.5.5: Select the central node p i and select other points in the directed graph in order Construct a directed edge between two points, use the three-dimensional coordinates, and use a multi-layer perceptron to calculate the edge feature vector of this edge. The output dimension of the MLP is C21; and this operation is applied to all other points Repeat the above to obtain k2 edge features;

[0104] S3.5.6: Input the obtained k1 edge features into the maximum pooling layer, and obtain M2 features of each point in this directed graph as the central node, p′ i =max j：(i，j)∈e h θ (p i , p j ); concatenate the input of step S3.5 with M2 features as the input of the next step;

[0105] S3.5.7: Jump to step S3.5.1 and use the feature vector obtained in the previous step as input. Repeat 5 times. In each repeated operation, the output dimensions of the multilayer perceptron in step S3.5.5 are C22, C23, C24, C25, and C26 respectively. Use the multilayer perceptron with an output dimension of C27 to extract features from the final feature vector, and then perform a pooling operation on the feature vector through a pooling layer to obtain a feature vector of size L2×C27.

[0106] Preferably, the specific process of step S3.6 is:

[0107] S3.6.1: The number of points obtained in step S3.4.1 is L2. The previous process is reversed to interpolate the number of point clouds from L2 to L1. The points to be interpolated are selected one by one. According to the coordinates of the previously recorded point clouds, the distance matrix d of each point in L1 from the points in L2 is calculated, which is L1×L2.

[0108] S3.6.2: Sort each row in ascending order, take out the k points with the smallest median value in each row, and then calculate the weighted The weight Where ε represents a 1×10 -8 The minimum amount of , and finally the feature vector of size L1×C27 is calculated;

[0109] S3.6.3: Concatenate the feature vector obtained in step S3.3.7 with the feature vector in step S3.6.2 to obtain a feature vector of size L1×(C27+C17); use a multi-layer perceptron with an output dimension of C3 to convert the feature vector into a feature vector of size L1×C3;

[0110] S3.6.4: Repeat steps S3.6.1 to S3.6.3, the number of interpolation points is from L1 to n, the dimension of the MLP output is C4, and the feature vector of size n×(C3+C) is obtained;

[0111] S3.6.5: Input the feature vector obtained in step S3.6.4 into the fully connected layer. The dimension of the output of the fully connected layer is C4. The fully connected layer uses softmax to limit the final result to 0 to 1, and finally obtains the probability of each point belonging to each category. Convert the probability vector of each point into a one-hot encoding, and convert the one-hot encoding into a decimal category label. Finally, obtain the label of the category to which each point belongs, and proceed to step S3.7.

[0112] Preferably, the specific process of step S3.7 is:

[0113] S3.7.1: Remove the point clouds belonging to the background points from the obtained point clouds, and retain the point clouds belonging to the parts;

[0114] S3.7.2: Cluster the remaining point clouds using the coordinates using the dbscan algorithm in open3d, with the algorithm parameters eps set to 2 and min_points set to 5, to obtain the point clouds of components belonging to different clustering areas;

[0115] S3.7.3: Use the get_axis_aligned_bounding_box function for the point clouds of components in different areas to obtain the bounding box of each component area.

[0116] The present invention also discloses a computer program product, comprising a computer program, wherein the computer program executes the steps of the method described above when executed by a processor.

[0117] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0118] The present invention also discloses a 3D point cloud target detection system for autonomous driving, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0119] Compared with the prior art, the advantages of the present invention are:

[0120] The present invention uses a multi-view camera to capture scenes, and combined with the camera posture information, uses NeRF technology to render multi-view images to generate data samples of extreme scenes; on the basis of conventional scenes, rare dynamic objects (such as pedestrians or obstacles) are inserted to generate extreme training data. This method uses the high-precision rendering capability of NeRF to enhance the perception ability of the autonomous driving system in extreme scenes, and can generate high-quality, multi-view extreme scene image samples, effectively improving the safety and robustness of the autonomous driving system.

[0121] The present invention reduces errors caused by occlusion, data loss and other problems by repairing holes selected by users, ensures the integrity of point cloud data, and improves the accuracy of subsequent target detection and analysis tasks.

[0122] The object detection and positioning algorithm based on LDCNN point cloud of the present invention improves the ability to mine multiple local information of the network by using a multi-cascaded edge convolution self-network, and realizes part detection based on point cloud. The present invention uses the proposed multi-stage connected edge sub-convolution structure to replace the LDCNN network of PointNet in the PointNet++ network to extract structural information in the point cloud, improves the ability to mine multiple local information of the network, uses segmentation methods instead of target detection methods to fit more complex point cloud scenes, and automatically completes the labeling of parts, which greatly improves the efficiency compared to manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] Figure 1 It is a flow chart of the 3D point cloud target detection method in an embodiment of the present invention.

[0124] Figure 2 This is the LDCNN network structure diagram in the present invention.

[0125] Figure 3 This is a structural diagram of the multi-cascade edge convolutional network structure and the convolution module in the present invention. DETAILED DESCRIPTION

[0126] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0127] like Figure 1 As shown, the 3D point cloud target detection method for autonomous driving according to an embodiment of the present invention comprises the following steps:

[0128] Step 1: Obtain multi-view point cloud data in the autonomous driving scenario, use the neural radiation field to reconstruct the 3D geometric information and color information of the scene, generate extreme scene point cloud data, and then combine the generated extreme scene point cloud data with the real point cloud data to construct a training data set for autonomous driving;

[0129] Step 2: Convert the point cloud data into a grid, detect holes in the point cloud data and perform repair operations on them to generate a point cloud with repaired holes, and then merge it with the original point cloud data to form a complete three-dimensional point cloud;

[0130] Step 3: After preprocessing the 3D point cloud, input it into a pre-trained local dynamic convolutional neural network, output the point cloud segmentation result, remove the background points and cluster the target point cloud to obtain the point cloud of the inspection area, so as to realize the recognition and positioning of the parts.

[0131] Specifically, in step 1, the generation of extreme scene data for autonomous driving refers to using neural radiation field technology to collect image data containing the environment with a multi-view camera and obtain the corresponding camera posture information. Specifically, the multi-view image and camera posture information are used as input, and a light ray is randomly selected at each viewpoint using volume rendering technology, and several points are randomly sampled along the light ray to obtain a position set of multiple sampling points {X i} and the corresponding viewing angle {V i}; By inputting the position and viewing angle of each sampling point, predicting the color c and volume density σ of each sampling point, and calculating the loss between the color of each light and the real image color based on the predicted value, the parameters of the neural network are optimized; on the basis of conventional scenes, rare dynamic objects (such as pedestrians, obstacles, etc.) are inserted into the three-dimensional scene, which are regarded as new training data to generate new extreme scene data samples; by synthesizing extreme scene data, the light color and object surface consistency loss are calculated using the volume rendering method to minimize the color loss L color and image consistency cost L photo , further optimize the neural radiation field network model; use the optimized neural radiation field model to render extreme scenes from different perspectives, generate extreme scene images from multiple perspectives, and use them to train the autonomous driving system.

[0132] The present invention uses a multi-view camera to capture scenes, and combined with the camera posture information, uses NeRF technology to render multi-view images to generate data samples of extreme scenes; on the basis of conventional scenes, rare dynamic objects (such as pedestrians or obstacles) are inserted to generate extreme training data. This method uses the high-precision rendering capability of NeRF to enhance the perception ability of the autonomous driving system in extreme scenes, and can generate high-quality, multi-view extreme scene image samples, effectively improving the safety and robustness of the autonomous driving system.

[0133] In terms of generating negative samples from point cloud data, it is difficult to collect some negative samples due to security and other reasons. The data-based sample generation technology separates the background and foreground targets in the point cloud through the neural radiation field, and then re-renders to generate lidar point cloud data for certain extreme scenes, thereby making up for the shortcoming of the difficulty in collecting negative sample data.

[0134] The data-based point cloud sample generation algorithm uses neural radiation field technology to directly render highly realistic lidar point cloud data, which greatly reduces the sample collection cost and greatly increases the proportion of negative sample data. Autonomous driving scenarios such as pedestrians crossing the road are unlikely to occur in reality, so it is difficult to collect enough samples, which is a major defect for autonomous driving algorithms that require a large number of diverse samples for training. The present invention extracts information based on existing conventional data and reconstructs it using neural radiation field technology to efficiently render high-quality virtual images and point cloud data, thereby simulating rare and challenging driving scenes, and directly rendering highly realistic autonomous driving point cloud data, which greatly reduces the sample collection cost and greatly increases the proportion of extreme sample data.

[0135] Specifically, in step 2, due to equipment occlusion, light reflection, complex geographical environment and equipment accuracy limitations, the point cloud data collected by the structured light camera often has incomplete parts, forming holes. The existence of these holes will cause the point cloud model to be discontinuous, affecting the subsequent three-dimensional reconstruction, component detection and analysis tasks. Due to the complex structure of the point cloud, the direct use of the hole filling algorithm has limited effect. The present invention uses interactive tools to enable users to independently select the area to be repaired, and locally triangulate the selected area based on the triangulation algorithm (Delaunay triangulation); the minimum dihedral angle and minimum area optimization algorithm are used for geometric reconstruction, and finally the repaired triangular mesh is converted into point cloud data to ensure the geometric consistency of the repaired area with the surrounding point cloud. This method enhances the flexibility and accuracy of point cloud repair, and is particularly suitable for irregular hole repair. By repairing the holes selected by the user, the errors caused by occlusion, data loss and other problems are reduced, the integrity of the point cloud data is ensured, and the accuracy of subsequent target detection and analysis tasks is improved.

[0136] Specifically, the specific process of step 2 is:

[0137] Collect point cloud data to obtain a three-dimensional point cloud model with holes;

[0138] Providing user interaction tools to enable users to independently select the area they wish to repair to form a target repair area;

[0139] Perform local triangulation on the selected patching area to generate an initial triangulated mesh. The triangulation method includes Delaunay triangulation or Poisson surface reconstruction.

[0140] Identify holes in the triangular mesh and extract hole boundary points;

[0141] The hole area is geometrically reconstructed and the filled triangle mesh is generated using the minimum dihedral angle and minimum area optimization algorithm;

[0142] The triangle mesh of the repair area is smoothed, and the Laplace smoothing method is used to ensure the geometric consistency between the repair area and the surrounding area;

[0143] The repaired triangular mesh is converted back into point cloud data, and the barycentric coordinate sampling method is used to generate a point cloud of equal density;

[0144] The newly generated point cloud is merged with the original point cloud to form a complete 3D point cloud model.

[0145] The present invention converts the point cloud into a grid by human-computer interaction, detects the holes therein and performs operations such as filling, smoothing, sampling, etc., and adjusts the repair effect based on the human-computer interaction to generate a point cloud after the holes are repaired. The area to be repaired is selected according to user needs. After the hole repair algorithm completes the repair according to the user's selection, the generated point cloud is merged into the original point cloud, thereby ensuring the integrity of the point cloud data and improving the accuracy of subsequent target detection and analysis tasks.

[0146] In step 3, for the detection of point cloud components, a local dynamic convolutional neural network (LDCNN) is introduced to generate pre-processed data after outlier removal, downsampling, and normal vector calculation. The pre-processed data is input into the trained LDCNN to output the point cloud segmentation result. The background points are removed and the target point cloud is clustered to obtain the point cloud of the inspection area, thereby realizing the recognition and positioning of the components.

[0147] The detection and positioning of component point clouds refers to the use of deep learning methods to segment the input point clouds and locate them based on the results. First, the point clouds are preprocessed: outliers and invalid points are removed, uniform downsampling, normal vector calculation and other operations are performed on each point cloud to obtain a point cloud with low noise level, sparse density and more feature dimensions. Then, the point cloud is input into the LDCNN network, and the result of point cloud segmentation is input after sampling grouping, feature extraction, interpolation and other operations. Finally, the background points of the segmented point cloud are removed, and the target point cloud is clustered to obtain the target detection result of the component.

[0148] The target detection and positioning algorithm based on the point cloud of the LDCNN of the present invention improves the ability of the network to mine multiple local information by using a multi-cascaded edge convolution self-network, and realizes point cloud-based component detection. The point cloud-based component anomaly detection algorithm requires prior information on the location of the component, so it is necessary to pre-position the component through manual annotation, which requires the project to spend a lot of manpower and material resources to pre-annotate the data, greatly weakening the flexibility of the project. The present invention uses the proposed multi-level connected edge sub-convolution structure to replace the LDCNN network of PointNet in the PointNet++ network to extract structural information from the point cloud, improves the ability of the network to mine multiple local information, and uses segmentation methods instead of target detection methods to fit more complex point cloud scenes. Automatically complete the labeling of components, which greatly improves efficiency compared to manual labeling.

[0149] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0150] S01. Point cloud generation: Obtain data from the camera or lidar and process it, specifically:

[0151] S1.1: Collect and preprocess multi-view data in the scene.

[0152] Specifically, multi-view RGB images and sparse LiDAR point cloud data are collected from the scene, and the multi-view image I = {I1, I2, …, I n} and its corresponding camera perspective V = {v1,v2,…,v n}Preprocessing;

[0153] S1.2: Reconstruct the 3D geometry and color information of the scene using neural radiance fields.

[0154] S1.2.1: Taking the preprocessed multi-view images and camera poses as input, the neural radiance field reconstructs the 3D geometric information of the scene through volume rendering. The model learns a mapping function f:(x,v)→(c,σ), where x is the coordinate in the scene, v is the camera view, c is the color, and σ is the density;

[0155] S1.2.2: The volume rendering formula is as follows:

[0156]

[0157] in, is the cumulative transmittance, σ i is the density, δ i is the distance between sampling points, c i is the color value.

[0158] S1.3: Generate depth map and calculate 3D point cloud position.

[0159] S1.3.1: The depth map D(r) of each ray is generated using volume rendering of the neural radiance field, which is calculated as:

[0160]

[0161] Among them, z i is the depth of the sampling point;

[0162] S1.3.2: Calculate the position of each 3D point p in the point cloud based on the ray starting point o and direction d:

[0163] p=o+D(r)·d

[0164] Among them, d = (cosθcosφ, sinθsinφ, cosφ) is the direction of the light, and θ and φ are the azimuth and elevation angles respectively.

[0165] S1.4: Generate LiDAR point cloud data

[0166] S1.4.1: Based on the 3D geometric information generated in step S1.3, add extreme scene dynamic objects, such as pedestrians crossing the track.

[0167] The motion of dynamic objects (such as pedestrians) is modeled and their motion trajectory M(t) is set, where t is the time parameter. The pedestrian's moving path M(t) = {x(t), y(t), z(t)} is simulated through a known pedestrian gait model or other extreme behavior model, and the path is synthesized with the 3D reconstruction data of the track scene.

[0168] S1.4.2: The simulation of the LiDAR sensor includes setting the range of light emission angles and resolution. For example, setting the sensor's rotation speed to ω and the field of view angle to [θ min ,θ max ], with a horizontal resolution of Δθ and a vertical resolution of Δφ. For each frame of LiDAR data, the 3D geometric information reconstructed by the neural radiation field is used to generate the depth value D(r) and color information c(r) corresponding to each ray r.

[0169] S1.4.3: According to the parameters of the LiDAR sensor, set the starting point o and direction d of the virtual light, and for each light ray r, calculate the position p of the 3D point according to the depth value D(r). The generated point cloud can be represented by the symbol P = {p1,p2,…,p n}, where n is the number of points in each frame of LiDAR point cloud.

[0170] S1.4.4: Use the multi-frame point cloud data generated in step S1.4.3 to synthesize the complete point cloud in the scene. i , we have the corresponding point cloud data P i Through the time step Δt, multiple frame point clouds at different time points are synthesized into a dynamic scene point cloud P total :

[0171]

[0172] Where N is the number of frames in the time step.

[0173] S1.5: Calculate semantic labels and generate final training data

[0174] S1.5.1: Generate semantic labels for each point through volume rendering. The calculation process of semantic labels is similar to volume rendering of color and depth. For each ray r, calculate the corresponding semantic label S(r):

[0175]

[0176] Among them, s i is the semantic category of each sampling point, T i is the cumulative transmittance, σ i is the density of sampling points, δ i is the distance between sampling points.

[0177] S1.5.2: Use a pre-trained semantic segmentation model (e.g., SegFormer) to perform semantic segmentation on the multi-view RGB images in the scene and generate weak labels S weak This tag is combined with the point cloud in the scene to generate preliminary semantic annotations:

[0178] S weak ={track, pedestrian, vehicle, obstacle}

[0179] S1.5.3: Use the multi-view geometric consistency of the data to correct the weak labels generated by the neural radiation field model. Align the semantic labels generated by the neural radiation field with the geometric information of the multi-view RGB image, correct the label points with large errors, and obtain a more accurate semantic label S final .

[0180] S1.5.4: The semantic label S of each point obtained in step S1.5.3 fina1 With LiDAR point cloud data total Combined to form the final training data set. Each point p i Corresponding to a semantic category label s i , forming a semantic point cloud dataset D semantic :

[0181] D semantic ={(p1,s1),(p2,s2),...,(p n ,s n )}

[0182] Where n is the number of points in the point cloud, s i For point p i The corresponding semantic category.

[0183] S1.5.5: Combine the generated extreme scenario point cloud data with the real point cloud data to construct a training dataset for autonomous driving.

[0184] S02. Point cloud hole filling: After the point cloud is generated, there are holes or incomplete areas in the point cloud data. These holes are filled through the point cloud hole filling step, and the missing data is estimated through methods such as interpolation or surface reconstruction; specifically:

[0185] S2.1: Import point cloud, user area selection.

[0186] S2.1.1: Import the point cloud into the 3D preview tool. Through the interactive tool, the user can select the point cloud of the target area by clicking or selecting a box. Where P s It is the point cloud area that needs to be repaired, selected by the user;

[0187] S2.1.2: The system marks the hole boundaries of the user-selected area, and the boundary point set is denoted as B = {b1, b2, ..., b k}, where b i ∈P s .

[0188] S2.2: Regional triangulation:

[0189] S2.2.1: For the area P selected by the user s Perform local triangulation and use the Delaunay triangulation method to generate a triangular mesh, denoted as T = {Δ1, Δ2, ..., Δ m}, where each triangle consists of three points Δ i =(p a , p b , p c ). Delaunay triangulation is done by max(min(∠ abc , ∠ bca , ∠ cab ))Maximize the minimum interior angle of a triangle to avoid generating long, narrow triangles.

[0190] S2.2.2: For complex geometric areas, the Poisson surface reconstruction algorithm is used to generate a smooth triangular mesh T′ based on the solution of the Poisson equation:

[0191]

[0192] Where N(p) is the normal vector field of the point and ρ(p) is the density function of the point.

[0193] S2.3: Hole Detection and Repair

[0194] S2.3.1: Identify the hole boundary point set B, and reconstruct the hole area geometrically based on the boundary points, and interpolate to generate new points p i , the interpolation formula is:

[0195] p i =αb1+βb2+(1-α-β)b3

[0196] Where α, β∈[0, 1] are weight parameters in the barycentric coordinate system.

[0197] S2.3.2: Generate filled triangles using minimum area triangulation, optimizing triangle area A:

[0198]

[0199] S2.3.3: Perform Laplace smoothing on the repaired triangular mesh T′, and the smoothed point cloud coordinates p′ i for:

[0200]

[0201] Where N(i) is the point p i The neighborhood of , λ is the smoothing coefficient.

[0202] S2.4: Convert triangular mesh to point cloud

[0203] Generate a point cloud on the triangular mesh within the patch area, and generate a new point cloud set P by sampling the barycentric coordinates. new :

[0204] p new =αp a +βp b +(1-α-β)p c

[0205] Where α, β∈[0, 1] are the sampling weights in the barycentric coordinate system.

[0206] S2.5: Point cloud merging

[0207] S2.5.1: The newly generated point cloud P newMerge with the original point cloud P:

[0208] P final =P∪P new

[0209] S2.5.2: Use KD-tre e The structure organizes the merged point cloud data and removes duplicate points.

[0210] S2.5.3: The user selects whether the hole filling is completed. If yes, the hole filling function is exited; otherwise, the process goes to step S2.1 to continue the hole filling.

[0211] S03, Point cloud object detection: Identify the part of the point cloud that represents the object, specifically:

[0212] S3.1: Perform data preprocessing on all point clouds.

[0213] S3.1.1: Use the remove_non_finite_points and remove-radius_outliers functions in open3d to remove invalid points in the point cloud;

[0214] S3.1.2: Based on step S3.1.1, use uniform_down_sample in open3d to uniformly downsample the point cloud, and select a multiple of 30;

[0215] S3.1.3: Based on step S3.1.2, the point cloud is uniformly downsampled using the uniform_down_sample function in open3d, with a multiple of 30;

[0216] S3.1.4: Based on step S3.1.3, the normal vector of the point cloud is calculated using the estimate_normals function in open3d, the dimension of the point cloud is expanded from 3 dimensions to 6 dimensions, and the normal vector of the point cloud is normalized;

[0217] S3.2: Point cloud is represented by {p1,p2,.....p n} represents, where p represents each point in the point cloud, n represents the number of points in each point cloud, and the farthest point sampling algorithm is used to perform farthest point sampling on all point clouds to determine the sampling center point, and determine the neighborhood represented by each center point to construct an empty set S1;

[0218] S3.2.1: Randomly select a point p r ∈{p1,p2,.....p n}, added to set S1;

[0219] S3.2.2: Calculate {p1,p2,.....p nThe distance between each point in} and the set S1 is defined as the smallest distance between each point in the set. The distance between two points is calculated as:

[0220] where pi x , pi y , pi z They correspond to the three-dimensional coordinates of the representative points respectively;

[0221] S3.2.3: According to the point with the largest value among the results calculated in step S3.2.2, add it to set S1;

[0222] S3.2.4: If the number of points in S1 is greater than a certain threshold L1, then return to step S3.2.2 and repeat the execution, otherwise, proceed to the next step;

[0223] S3.2.5: Select the points in set S1 in the order of arrangement;

[0224] S3.2.6: Based on the point selected in step S3.2.5 as the center of the sphere, the number of points contained in the radius R1 is I1, and M1 points are randomly selected from the I1 points to obtain a spherical local area with a radius of R1 containing M1 points, and the center of the sphere is the point selected in step S3.2.5.

[0225] S3.2.7: Determine whether the points in set S1 have been completely selected. If not, return to step S3.2.5 and repeat. Otherwise, L1 spherical local areas {B1, B2, .....B L}, the center of each spherical local area is a point in set S1, and the process goes to step S3.3, where the parameters L1, M1, and R1 in step S3.2 are set to 512, 32, and 0.2.

[0226] S3.3: For each point in the L1 spherical local areas obtained in step S3.2, a directed neighborhood graph is constructed using the KNN algorithm. The edge features of each directed graph are calculated using a multi-layer perceptron, and then the maximum pooling operation is used to obtain the features of the sampling center points in the M1 directed graphs.

[0227] S3.3.1: For the L1 spherical regions B obtained in step S3.2.7 i , i∈[1,L1], select the points in order as the sampling center point, use the distance calculation formula in step S3.2.2 to calculate the distances from M1-1 other points to this point, and arrange them in ascending order according to the distance size.

[0228] S3.3.2: Take the first k1 points calculated in step S3.3.1 as the nearest neighboring points of the sampling center point, and construct a directed graph G = (V, E); V = {1, 2, ..., k1} represents the nodes, and the edges formed by the points are E = V × V;

[0229] S3.3.3: Determine whether the points in the spherical local area have been completely traversed. If not, repeat step S3.3.1. If they have been completely traversed, proceed to step S3.3.4.

[0230] S3.3.4: Select the central node p i , i∈[1,M1] and other points in its directed graph Directed edge ((i, j i1 ),.......(i,j k1 )), define the edge feature as e ij =h θ (p i , p j )=h θ (p i , p j -p i ), where the function h is a nonlinear function containing a series of learnable parameters θ;

[0231] S3.3.5: Select the central node p i and select other points in the directed graph in order Construct a directed edge between two points, use their 3D coordinates, and use a multi-layer perceptron to calculate the edge feature vector of this edge. The output dimension of the MLP is C11; and do this for all other points. Repeat the above process to obtain k1 edge features.

[0232] S3.3.6: Input the obtained k1 edge features into the maximum pooling layer, and obtain M1 features of each point in this directed graph as the central node, p′ i =max j：(i，j)∈e h θ (p i , p j ); The M1 features of the input domain of S3.3 are concatenated as the input of the next step.

[0233] S3.3.7: Jump to step S3.3.1, use the feature vector obtained in the previous step as input, repeat 5 times, and in each repetition, the output dimensions of the multilayer perceptron in step S3.3.5 are C12, C13, C14, C15, and C16 respectively. The final feature vector is extracted using a multilayer perceptron with an output dimension of C17, and then the feature vector is pooled through a pooling layer to obtain a feature vector of size L1×C17. The parameters k1, C11, C12, C13, C14, C15, C16, and C17 in step S3.3 are set to 128, 64, 64, 64, 64, 64, 128, and 256.

[0234] S3.4: The dimension of the input point cloud is L1×C17, represented by the symbol {p1, p2, .....p L1} represents, where p represents each point in the point cloud, L1 represents the number of points in each point cloud, and the farthest point sampling algorithm is used to perform farthest point sampling on all point clouds to determine the sampling center point, and determine the neighborhood represented by each center point to construct an empty set S2.

[0235] S3.4.1: Randomly select a point pr∈{p1, P2, ....P L1}, added to set S2;

[0236] S3.4.2: Calculate {p1, p2, .....p L1 The distance from each point in} to set S2, where the definition of the distance from the point to the set is the same as step S3.2.2,

[0237] S3.4.3: According to the point with the largest value among the results calculated in step S3.4.2, add it to set S2;

[0238] S3.4.4: If the number of points in S2 is greater than a certain threshold L2, then return to step S3.4.2 and repeat the execution, otherwise, proceed to the next step;

[0239] S3.4.5: Select the points in set S2 in the order of arrangement;

[0240] S3.4.6: Based on the point selected in step S3.4.5 as the center of the sphere, the number of points contained in the radius R2 is 12, and M2 points are randomly selected from the 12 points to obtain a spherical local area with a radius of R2 containing M2 points, and the center of the sphere is the point selected in step S3.4.5.

[0241] S3.4.7: Determine whether the points in set S2 have been completely selected. If not, return to step S3.4.5 and repeat. Otherwise, L2 spherical local areas {B1, B2, .....B L2}, the center of each spherical local area is a point in set S2, and the process goes to step S3.5. The parameters L2, M2, and R2 in step S3.4 are set to 128, 64, and 0.4.

[0242] S3.5: For each point in the L2 spherical local areas obtained in step S3.3, a directed neighborhood graph is constructed using the KNN algorithm. The edge features of each directed graph are calculated using a multi-layer perceptron, and then the maximum pooling operation is used to obtain the features of the sampling center points in the M2 directed graphs.

[0243] S3.5.1: For the L2 spherical regions B obtained in step S3.4.7 i , i∈[1, L2], select the points in order as the sampling center point, use the distance calculation formula in step S3.2.2 to calculate the distances from M2-1 other points to this point, and arrange them in ascending order according to the distance size.

[0244] S3.5.2: Take the first k2 points calculated in step S3.5.1 as the nearest neighboring points of the sampling center point, and construct a directed graph G = (V, E);

[0245] V = {1, 2, ..., k2} represents the nodes, and the edge formed by the nodes is E = V × V;

[0246] S3.5.3: Determine whether the points in the spherical local area have been completely traversed. If not, repeat step S3.5.1. If they have been completely traversed, proceed to step S3.5.4.

[0247] S3.5.4: Select the central node p i , i∈[1,M2] and other points in its directed graph Directed edge ((i, j i1 ),.......(i,j k2 ), and the edge feature e ij =h θ (p i , p j )=h θ (p i , p j -p i ), the definition of function h is the same as in step S3.3.4.

[0248] S3.5.5: Select the central node p i and select other points in the directed graph in order Construct a directed edge between two points, use their 3D coordinates, and use a multi-layer perceptron to calculate the edge feature vector of this edge. The output dimension of the MLP is C21. Repeat this operation on all other points. Repeat the above process to obtain k2 edge features.

[0249] S3.5.6: Input the obtained k1 edge features into the maximum pooling layer, and obtain M2 features of each point in this directed graph as the central node, p′ i =max j：(i，j)∈e h θ (p i , p j ). The input of step S3.5 is concatenated with the M2 features as the input of the next step.

[0250] S3.5.7: Jump to step S3.5.1, use the feature vector obtained in the previous step as input, and repeat 5 times. In each repetition, the output dimensions of the multilayer perceptron in step S3.5.5 are C22, C23, C24, C25, and C26 respectively. The final feature vector is extracted using a multilayer perceptron with an output dimension of C27, and then the feature vector is pooled through a pooling layer to obtain a feature vector of size L2×C27, and then enter step S3.6. The parameters k2, C21, C22, C23, C24, C25, C26, and C27 are set to 64, 64, 64, 64, 64, 128, and 512;

[0251] S3.6: By recording the coordinates of each point, the distance reverse interpolation algorithm is used to recover the features of the points lost after downsampling from the few points in the previous layer.

[0252] S3.6.1: The number of points obtained in step S3.4.1 is L2. The previous process should be reversed to interpolate the number of point clouds from L2 to L1. The points to be interpolated are selected in turn. According to the coordinates of the previously recorded point cloud, the distance matrix d of size L1×L2 from each point in L1 to the L2 points is calculated.

[0253] S3.6.2: Sort each row in ascending order, take out the k points with the smallest median value in each row, and then calculate the weighted The weight Where ε represents a 1×10 -8 The minimum amount of , and finally the feature vector of size L1×C27 is calculated.

[0254] S3.6.3: Concatenate the feature vector obtained in step S3.3.7 with the feature vector in step S3.6.2 to obtain a feature vector of size L1×(C27+C17). Use a multi-layer perceptron with an output dimension of C3 to convert the feature vector to a feature vector of size L1×C3.

[0255] S3.6.4: Repeat steps S3.6.1 to S3.6.3. The number of interpolation points is from L1 to n. The dimension of the MLP output is C4, and a feature vector of size n×(C3+C) is obtained.

[0256] S3.6.5: Input the feature vector obtained in step S3.6.4 into the fully connected layer. The dimension of the output of the fully connected layer is C4. The fully connected layer uses softmax to limit the final result to 0 to 1, and finally obtains the probability of each point belonging to each category. Convert the probability vector of each point to a one-hot encoding, and convert the one-hot encoding to a decimal category label, and finally obtain the label of each point belonging to the category, and enter step S3.7. The parameters k, t, C3, and C4 in step S3.6 are set to 3, 2, 256, and 2. The structure of the entire network is as follows: Figure 2 As shown in Figure 2, the multi-cascade edge convolution structure is as follows: Figure 3 shown.

[0257] S3.7: After obtaining the category label of each point, the background point cloud is removed to obtain the point cloud with the category label as the component, and then the point cloud of the component is clustered using a clustering algorithm to obtain the point cloud belonging to each component.

[0258] S3.7.1: From the obtained point cloud, remove the point cloud that belongs to the background point category and retain the point cloud that belongs to the component category.

[0259] S3.7.2: The remaining point clouds are clustered using the coordinates using the dbscan algorithm in open3d. The algorithm parameters eps are set to 2 and min_points is set to 5 to obtain the component point clouds belonging to different clustering areas.

[0260] S3.7.3: Use the get_axis_aligned_bounding_ox function on the point clouds of components in different areas to get the bounding box of each component area.

[0261] The method and system for hole repair, component detection, and data generation based on point cloud of the present invention completes point cloud hole repair by edge detection, filling, smoothing, sampling and other operations through human-computer interaction, merges the repaired point cloud into the original point cloud, and completes the segmentation of the point cloud through a local dynamic convolutional neural network to detect the target. In the autonomous driving scene, information is extracted based on existing conventional data and reconstructed using neural radiation field technology to efficiently render high-quality virtual images and point cloud data, thereby simulating rare and challenging driving scenes; this method not only enriches the training data set, but also helps to improve the autonomous driving system's ability to respond to potential dangers, thereby improving its overall safety and reliability.

[0262] The present invention also discloses a computer program product, including a computer program, which executes the steps of the method described above when executed by a processor. The present invention further discloses a computer-readable storage medium, on which a computer program is stored, which executes the steps of the method described above when executed by a processor. The present invention also discloses a 3D point cloud target detection system for autonomous driving, including an interconnected memory and a processor, on which a computer program is stored, which executes the steps of the method described above when executed by a processor. The product, medium and system of the present invention correspond to the above method and also have the advantages described in the above method.

[0263] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules, and the processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0264] Glossary:

[0265] Point cloud: A point cloud is a data set of points in space that can represent a three-dimensional shape or object, usually acquired by a 3D scanner.

[0266] Delaunay triangulation: It is a triangulation method for point clouds.

[0267] Laplace smoothing method: also known as Add-one Smoothing, is a technique used to deal with zero probability problems in statistical modeling.

[0268] Centroid sampling: A clustering-based sampling method that finds the cluster center of the majority class samples and then deletes some of the majority class samples closest to the cluster center to achieve sampling balance.

[0269] Normal vector: It is a concept in spatial analytic geometry. The vector represented by a straight line perpendicular to a plane is the normal vector of the plane.

[0270] Convolutional Neural Networks: Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0271] Neural Radiance Field: Neural Radiance Field (NeRF) is a neural network that can reconstruct complex three-dimensional scenes from a partial set of two-dimensional images.

[0272] Lidar: LiDAR (Light Laser Detection and Ranging) is the abbreviation of laser detection and ranging system.

[0273] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A 3D point cloud target detection method for autonomous driving, characterized in that: Includes steps: S01. Obtain multi-view point cloud data in an autonomous driving scenario, reconstruct the 3D geometric information and color information of the scene using the neural radiation field, generate extreme scene point cloud data, and then combine the generated extreme scene point cloud data with the real point cloud data to construct a training data set for autonomous driving; S02, converting the point cloud data into a grid, detecting holes in the point cloud data and performing repair operations on the holes to generate a point cloud after the holes are repaired, and then merging the point cloud data with the original point cloud data to form a complete three-dimensional point cloud; S03. After preprocessing the three-dimensional point cloud, it is input into a pre-trained local dynamic convolutional neural network, and the result of point cloud segmentation is output. The background points are removed and the target point cloud is clustered to obtain the point cloud of the inspection area, thereby realizing the recognition and positioning of the components.

2. The 3D point cloud target detection method for autonomous driving according to claim 1, characterized in that: The specific process of step S01 is: S1.

1. Obtain multi-view point cloud data in the autonomous driving scenario; S1.

2. Reconstruct the 3D geometry and color information of the scene using neural radiation fields based on multi-view point cloud data. S1.3, generate a depth map of each ray and calculate the 3D point cloud position using volume rendering of the neural radiance field; S1.4, generating LiDAR point cloud data based on 3D point cloud information; S1.

5. Calculate semantic labels and generate final training data.

3. The 3D point cloud target detection method for autonomous driving according to claim 2, characterized in that: The specific process of step S1.4 is as follows: S1.4.1: Based on the 3D geometric information generated in step S1.3, add extreme scene dynamic objects; specifically, model the motion of the dynamic objects and set their motion trajectory M(t), where t is a time parameter; simulate the pedestrian's moving path M(t) = {x(t), y(t), z(t)} through a known pedestrian gait model, and synthesize the path with the 3D reconstruction data of the track scene; S1.4.2: For each frame of LiDAR data, use the 3D geometry information reconstructed by the neural radiance field to generate the depth value D(r) and color information C(r) corresponding to each ray r; S1.4.3: According to the parameters of the LiDAR sensor, set the starting point o and direction d of the virtual light. For each light ray r, calculate the position p of the 3D point according to the depth value D(r); the generated point cloud is represented by the symbol P = {p1, p2, ..., p n }, where n is the number of points in each frame of LiDAR point cloud; S1.4.4: Synthesize the multi-frame point cloud data generated in step S1.4.3 into a complete point cloud in the scene; for each frame f i , there is corresponding point cloud data P i ; Through the time step Δt, the multi-frame point clouds at different time points are synthesized into the dynamic scene point cloud P total : Where N is the number of frames in the time step.

4. The 3D point cloud target detection method for autonomous driving according to claim 3, characterized in that: The specific process of step S1.5 is: S1.5.1: Generate semantic labels for each point through volume rendering; for each ray r, calculate the corresponding semantic label S(r): Among them, s i is the semantic category of each sampling point, T i is the cumulative transmittance, σ i is the density of sampling points, δ i is the distance between sampling points; S1.5.2: Use the pre-trained semantic segmentation model to perform semantic segmentation on the multi-view RGB images in the scene and generate weak labels S weak ; This label is combined with the point cloud in the scene to generate preliminary semantic annotations: S weak ={track, pedestrian, vehicle, obstacle} S1.5.3: Use the multi-view geometric consistency of the data to correct the weak labels generated by the neural radiation field model; align the semantic labels generated by the neural radiation field with the geometric information of the multi-view RGB image, correct the label points with large errors, and obtain more accurate semantic labels S final ; S1.5.4: The semantic label S of each point obtained in step S1.5.3 final With LiDAR point cloud data total Combined to form the final training data set; each point p i Corresponding to a semantic category label s i , forming a semantic point cloud dataset D semantic : D semantic ={(p1,s1),(p2,s2),…,(p n ,s n )} Where n is the number of points in the point cloud, s i For point p i The corresponding semantic category; S1.5.5: Combine the generated extreme scenario point cloud data with the real point cloud data to construct a training dataset for autonomous driving.

5. The 3D point cloud target detection method for autonomous driving according to any one of claims 1 to 4, characterized in that: The specific process of step S02 is: S2.1, import point cloud and select the point cloud of target area; S2.2, locally triangulating the point cloud of the selected target area; S2.3, identify holes and repair them; S2.4, generating a point cloud on the triangular mesh within the patching area, and generating a new point cloud by sampling the barycentric coordinates; S2.

5. Merge the newly generated point cloud with the original point cloud P.

6. The 3D point cloud target detection method for autonomous driving according to claim 5, characterized in that: The specific process of step S2.2 is: S2.2.1: For the area P selected by the user s Perform local triangulation and use the Delaunay triangulation method to generate a triangular mesh, denoted as T = {△1,△2,...,△ m }, where each triangle consists of three points △ i =(p a ,p b ,p c ); Delaunay triangulation is done by max(min(∠ abc ,∠ bca ,∠ cab ))Maximize the minimum interior angle of the triangle to avoid generating long and narrow triangles; S2.2.2: For complex geometric areas, use the Poisson surface reconstruction algorithm to generate a smooth triangular mesh T ′ , based on the solution of Poisson's equation: Where N(p) is the normal vector field of the point and ρ(p) is the density function of the point.

7. The 3D point cloud target detection method for autonomous driving according to claim 6, characterized in that: The specific process of step S2.3 is: S2.3.1: Identify the hole boundary point set B, and reconstruct the hole area geometrically based on the boundary points, and interpolate to generate new points p i , the interpolation formula is: p i =αb1+βb2+(1-α-β)b3 Among them, α, β∈[0,1] are weight parameters in the barycentric coordinate system; S2.3.2: Generate filled triangles using minimum area triangulation, optimizing triangle area A: S2.3.3: For the repaired triangular mesh T ′ Perform Laplace smoothing, and the smoothed point cloud coordinates p i ′ for: Where N(i) is the point p i The neighborhood of , λ is the smoothing coefficient.

8. The 3D point cloud target detection method for autonomous driving according to claim 7, characterized in that: The specific process of step S2.5 is: Step 2.5.1: The newly generated point cloud P new Merge with the original point cloud P: P final =P∪P new Step 2.5.2: Use the KD-tree structure to organize the merged point cloud data and remove duplicate points; Step 2.5.3: The user selects whether the hole filling is completed. If so, the hole filling function is exited; otherwise, the process goes to step S2.1 to continue filling the hole.

9. The 3D point cloud target detection method for autonomous driving according to any one of claims 1 to 4, characterized in that: The specific process of step S03 is: S3.1, perform data preprocessing on all point clouds; S3.2, use the farthest point sampling algorithm to perform farthest point sampling on all point clouds to determine the sampling center point, and determine the neighborhood represented by each center point to construct an empty set S1; S3.3, for each point in the L1 spherical local areas obtained in step S3.2, a directed neighborhood graph is constructed using the KNN algorithm, edge features of the edges in each directed graph are calculated using a multi-layer perceptron, and then the features of the sampling center points in the M1 directed graphs are obtained using a maximum pooling operation; S3.4, use the farthest point sampling algorithm to perform farthest point sampling on all point clouds to determine the sampling center point, and determine the neighborhood represented by each center point to construct an empty set S2; S3.5, for each point in the L2 spherical local areas obtained in step S3.3, a directed neighborhood graph is constructed using the KNN algorithm, edge features of the edges in each directed graph are calculated using a multi-layer perceptron, and then the features of the sampling center points in the M2 directed graphs are obtained using a maximum pooling operation; S3.6, by recording the coordinates of each point, using the distance reverse interpolation algorithm to recover the features of the points lost by downsampling from the few points in the previous layer; S3.

7. By obtaining the category label of each point, after removing the background point cloud, the point cloud with the category label as the component is obtained, and then the point cloud of the component is clustered using a clustering algorithm to obtain the point cloud belonging to each component.

10. The 3D point cloud target detection method for autonomous driving according to claim 9, characterized in that: The specific process of step S3.1 is: S3.1.1: Use the remove_non_finite_points and remove_radius_outliers functions in open3d to remove invalid points in the point cloud; S3.1.2: Based on step S3.1.1, use uniform_down_sample in open3d to uniformly downsample the point cloud, and select a multiple of 30; S3.1.3: Based on step S3.1.2, the point cloud is uniformly downsampled using the uniform_down_sample function in open3d, with a multiple of 30; S3.1.4: Based on step S3.1.3, the normal vector of the point cloud is calculated using the estimate_normals function in open3d, the dimension of the point cloud is expanded from 3 dimensions to 6 dimensions, and the normal vector of the point cloud is normalized.

11. The 3D point cloud target detection method for autonomous driving according to claim 10, characterized in that: The specific process of step S3.2 is: S3.2.1: Randomly select a point p r ∈{p1,p2,.....p n }, added to set S1; S3.2.2: Calculate {p1,p2,.....p n The distance between each point in} and the set S1; S3.2.3: According to the point with the largest value among the results calculated in step S3.2.2, add it to set S1; S3.2.4: Determine whether the number of points in set S1 is greater than a threshold L1. If not, return to step S3.2.2 and repeat. Otherwise, proceed to the next step. S3.2.5: Select the points in set S1 in the order of arrangement; S3.2.6: Based on the point selected in step S3.2.5 as the center of the sphere, the number of points contained within the radius R1 is I1, M1 points are randomly selected from the I1 points to obtain a spherical local area containing M1 points and with a radius of R1, with the center of the sphere being the point selected in step S3.2.5; S3.2.7: Determine whether the points in set S1 have been completely selected. If not, return to step S3.2.5 and repeat. Otherwise, L1 spherical local areas {B1, B2, .....B L }, the center of each spherical local area is a point in set S1, and the process goes to step S3.

3.

12. The 3D point cloud target detection method for autonomous driving according to claim 11, characterized in that: The specific process of step S3.3 is: S3.3.1: For the L1 spherical regions B obtained in step S3.2.7 i , i∈[1,L1], select the points in order as the sampling center point, use the distance calculation formula in step S3.2.2 to calculate the distances from M1-1 other points to this point, and arrange them in ascending order according to the distance size; S3.3.2: Take the first k1 points calculated in step S3.3.1 as the nearest neighboring points of the sampling center point, and construct a directed graph G = (V, E); V = {1, 2, ..., k1} represents the nodes, and the edges formed by the points are E = V × V; S3.3.3: Determine whether the points in the spherical local area have been completely traversed. If not, repeat step S3.3.

1. If they have been completely traversed, proceed to step S3.3.

4. S3.3.4: Select the central node p i ,i∈[1,M1] and other points in its directed graph Directed edge ((i,j i1 ),……(i,j k1 )), define the edge feature as e ij =h θ (p i ,p j )=h θ (p i ,p j -p i ), where the function h is a nonlinear function containing a series of learnable parameters θ; S3.3.5: Select the central node p i and select other points in the directed graph in order Construct a directed edge between two points, use the three-dimensional coordinates, and use a multi-layer perceptron to calculate the edge feature vector of this edge. The output dimension of the MLP is C11; and this operation is applied to all other points Repeat the above to obtain k1 edge features; S3.3.6: Input the obtained k1 edge features into the maximum pooling layer, and obtain the features of each point in the M1 directed graph as the central node, p′ i =max j:(i,j)∈e h θ (p i ,p j ); concatenate the input domain M1 features of step S3.3 as the input of the next step; S3.3.7: Jump to step S3.3.1, take the feature vector obtained in the previous step as input, repeat n times, and in each repeated operation, the output dimensions of the multilayer perceptron in step S3.3.5 are C12, C13, C14, C15, and C16 respectively. The final feature vector is extracted using a multilayer perceptron with an output dimension of C17, and then the feature vector is pooled through a pooling layer to obtain a feature vector of size L1×C17.

13. The 3D point cloud target detection method for autonomous driving according to claim 12, characterized in that: The specific process of step S3.4 is: S3.4.1: Randomly select a point p r ∈{p1,p2,.....p L1 }, added to set S2; S3.4.2: Calculate {p1,p2,.....p L1 The distance from each point in} to set S2, where the definition of the distance from the point to the set is the same as step S3.2.2, S3.4.3: According to the point with the largest value among the results calculated in step S3.4.2, add it to set S2; S3.4.4: If the number of points in set S2 is greater than a certain threshold L2, then return to step S3.4.2 and repeat the execution; otherwise, proceed to the next step; S3.4.5: Select the points in set S2 in the order of arrangement; S3.4.6: Based on the point selected in step S3.4.5 as the center of the sphere, the number of points contained within the radius R2 is I2, and M2 points are randomly selected from the I2 points to obtain a spherical local area with a radius of R2 containing M2 points, and the center of the sphere is the point selected in step S3.4.5; S3.4.7: Determine whether the points in set S2 have been completely selected. If not, return to step S3.4.5 and repeat. Otherwise, L2 spherical local areas {B1, B2, .....B L2 }, the center of each spherical local area is a point in set S2, and the process goes to step S3.

5.

14. The 3D point cloud target detection method for autonomous driving according to claim 13, characterized in that: The specific process of step S3.5 is: S3.5.1: For the L2 spherical regions B obtained in step S3.4.7 i , i∈[1,L2], select the points in order as the sampling center point, use the distance calculation formula in step S3.2.2 to calculate the distances from M2-1 other points to this point, and arrange them in ascending order according to the distance size; S3.5.2: Take the first k2 points calculated in step S3.5.1 as the nearest neighboring points of the sampling center point, and construct a directed graph G = (V, E); V = {1, 2, ..., k2} represents the nodes, and the edges formed by the points are E = V × V; S3.5.3: Determine whether the points in the spherical local area have been completely traversed. If not, repeat step S3.5.

1. If they have been completely traversed, proceed to step S3.5.

4. S3.5.4: Select the central node p i ,i∈[1,M2] and other points in its directed graph Directed edge ((i,j i1 ),……(i,j k2 ), and the edge feature e ij =h θ (p i ,p j )=h θ (p i ,p j -p i ), the definition of function h is the same as in step S3.3.4; S3.5.5: Select the central node p i and select other points in the directed graph in order Construct a directed edge between two points, use the three-dimensional coordinates, and use a multi-layer perceptron to calculate the edge feature vector of this edge. The output dimension of the MLP is C21; and this operation is applied to all other points Repeat the above to obtain k2 edge features; S3.5.6: Input the obtained k1 edge features into the maximum pooling layer, and obtain M2 features of each point in this directed graph as the central node, p′ i =max j:(i,j)∈e h θ (p i ,p j ); concatenate the input of step S3.5 with M2 features as the input of the next step; S3.5.7: Jump to step S3.5.1 and use the feature vector obtained in the previous step as input. Repeat 5 times. In each repeated operation, the output dimensions of the multilayer perceptron in step S3.5.5 are C22, C23, C24, C25, and C26 respectively. Use the multilayer perceptron with an output dimension of C27 to extract features from the final feature vector, and then perform a pooling operation on the feature vector through a pooling layer to obtain a feature vector of size L2×C27.

15. The 3D point cloud target detection method for autonomous driving according to claim 14, characterized in that: The specific process of step S3.6 is: S3.6.1: The number of points obtained in step S3.4.1 is L2. The previous process is reversed to interpolate the number of point clouds from L2 to L1. The points to be interpolated are selected one by one. According to the coordinates of the previously recorded point clouds, the distance matrix d of each point in L1 from the points in L2 is calculated, which is L1×L2. S3.6.2: Sort each row in ascending order, take out the k points with the smallest median value in each row, and then calculate the weighted The weight Where ε represents a 1×10 -8 The minimum amount of , and finally the feature vector of size L1×C27 is calculated; S3.6.3: Concatenate the feature vector obtained in step S3.3.7 with the feature vector in step S3.6.2 to obtain a feature vector of size L1×(C27+C17); use a multi-layer perceptron with an output dimension of C3 to convert the feature vector into a feature vector of size L1×C3; S3.6.4: Repeat steps S3.6.1 to S3.6.3, the number of interpolation points is from L1 to n, the dimension of the MLP output is C4, and the feature vector of size n×(C3+C) is obtained; S3.6.5: Input the feature vector obtained in step S3.6.4 into the fully connected layer. The dimension of the output of the fully connected layer is C4. The fully connected layer uses softmax to limit the final result to 0 to 1, and finally obtains the probability of each point belonging to each category. Convert the probability vector of each point into a one-hot encoding, and convert the one-hot encoding into a decimal category label. Finally, obtain the label of the category to which each point belongs, and proceed to step S3.

7.

16. The 3D point cloud target detection method for autonomous driving according to claim 15, characterized in that: The specific process of step S3.7 is: S3.7.1: Remove the point clouds belonging to the background points from the obtained point clouds, and retain the point clouds belonging to the parts; S3.7.2: Cluster the remaining point clouds using the coordinates using the dbscan algorithm in open3d, with the algorithm parameters eps set to 2 and min_points set to 5, to obtain the point clouds of components belonging to different clustering areas; S3.7.3: Use the get_axis_aligned_bounding_box function for the point clouds of components in different areas to obtain the bounding box of each component area.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are performed.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 16.

19. A 3D point cloud target detection system for autonomous driving, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 16.

Images