Millimeter wave radar point cloud denoising method and device, equipment and storage medium

Abstract

The application discloses a millimeter wave radar point cloud denoising method and device, equipment and storage medium, belongs to image processing technical field. The method comprises the following steps: superimposing a plurality of millimeter wave radar point clouds to obtain a millimeter wave radar dense point cloud, wherein the plurality of millimeter wave radar point clouds are located in the same vehicle coordinate system; classifying the millimeter wave radar dense point cloud by a point cloud two-classification network to determine that each point in the millimeter wave radar dense point cloud belongs to an effective point or a noise point; and removing the points belonging to the noise points from the millimeter wave radar dense point cloud to obtain a denoised millimeter wave radar dense point cloud. The application aims to improve the quality of the millimeter wave dense point cloud obtained by superimposing multiple frames, and solves the technical problem that there are many noise points in the millimeter wave dense point cloud obtained by superimposing multiple frames, and the quality is low.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for denoising millimeter-wave radar point clouds. Background Technology

[0002] During autonomous driving, different tasks and levels of autonomous driving are achieved by acquiring sensor data from various sensors (cameras, LiDAR, ultrasonic radar, and millimeter-wave radar) mounted on the vehicle. Compared to the other three types of sensors, millimeter-wave radar is more robust in adverse weather conditions and can assist in various autonomous driving tasks. However, compared to LiDAR point clouds, millimeter-wave point clouds are sparser. In practical applications, multiple frames of millimeter-wave point clouds need to be superimposed to obtain dense millimeter-wave point clouds. However, single frames of millimeter-wave point clouds usually contain noise. When multiple frames of millimeter-wave point clouds are superimposed to obtain dense millimeter-wave point clouds, the noise is also superimposed, resulting in a large amount of noise and lower quality in the dense millimeter-wave point cloud. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, device and storage medium for denoising millimeter-wave radar point clouds, which aims to improve the quality of dense millimeter-wave point clouds obtained by superimposing multiple frames and solve the technical problem that there are many noise points and low quality in existing dense millimeter-wave point clouds obtained by superimposing multiple frames.

[0004] This application provides a method for noise reduction of millimeter-wave radar point clouds, including the following steps:

[0005] Several frames of millimeter-wave radar point clouds are superimposed to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system.

[0006] The dense point cloud of the millimeter-wave radar is classified using a point cloud binary classification network to determine whether each point in the dense point cloud of the millimeter-wave radar is a valid point or a noise point.

[0007] The noise-generating points are removed from the dense point cloud of the millimeter-wave radar to obtain the noise-reduced dense point cloud of the millimeter-wave radar.

[0008] In one possible implementation of this application, the step of classifying the dense point cloud of the millimeter-wave radar using a point cloud binary classification network to determine whether each point in the dense point cloud of the millimeter-wave radar belongs to a valid point or a noise point includes:

[0009] The dense point cloud of the millimeter-wave radar is voxelized to obtain several voxels.

[0010] Extract the voxel features of each voxel, wherein the voxel features include the position coordinates of the millimeter-wave radar point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system, and the distance between the millimeter-wave radar point cloud contained in the voxel and the centroid.

[0011] The voxel features of each voxel are input into a point cloud binary classification network, and the probability prediction value corresponding to each voxel is output. The point cloud binary classification network is a neural network based on sparse convolution.

[0012] If the predicted probability value is greater than the preset probability threshold, then all millimeter-wave radar point clouds contained in the voxel are determined to be valid points.

[0013] If the predicted probability value is less than or equal to a preset probability threshold, then all millimeter-wave radar point clouds contained in the voxel are determined to be noise.

[0014] In one possible implementation of this application, before the step of voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels, the method further includes:

[0015] The Z-axis value in the position coordinates of the dense point cloud of the millimeter-wave radar is updated to a preset Z-axis threshold to obtain the updated dense point cloud of the millimeter-wave radar.

[0016] The step of voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels includes:

[0017] The updated dense point cloud of the millimeter-wave radar was voxelized to obtain several voxels.

[0018] In one possible implementation of this application, the point cloud binary classification network is trained in the following manner:

[0019] Construct a pre-defined point cloud binary classification network based on sparse convolution;

[0020] Acquire the original millimeter-wave radar dense point cloud and the lidar point cloud at the same moment, wherein the original millimeter-wave radar dense point cloud and the lidar point cloud are both located in the same vehicle coordinate system;

[0021] Based on the lidar point cloud, the original millimeter-wave radar dense point cloud is labeled to obtain the actual category label of the original millimeter-wave radar dense point cloud;

[0022] The original millimeter-wave radar dense point cloud after annotation is voxelized to obtain several voxels.

[0023] Voxel features are extracted from the voxels and used as training sample data. The voxel features include: the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the original millimeter-wave radar dense point cloud contained in the voxel in the vehicle coordinate system, and the distance between the original millimeter-wave radar dense point cloud contained in the voxel and the centroid.

[0024] The training sample data is input into the preset point cloud binary classification network, and the probability prediction value corresponding to the training sample data is output.

[0025] Based on the probability prediction value, determine the predicted category label of the original millimeter-wave radar dense point cloud contained in the voxel corresponding to the training sample data;

[0026] Determine whether the network loss calculated based on the actual category label and the predicted category label has converged. If the network loss has not converged, update the preset point cloud binary classification network based on the network loss, and return to execute the steps of inputting the training sample data into the preset point cloud binary classification network and outputting the probability prediction value corresponding to the training sample data, and subsequent steps, until the network loss converges to obtain the trained point cloud binary classification network.

[0027] In one possible implementation of this application, the step of annotating the original millimeter-wave radar dense point cloud based on the lidar point cloud includes:

[0028] Cluster the lidar point cloud and the original millimeter-wave radar dense point cloud, and extract key points from the clustered lidar point cloud and the original millimeter-wave radar dense point cloud to obtain the lidar point cloud key point set and the original millimeter-wave radar dense point cloud key point set, wherein the key point is the centroid corresponding to the point cloud contained in each cluster.

[0029] For the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud, where there is a correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud, the distance between the two points is calculated. If the distance is less than a preset first distance threshold, the correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud is retained; otherwise, the correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud is not retained.

[0030] Based on the key points of the lidar point cloud that retain the correspondence and the key points of the original millimeter-wave radar dense point cloud, the lidar point cloud in the cluster corresponding to the key points of the lidar point cloud is extracted from the lidar point cloud, and the original millimeter-wave radar dense point cloud in the cluster corresponding to the key points of the original millimeter-wave radar dense point cloud is extracted from the original millimeter-wave radar dense point cloud.

[0031] Calculate the distance between the lidar point cloud and the original millimeter-wave radar dense point cloud in the same cluster. If the distance is greater than a preset second distance threshold, the original millimeter-wave radar dense point cloud is marked as a noise point; otherwise, the original millimeter-wave radar dense point cloud is marked as a valid point.

[0032] In one possible implementation of this application, the preset first distance threshold is proportional to the distance between the key points of the lidar point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system.

[0033] This application also provides a millimeter-wave radar point cloud noise reduction device, the device comprising:

[0034] The preprocessing module is used to superimpose several frames of millimeter-wave radar point clouds to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system.

[0035] The classification module is used to classify the dense point cloud of the millimeter-wave radar using a trained point cloud binary classification network, and to determine whether each point in the dense point cloud of the millimeter-wave radar belongs to a valid point or a noise point.

[0036] The noise reduction module is used to remove noise points from the dense point cloud of the millimeter-wave radar to obtain a noise-reduced dense point cloud of the millimeter-wave radar.

[0037] In one possible embodiment of this application, the classification module includes: a voxelization unit, used to voxelize the dense point cloud of the millimeter-wave radar to obtain a plurality of voxels; a feature extraction unit, used to extract voxel features of each voxel, wherein the voxel features include the position coordinates of the millimeter-wave radar point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system, and the distance between the millimeter-wave radar point cloud contained in the voxel and the centroid; a probability prediction unit, used to input the voxel features of each voxel into a point cloud binary classification network and output a probability prediction value corresponding to each voxel, wherein the point cloud binary classification network is a neural network based on sparse convolution; and a classification unit, used to determine that all millimeter-wave radar point clouds contained in the voxel belong to valid points if the probability prediction value is greater than a preset probability threshold, and to determine that all millimeter-wave radar point clouds contained in the voxel belong to noise points if the probability prediction value is less than or equal to the preset probability threshold.

[0038] And / or the classification module further includes: an update unit, used to update the Z-axis value in the position coordinates of the millimeter-wave radar dense point cloud to a preset Z-axis threshold, to obtain an updated millimeter-wave radar dense point cloud; the voxelization unit is used to perform voxelization processing on the updated millimeter-wave radar dense point cloud to obtain several voxels;

[0039] And / or the device further includes a network training module, the network training module comprising: a construction unit for constructing a preset point cloud binary classification network based on sparse convolution; a point cloud data acquisition unit for acquiring the original millimeter-wave radar dense point cloud and the lidar point cloud at the same time, wherein the original millimeter-wave radar dense point cloud and the lidar point cloud are both located in the same vehicle coordinate system; a labeling unit for labeling the original millimeter-wave radar dense point cloud based on the lidar point cloud to obtain the actual category label of the original millimeter-wave radar dense point cloud; and a training sample data acquisition unit for performing voxelization processing on the labeled original millimeter-wave radar dense point cloud to obtain several voxels, and extracting the voxel features of the voxels as training sample data, wherein the voxel features include: the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel coordinate system, the original millimeter-wave radar dense point cloud contained in the voxel, and the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel. The system includes: the position coordinates of the centroid corresponding to the dense point cloud of the millimeter-wave radar in the vehicle coordinate system; the distance between the original dense point cloud of the millimeter-wave radar contained in the voxel and the centroid; a training unit, used to input the training sample data into the preset point cloud binary classification network, output the probability prediction value corresponding to the training sample data; determine the predicted category label of the original dense point cloud of the millimeter-wave radar contained in the voxel corresponding to the training sample data based on the probability prediction value; and determine whether the network loss calculated based on the actual category label and the predicted category label has converged. If the network loss has not converged, the preset point cloud binary classification network is updated based on the network loss, and the system returns to execute the steps of inputting the training sample data into the preset point cloud binary classification network and outputting the probability prediction value corresponding to the training sample data, and subsequent steps, until the network loss converges, so as to obtain a trained point cloud binary classification network.

[0040] And / or the annotation unit includes: a first extraction subunit, used to cluster the lidar point cloud and the original millimeter-wave radar dense point cloud, and extract key points from the clustered lidar point cloud and the original millimeter-wave radar dense point cloud to obtain a lidar point cloud key point set and an original millimeter-wave radar dense point cloud key point set, wherein the key point is the centroid corresponding to the point cloud contained in each cluster; a correspondence verification subunit, used to calculate the distance between key points of the lidar point cloud and key points of the original millimeter-wave radar dense point cloud that have a correspondence in the key point set of the lidar point cloud and the key point set of the original millimeter-wave radar dense point cloud, and if the distance is less than a preset first distance threshold, then the correspondence between the key points of the lidar point cloud and the key points of the original millimeter-wave radar dense point cloud is preserved. If the key points are retained, the correspondence between the key points of the lidar point cloud and the key points of the original millimeter-wave radar dense point cloud is not retained; the second extraction subunit is used to extract the lidar point cloud in the cluster corresponding to the key points of the lidar point cloud from the lidar point cloud, and extract the original millimeter-wave radar dense point cloud in the cluster corresponding to the key points of the original millimeter-wave radar dense point cloud from the original millimeter-wave radar dense point cloud, based on the key points of the lidar point cloud with retained correspondence and the key points of the original millimeter-wave radar dense point cloud; the labeling subunit is used to calculate the distance between the lidar point cloud in the same cluster and the original millimeter-wave radar dense point cloud. If the distance is greater than a preset second distance threshold, the original millimeter-wave radar dense point cloud is labeled as noise; otherwise, the original millimeter-wave radar dense point cloud is labeled as valid point.

[0041] And / or the preset first distance threshold is proportional to the distance between the key points of the lidar point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system.

[0042] This application also provides a millimeter-wave radar point cloud noise reduction device, the device comprising: a memory, a processor, and a millimeter-wave radar point cloud noise reduction program stored in the memory and executable on the processor, the millimeter-wave radar point cloud noise reduction program being configured to implement the steps of the millimeter-wave radar point cloud noise reduction method as described above.

[0043] This application also provides a storage medium storing a millimeter-wave radar point cloud denoising program, which, when executed by a processor, implements the steps of the millimeter-wave radar point cloud denoising method described above.

[0044] This application provides a method, apparatus, device, and storage medium for denoising millimeter-wave radar point clouds. Compared with existing technologies where dense millimeter-wave point clouds obtained by superimposing multiple frames contain numerous noise points and have low quality, this application superimposes several frames of millimeter-wave radar point clouds to obtain a dense millimeter-wave radar point cloud, wherein all of the several frames of millimeter-wave radar point clouds are located in the same vehicle coordinate system. A point cloud binary classification network is used to classify the dense millimeter-wave radar point cloud, determining whether each point in the dense millimeter-wave radar point cloud is a valid point or a noise point. Points belonging to noise are removed from the dense millimeter-wave radar point cloud to obtain the denoised dense millimeter-wave radar point cloud. In this application, a point cloud binary classification network is used to classify the dense millimeter-wave radar point cloud obtained by superimposing multiple frames, determining whether each point in the dense millimeter-wave radar point cloud is a valid point or a noise point, and removing points belonging to noise from the dense millimeter-wave radar point cloud, thereby improving the quality of the dense millimeter-wave point cloud obtained by superimposing multiple frames. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating an embodiment of the millimeter-wave radar point cloud noise reduction method of this application;

[0046] Figure 2 for Figure 1 A detailed flowchart of step S20;

[0047] Figure 3 This is a schematic diagram of the first scenario for the millimeter-wave radar point cloud noise reduction method of this application.

[0048] Figure 4 This is a schematic diagram of the second scenario of the millimeter-wave radar point cloud noise reduction method of this application;

[0049] Figure 5 This is a schematic diagram of the structure of a millimeter-wave radar point cloud noise reduction device in the hardware operating environment involved in the embodiments of this application.

[0050] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0052] This application provides a method for denoising millimeter-wave radar point clouds, such as... Figure 1 As shown, in one embodiment of the millimeter-wave radar point cloud noise reduction method of this application, the millimeter-wave radar point cloud noise reduction method includes the following steps:

[0053] Step S10: Superimpose several frames of millimeter-wave radar point clouds to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system.

[0054] Step S20: Classify the dense point cloud of the millimeter-wave radar using a point cloud binary classification network to determine whether each point in the dense point cloud of the millimeter-wave radar is a valid point or a noise point.

[0055] Step S30: Remove noise points from the dense point cloud of the millimeter-wave radar to obtain the noise-reduced dense point cloud of the millimeter-wave radar.

[0056] This embodiment aims to improve the quality of millimeter-wave dense point clouds obtained by superimposing multiple frames.

[0057] Specifically, in this application, a point cloud binary classification network is used to classify the dense point cloud of millimeter-wave radar obtained by superimposing multiple frames, to determine whether each point in the dense point cloud of millimeter-wave radar belongs to a valid point or a noise point, and points belonging to noise points are removed from the dense point cloud of millimeter-wave radar, thereby improving the quality of the dense point cloud of millimeter-wave radar obtained by superimposing multiple frames.

[0058] Furthermore, in this application, the point cloud binary classification network is a sparse convolutional neural network that voxels the dense point cloud of millimeter-wave radar, obtaining several voxels. The point cloud binary classification network determines the probability prediction value corresponding to each voxel. The probability prediction value corresponding to each voxel is the probability prediction value of the millimeter-wave radar point cloud contained in the voxel, thereby determining whether the millimeter-wave radar point cloud belongs to a valid point or noise point. The point cloud binary classification network is a sparse convolutional neural network, which can make the spatial structure of the point cloud classification network more compact and expand the receptive field.

[0059] Furthermore, since the Z-axis value in the position coordinates of the millimeter-wave radar point cloud is inaccurate, in this application, before performing voxelization processing on the dense millimeter-wave radar point cloud to obtain several voxels, the Z-axis value in the position coordinates of the dense millimeter-wave radar point cloud needs to be updated to a preset Z-axis threshold to obtain an updated dense millimeter-wave radar point cloud. That is, in this application, assigning the same Z-axis value to the position coordinates of the dense millimeter-wave radar point cloud can eliminate the impact of inaccurate Z-axis values ​​in the position coordinates of the millimeter-wave radar point cloud, which leads to inaccurate classification results and the inability to completely remove noise points from the dense millimeter-wave radar point cloud, resulting in lower quality of the millimeter-wave dense point cloud obtained by superimposing multiple frames.

[0060] Furthermore, according to the logic that millimeter-wave radar points near non-lidar points are noise points, the millimeter-wave radar points can be labeled. However, in practical applications, due to the large number of points in the millimeter-wave radar point cloud and the lidar point cloud, the calculation time consumption is huge. At the same time, although the lidar point cloud is relatively accurate, there are still a small number of lidar points that are outlier noise points. Therefore, labeling according to the distance between points in the millimeter-wave radar point cloud and the lidar point cloud is time-consuming and has low labeling accuracy.

[0061] In this application, the lidar point cloud and the original dense millimeter-wave radar point cloud are clustered respectively to obtain multiple clustering clusters corresponding to the lidar point cloud and multiple clustering clusters corresponding to the original dense millimeter-wave radar point cloud. Based on the key points of the multiple clustering clusters corresponding to the lidar point cloud, a lidar point cloud key point set is formed. Based on the key points of the multiple clustering clusters corresponding to the original dense millimeter-wave radar point cloud, an original dense millimeter-wave radar point cloud key point set is formed. Calculate the distance between the lidar point cloud key points and the original dense millimeter-wave radar point cloud key points corresponding to the same cluster. If the distance is less than the preset first distance threshold, the corresponding relationship between the lidar point cloud key points and the original dense millimeter-wave radar point cloud key points is retained. Otherwise, the corresponding relationship between the lidar point cloud key points and the original dense millimeter-wave radar point cloud key points is not retained. That is, using the principle that the valid points in the millimeter-wave radar point cloud and the lidar point cloud are consistent, a small number of outlier point clouds in the lidar point cloud are removed, and the accuracy of labeling the millimeter-wave radar point cloud according to the lidar point cloud is improved.

[0062] In this application, based on the lidar point cloud key points and the original dense millimeter-wave radar point cloud key points with the corresponding relationship retained, the lidar points in the clustering cluster corresponding to the lidar point cloud key points are extracted from the lidar point cloud, and the original dense millimeter-wave radar points in the clustering cluster corresponding to the original dense millimeter-wave radar point cloud key points are extracted from the original dense millimeter-wave radar point cloud. Calculate the distance between the lidar points and the original dense millimeter-wave radar points in the same cluster. If the distance is greater than the preset second distance threshold, the original dense millimeter-wave radar points are labeled as noise points. Otherwise, the original dense millimeter-wave radar points are labeled as valid points. The calculation complexity can be significantly reduced. The number of points in the millimeter-wave radar point cloud and the lidar point cloud is N and M respectively, and the number of classes formed by clustering is K (K << N, M). The computational cost of using the above labeling steps to label the millimeter-wave radar is KxK+(NxM) / K, which is less than the computational amount of the method of directly assigning labels: NxM.

[0063] Furthermore, in this application, a first distance threshold is preset, which is proportional to the distance between the key points of the lidar point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system. This can improve the accuracy of identifying a small number of outlier point clouds in the lidar point cloud, thereby improving the accuracy of labeling the millimeter-wave radar point cloud based on the lidar point cloud.

[0064] In this embodiment, the specific application scenario is autonomous parking and obstacle avoidance in the field of autonomous driving.

[0065] The specific steps are as follows:

[0066] Step S10: Superimpose several frames of millimeter-wave radar point clouds to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system.

[0067] As an example, such as Figure 2 As shown, a millimeter-wave radar is configured at each of the four corners of the vehicle. Each millimeter-wave radar generates a single frame of millimeter-wave radar point cloud every t1 time interval. All frames of millimeter-wave radar point clouds generated by the four millimeter-wave radars in the previous T time interval are superimposed to obtain a dense millimeter-wave radar point cloud, where T≥t1.

[0068] As an example, before overlaying all frames of millimeter-wave radar point clouds generated by four millimeter-wave radars within the previous T time interval, it is necessary to perform time synchronization processing on all frames of millimeter-wave radar point clouds to be overlaid, resulting in time-synchronized millimeter-wave radar point clouds. It should be noted that performing time synchronization processing on all frames of millimeter-wave radar point clouds to be overlaid helps to eliminate errors in the millimeter-wave radar acquisition process and improves the quality of the dense millimeter-wave point cloud obtained by multi-frame overlay.

[0069] As an example, before superimposing all frames of millimeter-wave radar point clouds generated by the four millimeter-wave radars within the previous T time period, it is necessary to unify all frames of millimeter-wave radar point clouds to be superimposed into the vehicle coordinate system. Since all frames of millimeter-wave radar point clouds to be superimposed are generated by millimeter-wave radars positioned at the four corners of the vehicle, each millimeter-wave radar has its corresponding radar coordinate system. When the same object is scanned by different millimeter-wave radars, the position coordinates of the point clouds corresponding to the same object are different in different radar coordinate systems. Therefore, the position coordinates of all frames of millimeter-wave radar point clouds to be superimposed are transformed from their respective radar coordinate systems to the vehicle coordinate system. The vehicle coordinate system has the rear axle center as the origin, the forward direction as the Y-axis, the right direction as the X-axis, and the top as the Z-axis. This ensures the accuracy of the point cloud data in the superimposed dense millimeter-wave radar point cloud, preventing the same object from appearing in multiple point clouds within the dense millimeter-wave radar point cloud.

[0070] Step S20: Classify the dense point cloud of the millimeter-wave radar using a point cloud binary classification network to determine whether each point in the dense point cloud of the millimeter-wave radar is a valid point or a noise point.

[0071] As an example, such as Figure 3 As shown, the step of classifying the dense point cloud of the millimeter-wave radar using a point cloud binary classification network to determine whether each point in the dense point cloud of the millimeter-wave radar belongs to a valid point or a noise point includes the following steps S21 to S25:

[0072] Step S21: Perform voxelization on the dense point cloud of the millimeter-wave radar to obtain several voxels.

[0073] As an example, the step of voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels includes the following steps:

[0074] Step S211: Set the size of each voxel according to the range of the dense point cloud of the millimeter-wave radar;

[0075] Step S212: Divide the three-dimensional space of the dense point cloud of the millimeter-wave radar into several small spaces of equal volume according to the size of each voxel, to obtain multiple voxels.

[0076] For example, the dimensions of the three-dimensional space where the dense point cloud of the millimeter-wave radar is located are [20, 20, 10] in the height, width and depth dimensions. The dimensions of each voxel are set to [0.2, 0.2, 0.1] in the height, width and depth dimensions. Multiple voxels are generated according to the above-mentioned dimensions, and the resolution of the generated voxels is 100×100×100.

[0077] Step S22: Extract the voxel features of each voxel, wherein the voxel features include the position coordinates of the millimeter-wave radar point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system, and the distance between the millimeter-wave radar point cloud contained in the voxel and the centroid.

[0078] As an example, the position coordinates of the centroid of the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system are calculated using the following centroid calculation formula:

[0079]

[0080] Where m represents the number of millimeter-wave radar point clouds contained in a voxel, (X i Y i Z i(X) represents the position coordinates of the i-th millimeter-wave radar point cloud in the vehicle coordinate system. c Y c Z c The value represents the position coordinates of the centroid of the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system.

[0081] As an example, in the vehicle coordinate system, given the position coordinates of each millimeter-wave radar point cloud in a voxel and the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel, the distance between each millimeter-wave radar point in the voxel and the centroid can be calculated according to the distance calculation formula between the two points.

[0082] As an example, a voxel coordinate system can be constructed with the centroid of the millimeter-wave radar point cloud contained within the voxel as the origin, or with the center of the millimeter-wave radar point cloud contained within the voxel as the origin, or with the centroid of the millimeter-wave radar point cloud contained within the voxel as the origin, or with one of the corner points of the voxel as the origin. It should be noted that after voxelizing the dense millimeter-wave radar point cloud, the origins of the voxel coordinate systems of several voxels obtained are of the same type, and the positive directions of the X, Y, and Z axes in the voxel coordinate systems are also the same.

[0083] Step S23: Input the voxel features of each voxel into the point cloud binary classification network and output the probability prediction value corresponding to each voxel. The point cloud binary classification network is a neural network based on sparse convolution.

[0084] As an example, using the ResNet neural network as the basic network architecture, the convolutional layers in the basic network architecture are replaced with sparse convolutional layers, and a Softmax function layer is connected after the last fully connected layer of the basic network architecture to obtain a point cloud binary classification network.

[0085] In practical applications, voxel features are input into multiple sparse convolutional layers in the point cloud binary classification network for feature extraction to obtain spatial features. These spatial features are then used as the input to the last fully connected layer in the point cloud binary classification network. The last fully connected layer obtains the probability prediction value of each voxel belonging to a valid point by weighted summation of the spatial features and then mapping through the Softmax function layer. This prediction value is then used as the final output value of the point cloud binary classification network.

[0086] As an example, the underlying network architecture can be an FCN neural network, or the underlying network unit can be a multilayer perceptron (MLP).

[0087] Step S24: If the predicted probability value is greater than the preset probability threshold, then it is determined that all millimeter-wave radar point clouds contained in the voxel are valid points.

[0088] Step S25: If the probability prediction value is less than or equal to the preset probability threshold, then it is determined that all millimeter-wave radar point clouds contained in the voxel belong to noise.

[0089] As an example, the preset probability threshold is 0.5. That is, when the probability prediction value corresponding to a voxel is greater than 0.5, all millimeter-wave radar point clouds contained in that voxel are considered valid points; when the probability prediction value corresponding to a voxel is less than or equal to 0.5, all millimeter-wave radar point clouds contained in that voxel are considered noise points.

[0090] Furthermore, since the Z-axis information in the position coordinates of the millimeter-wave radar point cloud obtained from the millimeter-wave radar is inaccurate, the following steps are included before performing voxelization processing on the dense point cloud of the millimeter-wave radar to obtain several voxels:

[0091] The Z-axis value in the position coordinates of the dense point cloud of the millimeter-wave radar is updated to a preset Z-axis threshold to obtain the updated dense point cloud of the millimeter-wave radar.

[0092] The step of voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels includes:

[0093] The updated dense point cloud of the millimeter-wave radar was voxelized to obtain several voxels.

[0094] As an example, the preset Z-axis threshold is any value in [-1, 1].

[0095] Step S30: Remove noise points from the dense point cloud of the millimeter-wave radar to obtain the noise-reduced dense point cloud of the millimeter-wave radar.

[0096] As an example, such as Figure 4 As shown, Figure 4 The left image in the image is a visualization of the point cloud before noise reduction. Figure 4 The right image in the image is a visualization of the point cloud after noise reduction.

[0097] Compared to existing technologies where multi-frame superposition results in dense millimeter-wave point clouds with significant noise and low quality, this application superimposes several frames of millimeter-wave radar point clouds to obtain a dense millimeter-wave radar point cloud, where all frames are located in the same vehicle coordinate system. A point cloud binary classification network is used to classify the dense millimeter-wave radar point cloud, determining whether each point is a valid point or a noise point. Noise points are then removed from the dense millimeter-wave radar point cloud to obtain a denoised dense millimeter-wave radar point cloud. In this application, the use of a point cloud binary classification network to classify the multi-frame superposition millimeter-wave radar point cloud, determining whether each point is a valid point or a noise point, and removing noise points improves the quality of the multi-frame superposition millimeter-wave dense point cloud.

[0098] In another embodiment of the millimeter-wave radar point cloud denoising method of this application, the millimeter-wave radar point cloud denoising method further includes a point cloud binary classification network training method, wherein the point cloud binary classification network is trained in the following manner:

[0099] Step A1: Construct a pre-defined point cloud binary classification network based on sparse convolution.

[0100] As an example, using the ResNet neural network as the basic network architecture, the convolutional layers in the basic network architecture are replaced with sparse convolutional layers, and a Softmax function layer is connected after the last fully connected layer of the basic network architecture to obtain a point cloud binary classification network.

[0101] As an example, the underlying network architecture can be an FCN neural network, or the underlying network unit can be a multilayer perceptron (MLP).

[0102] Step A2: Obtain the original millimeter-wave radar dense point cloud and lidar point cloud at the same time, wherein the original millimeter-wave radar dense point cloud and the lidar point cloud are both located in the same vehicle coordinate system.

[0103] As an example, such as Figure 2 As shown, a lidar is installed on the top of the vehicle, and the current time is the moment when the lidar generates the lidar point cloud. A millimeter-wave radar is installed at each of the four corners of the vehicle. Each millimeter-wave radar generates a single frame of millimeter-wave radar point cloud every time interval t1. All frames of millimeter-wave radar point clouds generated by the four millimeter-wave radars in the time interval T' before the current time are superimposed to obtain the original dense millimeter-wave radar point cloud, where T' ≥ t1.

[0104] As an example, before overlaying all frames of millimeter-wave radar point clouds generated by four millimeter-wave radars within the previous T time period, it is necessary to perform time synchronization processing on all frames of millimeter-wave radar point clouds to be overlaid, so as to obtain millimeter-wave radar point clouds with time synchronization to the current time.

[0105] As an example, before superimposing all frames of millimeter-wave radar point clouds generated by the four millimeter-wave radars within the previous T' time period, the position coordinates of all frames of millimeter-wave radar point clouds to be superimposed are transformed from their respective radar coordinate systems to the vehicle coordinate system. The vehicle coordinate system has the rear axle center as the origin, the forward direction as the Y-axis, the right direction as the X-axis, and the top as the Z-axis. Similarly, the positions of the lidar point clouds are also transformed from their radar coordinate systems to the vehicle coordinate system.

[0106] Step A3: Based on the lidar point cloud, label the original millimeter-wave radar dense point cloud to obtain the actual category label of the original millimeter-wave radar dense point cloud.

[0107] As an example, the step of annotating the original millimeter-wave radar dense point cloud based on the lidar point cloud includes the following steps:

[0108] Step A31: Cluster the lidar point cloud and the original millimeter-wave radar dense point cloud, and extract key points from the clustered lidar point cloud and the original millimeter-wave radar dense point cloud to obtain the lidar point cloud key point set and the original millimeter-wave radar dense point cloud key point set, wherein the key point is the centroid corresponding to the point cloud contained in each cluster.

[0109] As an example, the K-Mean clustering algorithm is used to cluster the lidar point cloud and the original millimeter-wave radar dense point cloud respectively, resulting in multiple clusters corresponding to the lidar point cloud and the original millimeter-wave radar dense point cloud.

[0110] As an example, the centroid of the LiDAR point cloud contained in each of the multiple clusters corresponding to the LiDAR point cloud is calculated and used as the key point of each cluster. Based on the key points of the multiple clusters corresponding to the LiDAR point cloud, a key point set of the LiDAR point cloud is constructed. Similarly, the centroid of the original millimeter-wave radar dense point cloud contained in each of the multiple clusters corresponding to the original millimeter-wave radar dense point cloud is calculated and used as the key point of each cluster. Based on the key points of the multiple clusters corresponding to the original millimeter-wave radar dense point cloud, a key point set of the original millimeter-wave radar dense point cloud is constructed.

[0111] Step A32: For the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud that have a corresponding relationship, calculate the distance between the two points. If the distance is less than a preset first distance threshold, the correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud is retained; otherwise, the correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud is not retained.

[0112] As an example, each key point in the LiDAR point cloud key point set corresponds to a cluster. Similarly, each key point in the original millimeter-wave radar dense point cloud key point set also corresponds to a cluster. The distance between the LiDAR point cloud key points and the original millimeter-wave radar dense point cloud key points corresponding to the same cluster is calculated. If the distance is less than a preset first distance threshold, the correspondence between the LiDAR point cloud key points and the original millimeter-wave radar dense point cloud key points is retained; otherwise, the correspondence is not retained. Specifically, the LiDAR point cloud key points and the original millimeter-wave radar dense point cloud key points corresponding to the same cluster are considered to have a correspondence.

[0113] As an example, the preset first distance threshold is proportional to the distance between the key points of the LiDAR point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system. That is, the smaller the distance between the key points of the LiDAR point cloud and the origin of the vehicle coordinate system, the smaller the preset first distance threshold; the larger the distance between the key points of the LiDAR point cloud and the origin of the vehicle coordinate system, the larger the preset first distance threshold. Alternatively, the smaller the distance between the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system, the smaller the preset first distance threshold; the larger the distance between the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system, the larger the preset first distance threshold.

[0114] Step A33: Based on the key points of the lidar point cloud with preserved correspondence and the key points of the original millimeter-wave radar dense point cloud, extract the lidar point cloud in the cluster corresponding to the key points of the lidar point cloud from the lidar point cloud, and extract the original millimeter-wave radar dense point cloud in the cluster corresponding to the key points of the original millimeter-wave radar dense point cloud from the original millimeter-wave radar dense point cloud.

[0115] As an example, each key point in the lidar point cloud corresponds to a cluster. Extracting lidar point clouds from the cluster corresponding to the key points in the lidar point cloud is equivalent to extracting lidar point clouds from the cluster corresponding to the key points in the lidar point cloud. Similarly, each key point in the original millimeter-wave radar dense point cloud corresponds to a cluster. Extracting original millimeter-wave radar dense point clouds from the cluster corresponding to the key points in the original millimeter-wave radar dense point cloud is equivalent to extracting original millimeter-wave radar dense point clouds from the cluster corresponding to the key points in the original millimeter-wave radar dense point cloud.

[0116] Step A34: Calculate the distance between the lidar point cloud and the original millimeter-wave radar dense point cloud in the same cluster. If the distance is greater than a preset second distance threshold, the original millimeter-wave radar dense point cloud is marked as a noise point; otherwise, the original millimeter-wave radar dense point cloud is marked as a valid point.

[0117] As an example, the distance between the lidar point cloud and the original millimeter-wave radar dense point cloud in the same cluster is calculated, that is, the distance between the lidar point cloud extracted in step A33 and the original millimeter-wave radar dense point cloud is calculated.

[0118] It should be noted that if the original millimeter-wave radar dense point cloud is labeled as noise in step A34, then the actual category label of the original millimeter-wave radar dense point cloud is noise; if the original millimeter-wave radar dense point cloud is labeled as valid points, then the actual category label of the original millimeter-wave radar dense point cloud is valid points.

[0119] Step A4: Perform voxelization on the labeled original millimeter-wave radar dense point cloud to obtain several voxels.

[0120] It should be noted that the specific steps in step A4 of performing voxelization on the labeled original millimeter-wave radar dense point cloud to obtain several voxels are the same as those in step S21 of the previous embodiment, and will not be repeated in this embodiment.

[0121] As an example, the steps of voxelizing the labeled raw millimeter-wave radar dense point cloud to obtain several voxels include:

[0122] A portion of the point cloud is randomly extracted from the original dense point cloud of the millimeter-wave radar after annotation, and the extracted point cloud is voxelized to obtain several voxels.

[0123] Step A5: Extract the voxel features of the voxel as training sample data. The voxel features include: the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the original millimeter-wave radar dense point cloud contained in the voxel in the vehicle coordinate system, and the distance between the original millimeter-wave radar dense point cloud contained in the voxel and the centroid.

[0124] It should be noted that the specific steps for extracting the voxel features obtained in step A4 in step A5 are the same as those for step S22 in the previous embodiment, and will not be repeated in this embodiment.

[0125] Step A6: Input the training sample data into the preset point cloud binary classification network and output the probability prediction value corresponding to the training sample data.

[0126] In practical applications, training sample data is input into a multi-layer coefficient convolutional layer in a pre-defined point cloud binary classification network for feature extraction to obtain spatial features. These spatial features are then used as the input to the last fully connected layer in the point cloud binary classification network. The last fully connected layer obtains the probability prediction value of the original millimeter-wave radar dense point cloud containing the voxels corresponding to the training sample data as valid points by weighted summation of the spatial features and mapping through the Softmax function layer.

[0127] Step A7: Based on the probability prediction value, determine the predicted category label of the original millimeter-wave radar dense point cloud contained in the voxel corresponding to the training sample data.

[0128] As an example, if the predicted probability value is greater than the preset probability threshold, the predicted category label of the original millimeter-wave radar dense point cloud contained in the voxel corresponding to the training sample data is determined to be a valid point; if the predicted probability value is less than or equal to the preset probability threshold, the predicted category label of the original millimeter-wave radar dense point cloud contained in the voxel corresponding to the training sample data is determined to be a noise point.

[0129] As an example, the preset probability threshold is 0.5.

[0130] Step A8: Determine whether the network loss calculated based on the actual category label and the predicted category label has converged. If the network loss has not converged, update the preset point cloud binary classification network based on the network loss, and return to execute the steps of inputting the training sample data into the preset point cloud binary classification network and outputting the probability prediction value corresponding to the training sample data, and subsequent steps, until the network loss converges to obtain the trained point cloud binary classification network.

[0131] As an example, based on a predetermined loss function, the actual category labels and predicted category labels of the original millimeter-wave radar dense point cloud are calculated, and the calculated loss value is used as the network loss.

[0132] As an example, the predetermined loss function is binary cross-entropy.

[0133] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of a millimeter-wave radar point cloud noise reduction device in the hardware operating environment involved in the embodiments of this application.

[0134] like Figure 5 As shown, the millimeter-wave radar point cloud noise reduction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0135] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on millimeter-wave radar point cloud noise reduction equipment, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0136] like Figure 5 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a millimeter-wave radar point cloud noise reduction program.

[0137] exist Figure 5In the millimeter-wave radar point cloud noise reduction device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the millimeter-wave radar point cloud noise reduction device of this application can be set in the millimeter-wave radar point cloud noise reduction device, and the millimeter-wave radar point cloud noise reduction device implements the steps of the millimeter-wave radar point cloud noise reduction method described above by calling the millimeter-wave radar point cloud noise reduction program stored in the memory 1005 through the processor 1001.

[0138] The specific implementation of the millimeter-wave radar point cloud noise reduction device in this application is basically the same as the embodiments of the above-described millimeter-wave radar point cloud noise reduction method, and will not be repeated here.

[0139] This application also provides a millimeter-wave radar point cloud noise reduction device, the device comprising:

[0140] The preprocessing module is used to superimpose several frames of millimeter-wave radar point clouds to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system.

[0141] The classification module is used to classify the dense point cloud of the millimeter-wave radar using a trained point cloud binary classification network, and to determine whether each point in the dense point cloud of the millimeter-wave radar belongs to a valid point or a noise point.

[0142] The noise reduction module is used to remove noise points from the dense point cloud of the millimeter-wave radar to obtain a noise-reduced dense point cloud of the millimeter-wave radar.

[0143] In one possible embodiment of this application, the classification module includes: a voxelization unit, used to voxelize the dense point cloud of the millimeter-wave radar to obtain a plurality of voxels; a feature extraction unit, used to extract voxel features of each voxel, wherein the voxel features include the position coordinates of the millimeter-wave radar point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system, and the distance between the millimeter-wave radar point cloud contained in the voxel and the centroid; a probability prediction unit, used to input the voxel features of each voxel into a point cloud binary classification network and output a probability prediction value corresponding to each voxel, wherein the point cloud binary classification network is a neural network based on sparse convolution; and a classification unit, used to determine that all millimeter-wave radar point clouds contained in the voxel belong to valid points if the probability prediction value is greater than a preset probability threshold, and to determine that all millimeter-wave radar point clouds contained in the voxel belong to noise points if the probability prediction value is less than or equal to the preset probability threshold.

[0144] And / or the classification module further includes: an update unit, used to update the Z-axis value in the position coordinates of the millimeter-wave radar dense point cloud to a preset Z-axis threshold, to obtain an updated millimeter-wave radar dense point cloud; the voxelization unit is used to perform voxelization processing on the updated millimeter-wave radar dense point cloud to obtain several voxels;

[0145] And / or the device further includes a network training module, the network training module comprising: a construction unit for constructing a preset point cloud binary classification network based on sparse convolution; a point cloud data acquisition unit for acquiring the original millimeter-wave radar dense point cloud and the lidar point cloud at the same time, wherein the original millimeter-wave radar dense point cloud and the lidar point cloud are both located in the same vehicle coordinate system; a labeling unit for labeling the original millimeter-wave radar dense point cloud based on the lidar point cloud to obtain the actual category label of the original millimeter-wave radar dense point cloud; and a training sample data acquisition unit for performing voxelization processing on the labeled original millimeter-wave radar dense point cloud to obtain several voxels, and extracting the voxel features of the voxels as training sample data, wherein the voxel features include: the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel coordinate system, the original millimeter-wave radar dense point cloud contained in the voxel, and the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel. The system includes: the position coordinates of the centroid corresponding to the dense point cloud of the millimeter-wave radar in the vehicle coordinate system; the distance between the original dense point cloud of the millimeter-wave radar contained in the voxel and the centroid; a training unit, used to input the training sample data into the preset point cloud binary classification network, output the probability prediction value corresponding to the training sample data; determine the predicted category label of the original dense point cloud of the millimeter-wave radar contained in the voxel corresponding to the training sample data based on the probability prediction value; and determine whether the network loss calculated based on the actual category label and the predicted category label has converged. If the network loss has not converged, the preset point cloud binary classification network is updated based on the network loss, and the system returns to execute the steps of inputting the training sample data into the preset point cloud binary classification network and outputting the probability prediction value corresponding to the training sample data, and subsequent steps, until the network loss converges, so as to obtain a trained point cloud binary classification network.

[0146] And / or the annotation unit includes: a first extraction subunit, used to cluster the lidar point cloud and the original millimeter-wave radar dense point cloud, and extract key points from the clustered lidar point cloud and the original millimeter-wave radar dense point cloud to obtain a lidar point cloud key point set and an original millimeter-wave radar dense point cloud key point set, wherein the key point is the centroid corresponding to the point cloud contained in each cluster; a correspondence verification subunit, used to calculate the distance between key points of the lidar point cloud and key points of the original millimeter-wave radar dense point cloud that have a correspondence in the key point set of the lidar point cloud and the key point set of the original millimeter-wave radar dense point cloud, and if the distance is less than a preset first distance threshold, then the correspondence between the key points of the lidar point cloud and the key points of the original millimeter-wave radar dense point cloud is preserved. If the key points are retained, the correspondence between the key points of the lidar point cloud and the key points of the original millimeter-wave radar dense point cloud is not retained; the second extraction subunit is used to extract the lidar point cloud in the cluster corresponding to the key points of the lidar point cloud from the lidar point cloud, and extract the original millimeter-wave radar dense point cloud in the cluster corresponding to the key points of the original millimeter-wave radar dense point cloud from the original millimeter-wave radar dense point cloud, based on the key points of the lidar point cloud with retained correspondence and the key points of the original millimeter-wave radar dense point cloud; the labeling subunit is used to calculate the distance between the lidar point cloud in the same cluster and the original millimeter-wave radar dense point cloud. If the distance is greater than a preset second distance threshold, the original millimeter-wave radar dense point cloud is labeled as noise; otherwise, the original millimeter-wave radar dense point cloud is labeled as valid point.

[0147] And / or the preset first distance threshold is proportional to the distance between the key points of the lidar point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system.

[0148] The specific implementation of the millimeter-wave radar point cloud noise reduction device in this application is basically the same as the embodiments of the above-mentioned millimeter-wave radar point cloud noise reduction method, and will not be repeated here.

[0149] This application also provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the millimeter-wave radar point cloud noise reduction method described in any of the above claims.

[0150] The specific implementation of the storage medium in this application is basically the same as the embodiments of the above-described millimeter-wave radar point cloud noise reduction method, and will not be repeated here.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described millimeter-wave radar point cloud noise reduction method.

[0152] The specific implementation of the computer program product in this application is basically the same as the embodiments of the above-described millimeter-wave radar point cloud noise reduction method, and will not be repeated here.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0154] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0156] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for noise reduction of millimeter-wave radar point clouds, characterized in that, The millimeter-wave radar point cloud noise reduction method includes the following steps: Several frames of millimeter-wave radar point clouds from the four corners of the vehicle are superimposed to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system. The dense point cloud of the millimeter-wave radar is classified using a point cloud binary classification network to determine whether each point in the dense point cloud is a valid point or a noise point. The point cloud binary classification network is constructed based on a sparse convolutional neural network and is trained based on the actual category labels of the original dense point cloud of the millimeter-wave radar. The actual category labels are obtained by labeling the distance between the lidar point cloud and the original dense point cloud in the same cluster and a preset second distance threshold. The lidar point cloud and the original dense point cloud in the same cluster are determined based on key points of the lidar point cloud and key points of the original dense point cloud that retain correspondence. The key points of the lidar point cloud and the original dense point cloud that retain correspondence are determined based on the distance between key points of the lidar point cloud and key points of the original dense point cloud that have correspondence and a preset first distance threshold. The noise-generating points are removed from the dense point cloud of the millimeter-wave radar to obtain the noise-reduced dense point cloud of the millimeter-wave radar.

2. The millimeter-wave radar point cloud noise reduction method as described in claim 1, characterized in that, The step of classifying the dense point cloud of the millimeter-wave radar using a point cloud binary classification network to determine whether each point in the dense point cloud of the millimeter-wave radar belongs to a valid point or a noise point includes: The dense point cloud of the millimeter-wave radar is voxelized to obtain several voxels. Extract the voxel features of each voxel, wherein the voxel features include the position coordinates of the millimeter-wave radar point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system, and the distance between the millimeter-wave radar point cloud contained in the voxel and the centroid. The voxel features of each voxel are input into a point cloud binary classification network, and the probability prediction value corresponding to each voxel is output. The point cloud binary classification network is a neural network based on sparse convolution. If the predicted probability value is greater than the preset probability threshold, then all millimeter-wave radar point clouds contained in the voxel are determined to be valid points. If the predicted probability value is less than or equal to a preset probability threshold, then all millimeter-wave radar point clouds contained in the voxel are determined to be noise.

3. The millimeter-wave radar point cloud noise reduction method as described in claim 2, characterized in that, Before the step of voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels, the method further includes: The Z-axis value in the position coordinates of the dense point cloud of the millimeter-wave radar is updated to a preset Z-axis threshold to obtain the updated dense point cloud of the millimeter-wave radar. The step of voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels includes: The updated dense point cloud of the millimeter-wave radar was voxelized to obtain several voxels.

4. The millimeter-wave radar point cloud noise reduction method as described in claim 1, characterized in that, The point cloud binary classification network is trained in the following way: Construct a pre-defined point cloud binary classification network based on sparse convolution; Acquire the original millimeter-wave radar dense point cloud and the lidar point cloud at the same moment, wherein the original millimeter-wave radar dense point cloud and the lidar point cloud are both located in the same vehicle coordinate system; Based on the lidar point cloud, the original millimeter-wave radar dense point cloud is labeled to obtain the actual category label of the original millimeter-wave radar dense point cloud; The original millimeter-wave radar dense point cloud after annotation is voxelized to obtain several voxels. Voxel features are extracted from the voxels and used as training sample data. The voxel features include: the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the original millimeter-wave radar dense point cloud contained in the voxel in the vehicle coordinate system, and the distance between the original millimeter-wave radar dense point cloud contained in the voxel and the centroid. The training sample data is input into the preset point cloud binary classification network, and the probability prediction value corresponding to the training sample data is output. Based on the probability prediction value, determine the predicted category label of the original millimeter-wave radar dense point cloud contained in the voxel corresponding to the training sample data; Determine whether the network loss calculated based on the actual category label and the predicted category label has converged. If the network loss has not converged, update the preset point cloud binary classification network based on the network loss, and return to execute the steps of inputting the training sample data into the preset point cloud binary classification network and outputting the probability prediction value corresponding to the training sample data, and subsequent steps, until the network loss converges to obtain the trained point cloud binary classification network.

5. The millimeter-wave radar point cloud noise reduction method as described in claim 4, characterized in that, The step of annotating the original millimeter-wave radar dense point cloud based on the lidar point cloud includes: Cluster the lidar point cloud and the original millimeter-wave radar dense point cloud, and extract key points from the clustered lidar point cloud and the original millimeter-wave radar dense point cloud to obtain the lidar point cloud key point set and the original millimeter-wave radar dense point cloud key point set, wherein the key point is the centroid corresponding to the point cloud contained in each cluster. For the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud, where there is a correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud, the distance between the two points is calculated. If the distance is less than a preset first distance threshold, the correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud is retained; otherwise, the correspondence between the key points in the lidar point cloud and the key points in the original millimeter-wave radar dense point cloud is not retained. Based on the key points of the lidar point cloud that retain the correspondence and the key points of the original millimeter-wave radar dense point cloud, the lidar point cloud in the cluster corresponding to the key points of the lidar point cloud is extracted from the lidar point cloud, and the original millimeter-wave radar dense point cloud in the cluster corresponding to the key points of the original millimeter-wave radar dense point cloud is extracted from the original millimeter-wave radar dense point cloud. Calculate the distance between the lidar point cloud and the original millimeter-wave radar dense point cloud in the same cluster. If the distance is greater than a preset second distance threshold, the original millimeter-wave radar dense point cloud is marked as a noise point; otherwise, the original millimeter-wave radar dense point cloud is marked as a valid point.

6. The millimeter-wave radar point cloud noise reduction method as described in claim 5, characterized in that, The preset first distance threshold is proportional to the distance between the key points of the lidar point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system.

7. A millimeter-wave radar point cloud noise reduction device, characterized in that, The device includes: The preprocessing module is used to superimpose several frames of millimeter-wave radar point clouds from the four corners of the vehicle to obtain a dense millimeter-wave radar point cloud, wherein the several frames of millimeter-wave radar point clouds are all located in the same vehicle coordinate system. A classification module is used to classify the dense point cloud of the millimeter-wave radar using a trained point cloud binary classification network, determining whether each point in the dense point cloud of the millimeter-wave radar belongs to a valid point or a noise point. The point cloud binary classification network is constructed based on a sparse convolutional neural network and is trained based on the actual category labels of the original dense point cloud of the millimeter-wave radar. The actual category labels are labeled based on the distance between the lidar point cloud and the original dense point cloud in the same cluster and a preset second distance threshold. The lidar point cloud and the original dense point cloud in the same cluster are determined based on key points of the lidar point cloud and key points of the original dense point cloud that retain correspondence. The key points of the lidar point cloud and the original dense point cloud that retain correspondence are determined based on the distance between key points of the lidar point cloud and key points of the original dense point cloud that have correspondence and a preset first distance threshold. The noise reduction module is used to remove noise points from the dense point cloud of the millimeter-wave radar to obtain a noise-reduced dense point cloud of the millimeter-wave radar.

8. The millimeter-wave radar point cloud noise reduction device as described in claim 7, characterized in that, The classification module includes: a voxelization unit for voxelizing the dense point cloud of the millimeter-wave radar to obtain several voxels; a feature extraction unit for extracting voxel features of each voxel, wherein the voxel features include the position coordinates of the millimeter-wave radar point cloud contained in the voxel in the voxel coordinate system, the position coordinates of the centroid corresponding to the millimeter-wave radar point cloud contained in the voxel in the vehicle coordinate system, and the distance between the millimeter-wave radar point cloud contained in the voxel and the centroid; a probability prediction unit for inputting the voxel features of each voxel into a point cloud binary classification network and outputting a probability prediction value corresponding to each voxel, wherein the point cloud binary classification network is a neural network based on sparse convolution; and a classification unit for determining that if the probability prediction value is greater than a preset probability threshold, all millimeter-wave radar point clouds contained in the voxel belong to valid points; and if the probability prediction value is less than or equal to the preset probability threshold, all millimeter-wave radar point clouds contained in the voxel belong to noise points. And / or the classification module further includes: an update unit, used to update the Z-axis value in the position coordinates of the millimeter-wave radar dense point cloud to a preset Z-axis threshold, to obtain an updated millimeter-wave radar dense point cloud; the voxelization unit is used to perform voxelization processing on the updated millimeter-wave radar dense point cloud to obtain several voxels; And / or the device further includes a network training module, the network training module comprising: a construction unit for constructing a preset point cloud binary classification network based on sparse convolution; a point cloud data acquisition unit for acquiring the original millimeter-wave radar dense point cloud and the lidar point cloud at the same time, wherein the original millimeter-wave radar dense point cloud and the lidar point cloud are both located in the same vehicle coordinate system; a labeling unit for labeling the original millimeter-wave radar dense point cloud based on the lidar point cloud to obtain the actual category label of the original millimeter-wave radar dense point cloud; and a training sample data acquisition unit for performing voxelization processing on the labeled original millimeter-wave radar dense point cloud to obtain several voxels, and extracting the voxel features of the voxels as training sample data, wherein the voxel features include: the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel coordinate system, the original millimeter-wave radar dense point cloud contained in the voxel, and the position coordinates of the original millimeter-wave radar dense point cloud contained in the voxel in the voxel. The system includes: the position coordinates of the centroid corresponding to the dense point cloud of the millimeter-wave radar in the vehicle coordinate system; the distance between the original dense point cloud of the millimeter-wave radar contained in the voxel and the centroid; a training unit, used to input the training sample data into the preset point cloud binary classification network, output the probability prediction value corresponding to the training sample data; determine the predicted category label of the original dense point cloud of the millimeter-wave radar contained in the voxel corresponding to the training sample data based on the probability prediction value; and determine whether the network loss calculated based on the actual category label and the predicted category label has converged. If the network loss has not converged, the preset point cloud binary classification network is updated based on the network loss, and the system returns to execute the steps of inputting the training sample data into the preset point cloud binary classification network and outputting the probability prediction value corresponding to the training sample data, and subsequent steps, until the network loss converges, so as to obtain a trained point cloud binary classification network. And / or the annotation unit includes: a first extraction subunit, used to cluster the lidar point cloud and the original millimeter-wave radar dense point cloud, and extract key points from the clustered lidar point cloud and the original millimeter-wave radar dense point cloud to obtain a lidar point cloud key point set and an original millimeter-wave radar dense point cloud key point set, wherein the key point is the centroid corresponding to the point cloud contained in each cluster; a correspondence verification subunit, used to calculate the distance between key points of the lidar point cloud and key points of the original millimeter-wave radar dense point cloud that have a correspondence in the key point set of the lidar point cloud and the key point set of the original millimeter-wave radar dense point cloud, and if the distance is less than a preset first distance threshold, then the correspondence between the key points of the lidar point cloud and the key points of the original millimeter-wave radar dense point cloud is preserved. If the key points are retained, the correspondence between the key points of the lidar point cloud and the key points of the original millimeter-wave radar dense point cloud is not retained; the second extraction subunit is used to extract the lidar point cloud in the cluster corresponding to the key points of the lidar point cloud from the lidar point cloud, and extract the original millimeter-wave radar dense point cloud in the cluster corresponding to the key points of the original millimeter-wave radar dense point cloud from the original millimeter-wave radar dense point cloud, based on the key points of the lidar point cloud with retained correspondence and the key points of the original millimeter-wave radar dense point cloud; the labeling subunit is used to calculate the distance between the lidar point cloud in the same cluster and the original millimeter-wave radar dense point cloud. If the distance is greater than a preset second distance threshold, the original millimeter-wave radar dense point cloud is labeled as noise; otherwise, the original millimeter-wave radar dense point cloud is labeled as valid point. And / or the preset first distance threshold is proportional to the distance between the key points of the lidar point cloud or the key points of the original millimeter-wave radar dense point cloud and the origin of the vehicle coordinate system.

9. A millimeter-wave radar point cloud noise reduction device, characterized in that, The device includes: a memory, a processor, and a millimeter-wave radar point cloud denoising program stored in the memory and executable on the processor, the millimeter-wave radar point cloud denoising program being configured to implement the steps of the millimeter-wave radar point cloud denoising method as described in any one of claims 1 to 6.

10. A storage medium, characterized in that, The storage medium stores a millimeter-wave radar point cloud denoising program, which, when executed by a processor, implements the steps of the millimeter-wave radar point cloud denoising method as described in any one of claims 1 to 6.

Images