A lightweight point cloud denoising method based on convolutional neural network and related equipment

Abstract

The present invention relates to a lightweight point cloud denoising method and related device based on a convolutional neural network, comprising the following steps: step S1: extract shallow features; step S2: extract deep features and fuse features at the same time; step S3: perform a Dropout operation on the final features obtained to prevent overfitting, and then use convolution to reduce the number of feature channels to the number of point cloud categories, thereby obtaining a classification result of noise points, and then removing the noise points. The present invention uses Lidar point cloud data obtained in extreme weather to perform denoising tasks, and uses a LilaBlock module and a FEAR module to perform denoising tasks. Compared with the traditional WeatherNet denoising network, the Head part (FEAR) of the present invention is fully fused with the shallow features obtained after the LilaBlock convolution operation, so that the network can learn more hidden features in the point cloud data; at the same time, the FEAR module used in the present invention, after better fusion of features, maintains the advantage of lightweight, which is convenient for saving computing resources in subsequent denoising tasks.

Description

Technical Field

[0001] The present invention relates to three-dimensional visual point cloud processing technology, and in particular to a lightweight point cloud denoising method based on convolutional neural network and related equipment. Background Art

[0002] LiDAR sensors are often used for environmental perception in autonomous vehicles and mobile robots. However, severe weather conditions can cause radar sensors to capture unwanted measurement points, leading to missed detections and false alarms, which seriously affect the performance of LiDAR-based scene understanding. In heavy rain or dense fog, water droplets may even be misinterpreted as objects in front of the vehicle, causing the mobile robot to stop completely.

[0003] LiDAR technology faces severe challenges in bad weather, such as fog, dust, snow, rain, pollution, and haze. Since the radar receives backscattered light from water droplets such as rain or fog or any particles, it will have a serious impact on the distance measurement, resulting in erroneous measurement data in the point cloud data. In scene perception or target detection algorithms, these points are undesirable noise and need special processing, otherwise it will reduce the scene understanding performance of the algorithm.

[0004] In the 3D domain, many methods are based on spatial proximity or statistical distribution of point clouds, such as Statistical Outlier Removal (SOR) and Radius Outlier Removal (ROR) filters. SOR defines the neighborhood of a point based on the average distance of a point to all k neighbors compared to a threshold derived from the global average distance and standard deviation of all points. ROR filters directly count the number of neighbors within a radius r to decide whether to filter a point. But these filter types are not suitable for the denoising task of sparse point clouds contaminated by snow, and these methods discard single reflections that have no points in the neighborhood based on spatial proximity. Points with farther distances are increasingly filtered, such as SOR, ROR and even DROR. For autonomous vehicles, especially at higher speeds, valuable information is discarded and the range of the sensor is also limited by the filters. Summary of the invention

[0005] In view of this, it is necessary to provide a lightweight point cloud denoising method and related equipment based on convolutional neural networks to address the above problems.

[0006] In a first aspect, the present invention provides a lightweight point cloud denoising method based on a convolutional neural network, comprising the following steps:

[0007] Step S1: perform shallow feature extraction;

[0008] Step S2: perform deep feature extraction and feature fusion at the same time;

[0009] Step S3: Dropout operation is performed on the final features to prevent overfitting, and then convolution is used to reduce the number of feature channels to the number of point cloud categories, thereby obtaining the classification results of the noise points and removing the noise points.

[0010] Furthermore, the specific process of step S1 is as follows:

[0011] S11: Input the distance and intensity of each point in the Lidar point cloud data point by point;

[0012] S12: concat the two pieces of information;

[0013] S13: Use four convolution kernels of different scales on the combined data to extract the corresponding features, increase the feature dimension, fuse the four extracted features, and then use a 1×1 convolution kernel to further extract the fused features.

[0014] Furthermore, the specific process of step S2 is as follows:

[0015] Step S21: further enhancing the shallow features obtained in step S1;

[0016] Step S22: Fusing the enhancement result with the shallow feature to obtain a fused feature;

[0017] Step S23: Add the fusion feature obtained in step S22 and the shallow feature obtained in step S1 to obtain the final feature.

[0018] Furthermore, the specific operations of step S21 are as follows:

[0019] S211: feature amplification through two 1×1 convolution kernels;

[0020] S212: Use ReLU function between convolutional layers for nonlinear processing;

[0021] S213: Use 3×3 convolution kernel for feature extraction;

[0022] S214: After extraction, use SA module grouping to perform feature learning;

[0023] S215: Use ⊕ residual structure to prevent over-complexity of parameters.

[0024] Furthermore, the specific operations of step S22 are as follows:

[0025] First, the shallow features and the enhanced features obtained by the WIRW module are concatenated through the Concat operation to obtain the concatenated features; then it is divided into two branches, one of which amplifies the concatenated features through a 11 convolution kernel, performs nonlinear processing through Sigmoid, and multiplies them with the enhanced results; the other branch only amplifies the concatenated features through a 1×1 convolution kernel; the two results are subjected to residual operation to obtain the final fused features; the residual structure is used to retain the features after processing.

[0026] In a second aspect, the present invention provides a lightweight point cloud denoising device based on a convolutional neural network, comprising: a shallow layer extraction module: used for performing shallow layer feature extraction using a LilaBlock module;

[0027] Extraction and fusion module: used to perform deep feature extraction and feature fusion using the FEAR module;

[0028] Denoising module: It is used to perform Dropout operation on the final features to prevent overfitting, and then use convolution to reduce the number of feature channels to the number of point cloud categories, thereby obtaining the classification results of the noise points and removing the noise points.

[0029] Furthermore, the FEAR module includes:

[0030] WIRW module: used to further enhance the shallow features obtained;

[0031] DF module: used to fuse the enhancement results with the shallow features to obtain fused features;

[0032] ⊕Residual structure: used to add the obtained fusion features and the obtained shallow features to obtain the final features.

[0033] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the above-mentioned lightweight point cloud denoising method based on a convolutional neural network.

[0034] In a fourth aspect, the present invention provides a storage medium storing computer executable instructions. When the computer executable instructions are loaded and executed by a processor, the steps of the above-mentioned lightweight point cloud denoising method based on a convolutional neural network are implemented.

[0035] The advantages and beneficial effects of the present invention are:

[0036] The present invention uses Lidar point cloud data obtained in extreme weather to perform denoising tasks and improve denoising accuracy. The algorithm uses LilaBlock module and FEAR module to perform denoising tasks. Compared with the traditional WeatherNet denoising network, the Head part FEAR of the present invention is fully integrated with the shallow features obtained after the LilaBlock convolution operation, so that the network can learn more hidden features in the point cloud data; at the same time, the FEAR module used in the present invention, after better fusion of features, maintains the advantage of lightweight, which is convenient for saving computing resources in subsequent denoising tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The technical solution of the present invention will be further described below in conjunction with specific embodiments and drawings.

[0038] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0039] Figure 2 is a specific flow chart of step S1;

[0040] Figure 3 is a schematic diagram of the specific process of step S2;

[0041] Figure 4 is a schematic diagram of the specific process of step S21;

[0042] Figure 5 is a specific flow chart of step S22;

[0043] Figure 6 It is a schematic diagram of the overall structure of the present invention;

[0044] Figure 7 This is the structure diagram of the LilaBlock module and the FEAR module; Figure 7 (a) is the LilaBlock module structure diagram; Figure 7 (b) is the FEAR module structure diagram;

[0045] Figure 8 This is the structure diagram of the FEAR module; Figure 8 (a) is the structural diagram of the WIRW module; Figure 8 (b) is the structural diagram of the DF module;

[0046] Fig. 9 This is the SA module structure diagram; in the figure: w, h, c, g represent the height, width, number of channels and number of groups respectively; F GN represents group normalization; F gP represents global pooling; F C represents the fully connected structure; σ represents the Sigmoid activation function;

[0047] Fig.10 It is a real-time Lidar point cloud image;

[0048] Fig.11 It is the denoised Lidar point cloud image. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Embodiment 1:

[0051] Multiple LiDARs are installed in front and behind the autonomous vehicle to achieve 360° detection coverage. During the driving process, the on-board LiDAR continuously collects information about the surrounding environment of the vehicle and senses the environment. On-board LiDAR is greatly affected by rain, snow, and fog. When visibility is low, it will interfere with the collection of LiDAR data, forming a large number of irregular noise points, affecting the autonomous driving system's perception of the surrounding environment, and even causing missed detections and false alarms, thus causing the autonomous driving system to fail.

[0052] like Figure 1 As shown, this embodiment provides a lightweight point cloud denoising method based on a convolutional neural network, comprising the following steps:

[0053] Step S1: Use the LilaBlock module to extract shallow features; specifically, perform a concat operation on the distance and detection intensity of each point in the obtained Lidar point cloud data containing extreme weather conditions, and then perform shallow feature extraction through the LiLaBlock module. The LilaBlock (LiDAR-based semantic labeling block) module is a radar semantic segmentation module; concat is a splicing operation.

[0054] Step S2: Use the FEAR (feature enhance and resnet) module to extract deep features and fuse features: Specifically, use the FEAR module to further extract the shallow features obtained in step S1, and fuse them with the shallow features to obtain fused features. Among them: The FEAR (feature enhance and resnet) module is a feature enhancement and residual network module. Figure 7 As shown in (b), the FEAR module includes:

[0055] WIRW module: used to further enhance the shallow features F obtained in step S1;

[0056] DF (DeepFuse) module: used to fuse the enhancement results with shallow features to obtain fused features. DeepFuse stands for deep fusion.

[0057] Residual structure: used to add the fusion features obtained in step S22 and the shallow features obtained in step S1 to obtain the final features

[0058] The shallow features are passed through the WIRW module to obtain enhanced deep features. The deep features and shallow features are input into the DF module for fusion to obtain fused features. Finally, the residual structure is added to the shallow features to obtain the final features.

[0059] Step S3: Perform a Dropout operation on the obtained deep features to prevent overfitting, and then use convolution to reduce the number of feature channels to the number of point cloud categories, thereby obtaining the classification results of the noise points and removing the noise points. Among them: Dropout operation refers to pruning operation.

[0060] like Figure 2 As shown, the specific process of step S1 is:

[0061] S11: Input the distance and intensity of each point in the Lidar point cloud data point by point;

[0062] S12: Then concat the two pieces of information.

[0063] S13: The combined data contains the distance information and intensity information of the point cloud, using Figure 7 The LilaBlock module shown in (a) uses four convolution kernels of different scales to convolve the combined data, extract the corresponding features, and increase the feature dimension. Then, the four sets of extracted features are concat-joined, and finally a 1×1 convolution kernel is used to further extract the concatenated features to obtain the shallow features of the target. The four convolution kernels of different scales are: convolution kernel size 7×3, convolution kernel size 3×3, convolution kernel size 3×3, dilated convolution with a dilation rate of 2, and convolution kernel size 3×7.

[0064] Figure 7 In (a): Input represents the distance and intensity information of the input point cloud, Conv7×3 represents 7×3 convolution, Conv3×3 represents 3×3 convolution; Conv3×3 and Dilation2×2 represent 3×3 dilated convolution; Conv3×7 represents 3×7 convolution; Concat represents concatenation operation; Conv1×1 represents 1×1 convolution; Output represents the shallow features of the output target.

[0065] like Figure 3 As shown, the specific process of step S2 is:

[0066] Step S21: Figure 8 The WIRW module shown in (a) further enhances the shallow features obtained in step S1. The WIRW (Wide Identical Residual Weighting) module is a wide residual weighting module. The structure of the WIRW module is as follows: Figure 8 As shown in (a), Figure 8 In (a), Conv1 represents 1×1 convolution, Relu represents Relu activation function, Conv3 represents 3×3 convolution, and SA represents Shuffle Attention module. The enhancement process of WIRW module is as follows: first, two 1×1 convolution kernels are used to amplify the features; at the same time, Relu function is used between convolution layers for nonlinear processing; and 3×3 convolution kernel is used for feature extraction. After extraction, SA module is used for grouping for better feature learning, and finally, The residual structure prevents over-complexity of parameters.

[0067] Step S22: Figure 8 The DF (DeepFuse) module shown in (b) fuses the enhanced results with the shallow features to obtain fused features. DeepFuse stands for deep fusion. The structure of the DF module is as follows: Figure 8 (b) Figure 8 In (b), Concat represents concatenation operation, Conv1 represents 1×1 convolution, and Sigmoid represents Sigmoid activation function. The fusion process of DF module is shown in Figure 5 As shown:

[0068] First, the shallow features and the enhanced features obtained by the WIRW module are concatenated to obtain the concatenated features. Then, the concatenated features are divided into two branches. One branch amplifies the concatenated features through a 1×1 convolution kernel, performs nonlinear processing through Sigmoid, and multiplies the concatenated features with the enhanced results. The other branch amplifies the concatenated features through a 1×1 convolution kernel. The residual operation is performed on the two results to obtain the final fusion feature. Finally, use The residual structure enables features to be preserved after processing.

[0069] Step S23: Pass Figure 7 (b) The residual structure adds the fusion features obtained in step S22 and the shallow features obtained in step S1 to obtain the final features.

[0070] like Figure 4 As shown, the specific operations of step S21 are as follows:

[0071] S211: First, two 1×1 convolution kernels are used to amplify the features;

[0072] S212: Use ReLU function for nonlinear processing between convolutional layers;

[0073] S213: Use 3×3 convolution kernel to extract features;

[0074] S214: Use after extraction Fig. 9 The SA module shown in the figure is grouped for better feature learning. The learning process is as follows: first, the input features are grouped by the number of channels, and then each group is divided into two small groups. The channel attention mechanism and spatial attention mechanism are used to assign different weights to the features, and then the two small groups are concat spliced. Finally, the channels of the features processed by all groups are rearranged and the SA (Shuffle Attention) module is output.

[0075] S215: Use Figure 8 The ⊕ residual structure shown in (a) prevents the parameters from being overly complicated.

[0076] After the above steps, the Lidar point cloud data can be denoised. Fig.10 The data collected by the Lidar detector is sent to the network, and features are extracted through steps S1 and S2; finally, the data is classified through step S3, and the lidar data classified as noise is removed. The denoising effect is shown in the figure below. Fig.11 As shown in the figure, in the point cloud data obtained with the vehicle radar sensor as the center, the noise reduction Fig.11 Compared with the noise reduction Fig.10 The irregular and dense snow spots around the vehicle are removed effectively.

[0077] The method of this embodiment can understand the underlying data structure and summarize its characteristics under different distances and clutter distributions. In addition, the method can also combine the intensity information of the point cloud to identify objects at different distances and directions, and can more accurately identify noise points in bad weather conditions.

[0078] The above is a specific implementation method of a lightweight point cloud denoising method based on a convolutional neural network provided in an embodiment of the present application. Based on this, an embodiment of the present application also provides a lightweight point cloud denoising device based on a convolutional neural network.

[0079] like Figure 6 As shown, the present invention provides a lightweight point cloud denoising device based on a convolutional neural network, comprising:

[0080] Shallow extraction module: used for shallow feature extraction using LilaBlock;

[0081] Extraction and fusion module: used to use the FEAR module for deep feature extraction and feature fusion: Figure 7 The FEAR (feature enhance and resnet) module shown in (b) performs deep feature extraction on the shallow features obtained in step S1.

[0082] Denoising module: It is used to perform Dropout operation on the final features to prevent overfitting, and then use convolution to reduce the number of feature channels to the number of point cloud categories, thereby obtaining the classification results of the noise points and removing the noise points.

[0083] In the figure: Distance represents distance, Intensity represents intensity. Concat represents concatenation operation; FEAR represents data enhancement and residual network module, Dropout represents pruning operation, Conv1×1 represents 1×1 convolution, and Output represents output. The working process of a lightweight point cloud denoising device based on convolutional neural network is as follows:

[0084] The point-by-point distance and detection intensity of the obtained Lidar point cloud data containing extreme weather conditions are concat- ed and then shallow feature extraction is performed through the LiLaBlock module.

[0085] The FEAR (feature enhance and resnet) module is used to further extract the shallow features obtained and fuse them with the shallow features to obtain the fused features.

[0086] The obtained fusion features are subjected to Dropout operation to prevent overfitting, and then convolution is used to reduce the number of feature channels to the number of point cloud categories, thereby obtaining the classification results of the noise points and removing the noise points.

[0087] The present application also provides an electronic device, which may include a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. The electronic device may also include various network interfaces, power supplies and other components.

[0088] The present application also provides a storage medium, in which computer executable instructions are stored, and when the computer executable instructions are loaded and executed by the processor, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0089] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A lightweight point cloud denoising method based on convolutional neural network, comprising the following steps: Step S1: Use the LilaBlock module to perform shallow feature extraction; the LilaBlock module is a LiDAR-based semantic labeling block module, that is, a radar semantic segmentation module; Step S2: Use the FEAR module to perform deep feature extraction and feature fusion at the same time; the FEAR module is a feature enhancement and resnet module, i.e., a feature enhancement and residual network module; Step S3: Perform a Dropout operation on the final feature obtained. The Dropout operation refers to a pruning operation to prevent overfitting. Then, convolution is used to reduce the number of feature channels to the number of point cloud categories, thereby obtaining a classification result of the noise points and removing the noise points. The specific process of step S1 is: S11: Input the distance and intensity of each point in the Lidar point cloud data point by point; S12: perform concat on the two pieces of information; concat is a concatenation operation; S13: Use four convolution kernels of different scales on the combined data to extract corresponding features, increase the feature dimension, fuse the four extracted features, and then use a 1×1 convolution kernel to further extract the fused features; The specific process of step S2 is: Step S21: further enhancing the shallow features obtained in step S1; Step S22: Fusing the enhancement result with the shallow feature to obtain a fused feature; Step S23: Add the fusion feature obtained in step S22 and the shallow feature obtained in step S1 to obtain the final feature; The specific operations of step S21 are as follows: S211: feature amplification through two 1×1 convolution kernels; S212: Use ReLU function between convolutional layers for nonlinear processing; S213: Use 3×3 convolution kernel for feature extraction; S214: After extraction, use SA module to group and perform feature learning; SA module stands for Shuffle Attention module; S215: Use ⊕ residual structure to prevent over-complexity of parameters; The specific operations of step S22 are as follows: First, the shallow features and the enhanced features obtained by the WIRW module are concat-concatenated to obtain the concatenated features. The WIRW module is the Wide Identical Residual Weighting module. Then it is divided into two branches. One branch amplifies the concatenated features through a 1×1 convolution kernel and performs nonlinear processing through Sigmoid. Sigmoid represents the Sigmoid activation function and multiplies the enhanced results. The other branch only amplifies the concatenated features through a 1×1 convolution kernel. The two results are subjected to residual operation to obtain the final fused features. The residual structure is used to retain the features after processing.

2. A lightweight point cloud denoising device based on convolutional neural network, characterized in that: include: Shallow extraction module: used to perform shallow feature extraction using the LilaBlock module; the LilaBlock module is a LiDAR-based semantic labeling block module, i.e., a radar semantic segmentation module; The specific process of using the LilaBlock module for shallow feature extraction is: S11: Input the distance and intensity of each point in the Lidar point cloud data point by point; S12: perform concat on the two pieces of information; concat is a concatenation operation; S13: Use four convolution kernels of different scales on the combined data to extract corresponding features, increase the feature dimension, fuse the four extracted features, and then use a 1×1 convolution kernel to further extract the fused features; Extraction and fusion module: used to use the FEAR module for deep feature extraction and feature fusion; the FEAR module is a feature enhancement and resnet module, i.e. a feature enhancement and residual network module; The specific process of using the FEAR module for deep feature extraction and feature fusion is as follows: Step S21: Further enhance the shallow features obtained by shallow feature extraction using the LilaBlock module. The specific operations are as follows: S211: feature amplification through two 1×1 convolution kernels; S212: Use ReLU function between convolutional layers for nonlinear processing; S213: Use 3×3 convolution kernel for feature extraction; S214: After extraction, use SA module to group and perform feature learning; SA module stands for Shuffle Attention module; S215: Use ⊕ residual structure to prevent over-complexity of parameters; Step S22: Fuse the enhancement result with the shallow feature to obtain the fused feature. The specific operation is as follows: First, the shallow features and the enhanced features obtained by the WIRW module are concatenated through the Concat operation to obtain the concatenated features; the WIRW module is the Wide Identical Residual Weighting module, that is, the wide residual weighting module; then it is divided into two branches, one of which amplifies the concatenated features through a 1×1 convolution kernel and performs nonlinear processing through Sigmoid, where Sigmoid represents the Sigmoid activation function, and multiplies the enhanced results; the other branch only amplifies the concatenated features through a 1×1 convolution kernel; the two results are subjected to residual operations to obtain the final fused features; the residual structure is used to retain the features after processing; Step S23: Add the fusion feature obtained in step S22 and the shallow feature obtained by shallow feature extraction using the LilaBlock module to obtain the final feature; Denoising module: used to perform Dropout operation on the final features obtained. Dropout means pruning operation to prevent overfitting, and then use convolution to reduce the number of feature channels to the number of point cloud categories, so as to obtain the classification results of noise points and remove the noise points.

3. The lightweight point cloud denoising device based on convolutional neural network according to claim 2, characterized in that: The FEAR module includes: WIRW module: used to further enhance the shallow features obtained; DF module: used to fuse the enhancement results with shallow features to obtain fused features; DF module is DeepFuse module, i.e. deep fusion module; ⊕Residual structure: used to add the obtained fusion features and the obtained shallow features to obtain the final features.

4. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of a lightweight point cloud denoising method based on a convolutional neural network as claimed in claim 1 are implemented.

5. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of a lightweight point cloud denoising method based on a convolutional neural network as described in claim 1 are implemented.

Images