Automatic driving obstacle recognition method and system based on deep learning

Abstract

The invention provides an automatic driving obstacle recognition method and system based on deep learning, and relates to the technical field of automatic driving, and the method comprises the steps: collecting sample image data and sample point cloud data; performing feature extraction on the marked point cloud and the marked image by using a 3DCNN network, and performing training according to an extraction result to obtain a classifier and a prediction head; enhancing the unmarked image and the unmarked point cloud; performing feature extraction on the enhanced image and the enhanced point cloud to obtain a multi-scale image feature and a multi-scale point cloud feature; dynamically adjusting the weights of the multi-scale image features and the multi-scale point cloud features based on a feature fusion method of an attention mechanism to obtain fusion features; and inputting the fused features into the trained classifier and the prediction head to obtain an obstacle recognition result. According to the method, the accuracy, robustness and generalization ability of obstacle recognition are remarkably improved, the marking cost is reduced, and the high requirements of an automatic driving scene for real-time performance and reliability are met.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving obstacle recognition method and system based on deep learning. Background Art

[0002] Existing obstacle recognition methods for autonomous driving usually rely on multi-sensor data fusion (such as lidar point cloud and camera images), extract features through deep learning networks and perform obstacle detection. However, in practical applications, complex environments (such as lighting changes, rainy and foggy weather) can significantly reduce the quality of image data, and lidar point cloud data can also affect detection performance under sparse or noisy conditions.

[0003] In addition, the existing methods are not able to adapt to environmental changes during feature extraction and fusion, which makes it difficult for the robustness and accuracy of obstacle detection to meet the needs of autonomous driving. Therefore, there is an urgent need for an obstacle recognition method that can enhance the ability to adapt to complex environments. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an autonomous driving obstacle recognition method and system based on deep learning. Through technical means such as multimodal data fusion, attention mechanism, multi-scale feature extraction and data enhancement, the accuracy, robustness and generalization ability of obstacle recognition are significantly improved, while the labeling cost is reduced, meeting the high requirements of autonomous driving scenarios for real-time and reliability.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An automatic driving obstacle recognition method based on deep learning, comprising:

[0007] Collecting sample image data and sample point cloud data; the sample image data includes annotated images and unannotated images; the sample point cloud data includes annotated point clouds and unannotated point clouds;

[0008] Using a preset 3D CNN network to extract features from the annotated point cloud and the annotated image, and training based on the extraction results to obtain a classifier and a prediction head;

[0009] Performing denoising, color difference balancing and image enhancement on the unlabeled image to obtain an enhanced image;

[0010] Performing data enhancement on the unlabeled point cloud to obtain an enhanced point cloud;

[0011] Performing feature extraction on the enhanced image and the enhanced point cloud respectively to obtain multi-scale image features and multi-scale point cloud features;

[0012] A feature fusion method based on the attention mechanism dynamically adjusts the weights of multi-scale image features and multi-scale point cloud features to obtain fused features;

[0013] The fused features are input into the trained classifier and prediction head to obtain the obstacle recognition results.

[0014] Preferably, after extracting features from the annotated point cloud and the annotated image using a preset 3D CNN network and training based on the extraction results to obtain a classifier and a prediction head, the method further includes:

[0015] Normalizing the unlabeled image and the unlabeled point cloud to obtain a normalized unlabeled image and an unlabeled point cloud;

[0016] The normalized unlabeled image and unlabeled point cloud are temporally aligned to obtain the aligned unlabeled image and unlabeled point cloud.

[0017] Preferably, denoising, color difference balancing and image enhancement are performed on the unlabeled image to obtain an enhanced image, including:

[0018] Performing filtering on the unlabeled image to obtain a filtered image;

[0019] Constructing an image enhancement window, and using the image enhancement window to traverse the filtered image for denoising, to obtain a denoised image;

[0020] Perform histogram equalization on the denoised image to obtain a preliminarily enhanced image;

[0021] Performing color difference balance on the initially enhanced image to obtain a color difference balanced image;

[0022] Converting the color-balanced image into an HSV image;

[0023] The HSV image is enhanced according to a preset enhancement coefficient to obtain the final enhanced image.

[0024] Preferably, filtering the unlabeled image to obtain a filtered image includes:

[0025] Performing wavelet decomposition on the unlabeled image at multiple scales to obtain corresponding wavelet coefficients;

[0026] A filtering threshold is constructed according to the size and decomposition scale of the unlabeled image; the formula of the filtering threshold is: Among them, λ j represents the filtering threshold at the jth decomposition scale, σ j represents the standard deviation of the noise at the jth decomposition scale, N jrepresents the signal length at the jth decomposition scale, where j represents the decomposition scale;

[0027] The image filtering model is constructed using the filtering threshold; the formula of the image filtering model is: Wherein, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, ω j,k represents the kth wavelet coefficient at the jth decomposition scale, represents the wavelet coefficients after filtering;

[0028] The unlabeled image is filtered using the image filtering model to obtain the filtered image.

[0029] Preferably, constructing an image enhancement window, and using the image enhancement window to traverse the filtered image for denoising to obtain a denoised image, comprises:

[0030] Use the image recognition model to detect the filtered image and obtain the proportion value under each image enhancement window;

[0031] When the proportion value in the image enhancement window is greater than a preset threshold, median denoising is performed on the image in the corresponding image enhancement window to obtain an image after median denoising;

[0032] The image enhancement window is slid until the entire filtered image is traversed to obtain the denoised image.

[0033] Preferably, performing color balance on the preliminarily enhanced image to obtain a color-balanced image includes:

[0034] Calculate the pixel mean of the three RGB channels of the initially enhanced image; wherein the pixel mean calculation formula is: Where D represents the pixel mean, represents the pixel mean of the initially enhanced image in the red channel, represents the pixel mean value of the green channel of the initially enhanced image, Represents the pixel mean of the initially enhanced image in the blue channel;

[0035] The balance coefficient of each color channel is calculated according to the pixel mean; the formula of the balance coefficient is: Among them, K R Indicates the balance coefficient of the red channel, K G Indicates the balance coefficient of the green channel, K Brepresents the balance coefficient of the blue channel, R′ represents the pixel value of the red channel after balance, G′ represents the pixel value of the green channel after balance, B′ represents the pixel value of the blue channel after balance, R represents the pixel value of the preliminary enhanced image in the red channel, G represents the pixel value of the preliminary enhanced image in the green channel, and B represents the pixel value of the preliminary enhanced image in the blue channel;

[0036] The color difference of the initially enhanced image is balanced by using the balance coefficient to obtain a color difference balanced image.

[0037] Preferably, converting the color-balanced image into an HSV image comprises:

[0038] Normalizing each color channel of the image after color difference balance to obtain a normalized result;

[0039] Select the extreme value in the normalized result, and construct the color space conversion formula based on the normalized result and its extreme value; the calculation formula of the extreme value is: Among them, C max Indicates the maximum value in the normalized result, C min represents the minimum value in the normalized result; the color space conversion formula is: Among them, H represents the hue component, S represents the saturation component, and V represents the lightness component;

[0040] The color space conversion formula is used to convert the color difference balanced image into an HSV image.

[0041] Preferably, the HSV image is enhanced according to a preset enhancement coefficient to obtain the final enhanced image, including:

[0042] Calculate the mean brightness and saturation of each pixel in the HSV image;

[0043] Calculate the brightness variance and saturation variance of each pixel point according to the brightness mean and saturation mean of each pixel point;

[0044] An enhancement coefficient is constructed according to the brightness variance and the saturation variance; the formula of the enhancement coefficient is: Where W represents the number of pixels on the HSV image, υ(i,j) represents the brightness value at point (i,j), represents the mean brightness of all pixels in the HSV image, s(i,j) represents the saturation at point (i,j), and s W (x,y) represents the mean saturation of all pixels in the HSV image, σ υ (x,y) represents the brightness variance of all pixels in the HSV image, σ s(x,y) represents the saturation variance of all pixels in the HSV image;

[0045] constructing an enhancement model using the enhancement coefficient;

[0046] The HSV image is enhanced using the enhancement model.

[0047] Preferably, the formula of the fusion feature is:

[0048] F fusion =α·F image +β·F point

[0049] Among them, the weight coefficients α and β are dynamically calculated through the attention mechanism; the calculation formula of the weight coefficient α is: α=σ(W weight，image ·GAP(F image )+b image ); The calculation formula of weight coefficient β is: β=σ(W weight，point ·GAP(F point )+b point ), where F image Represents the multi-scale image features extracted from the image branch. The multi-scale image features are high-dimensional tensors with dimensions (C image ,H image ,W image ), where C image Indicates the number of channels, H image and W image Respectively represent the height and width of the feature map; F point Represents the multi-scale point cloud features extracted from the point cloud branch. The multi-scale point cloud features are high-dimensional tensors with dimensions (C point ,N point ), where C point Indicates the number of channels, N point Indicates the number of points of point cloud features; GAP() represents the global average pooling operation; b image and b point are all bias terms; σ is the Sigmoid activation function; W weight，image and W weight，point Respectively represent the weight matrices corresponding to image features and point cloud features; W weight，image The dimension is (1, C image ), W weight，point The dimension is (1, C point ).

[0050] An automatic driving obstacle recognition system based on deep learning, comprising:

[0051] A data acquisition module, used to acquire sample image data and sample point cloud data; the sample image data includes annotated images and unannotated images; the sample point cloud data includes annotated point clouds and unannotated point clouds;

[0052] A labeling feature extraction module, used to extract features from the labeling point cloud and the labeling image using a preset 3D CNN network, and to perform training based on the extraction results to obtain a classifier and a prediction head;

[0053] An image processing module, used for performing denoising, color difference balancing and image enhancement on the unlabeled image to obtain an enhanced image;

[0054] A point cloud processing module, used for performing data enhancement on the unlabeled point cloud to obtain an enhanced point cloud;

[0055] A multi-source feature extraction module, used to extract features from the enhanced image and the enhanced point cloud respectively to obtain multi-scale image features and multi-scale point cloud features;

[0056] The feature fusion module is used for the feature fusion method based on the attention mechanism to dynamically adjust the weights of multi-scale image features and multi-scale point cloud features to obtain fused features;

[0057] The recognition module is used to input the fused features into the trained classifier and prediction head to obtain the obstacle recognition results.

[0058] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0059] The present invention provides an automatic driving obstacle recognition method and system based on deep learning, the method comprising: collecting sample image data and sample point cloud data; the sample image data comprises annotated images and unannotated images; the sample point cloud data comprises annotated point clouds and unannotated point clouds; using a preset 3DCNN network to perform feature extraction on the annotated point cloud and the annotated image, and training according to the extraction results to obtain a classifier and a prediction head; performing denoising, color difference balance and image enhancement on the unannotated image to obtain an enhanced image; performing data enhancement on the unannotated point cloud to obtain an enhanced point cloud; performing feature extraction on the enhanced image and the enhanced point cloud respectively to obtain multi-scale image features and multi-scale point cloud features; a feature fusion method based on an attention mechanism to dynamically adjust the weights of multi-scale image features and multi-scale point cloud features to obtain fused features; inputting the fused features into the trained classifier and the prediction head to obtain an obstacle recognition result. The present invention significantly improves the accuracy, robustness and generalization ability of obstacle recognition through technical means such as multimodal data fusion, attention mechanism, multi-scale feature extraction and data enhancement, while reducing the labeling cost and meeting the high requirements of autonomous driving scenarios for real-time and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The purpose of the present invention is to provide an autonomous driving obstacle recognition method and system based on deep learning. Through technical means such as multimodal data fusion, attention mechanism, multi-scale feature extraction and data enhancement, the accuracy, robustness and generalization ability of obstacle recognition are significantly improved, while the annotation cost is reduced, meeting the high requirements of autonomous driving scenarios for real-time and reliability.

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides an automatic driving obstacle recognition method based on deep learning, comprising:

[0067] Step 100: Collect sample image data and sample point cloud data; the sample image data includes annotated images and unannotated images; the sample point cloud data includes annotated point clouds and unannotated point clouds;

[0068] Step 200: extract features from the annotated point cloud and the annotated image using a preset 3D CNN network, and perform training based on the extraction results to obtain a classifier and a prediction head;

[0069] Step 300: performing denoising, color difference balancing and image enhancement on the unlabeled image to obtain an enhanced image;

[0070] Step 400: performing data enhancement on the unlabeled point cloud to obtain an enhanced point cloud;

[0071] Step 500: extracting features from the enhanced image and enhanced point cloud respectively to obtain multi-scale image features;

[0072] Step 600: A feature fusion method based on an attention mechanism dynamically adjusts the weights of multi-scale image features and multi-scale point cloud features to obtain fused features;

[0073] Step 700: Input the fused features into the trained classifier and prediction head to obtain obstacle recognition results.

[0074] Specifically, step 100 of this embodiment includes:

[0075] In the autonomous driving scenario, on-board sensors (such as cameras and lidar) are used to collect sample image data and sample point cloud data. Image data is obtained through cameras and contains two-dimensional visual information of the vehicle's surroundings; point cloud data is obtained through lidar and contains geometric information in three-dimensional space. During the collection process, ensure that the data covers a variety of driving scenarios (such as urban roads, highways, and rural roads) and different weather and lighting conditions (such as sunny days, rainy days, and nights). The collected data is classified into labeled data and unlabeled data based on whether it has manual annotation information. The labeled data contains the manually labeled obstacle category and location information, while the unlabeled data is only the original collected data.

[0076] The generation of labeled data is completed through professional labeling tools and a manual labeling team. For image data, use labeling tools to select or segment obstacles (such as vehicles, pedestrians, traffic signs, etc.) and annotate category information; for point cloud data, use 3D labeling tools to select or segment obstacles in the point cloud and annotate their 3D bounding boxes and category information. After labeling is completed, the labeled data is quality checked to ensure the accuracy and consistency of the labeling. The labeled data will be used to supervise the training of the learning model to provide high-quality feature and label pairs.

[0077] Unlabeled data does not require manual labeling, and the raw data is obtained directly from the acquisition device. In order to improve the quality and usability of unlabeled data, it needs to be preprocessed and enhanced. For unlabeled image data, denoising, color balance and image enhancement operations are performed to improve the clarity and contrast of the image; for unlabeled point cloud data, data enhancement operations (such as random rotation, translation, scaling or adding noise) are performed to increase the diversity and robustness of point cloud data. The preprocessed and enhanced unlabeled data will be used for the training of unsupervised learning or semi-supervised learning models to further improve the generalization ability of the model.

[0078] Optionally, the specific process of step 200 of this embodiment is as follows:

[0079] 1. Design and initialization of 3D CNN network. First, a preset 3D CNN network is designed to process annotated point cloud and annotated image data simultaneously. The network contains two branches: one for processing point cloud data and the other for processing image data. The point cloud branch uses a 3D CNN structure based on voxelization or point cloud convolution (such as VoxelNet or PointNet++) to extract the spatial geometric features of the point cloud; the image branch uses a classic 2D CNN network (such as ResNet or EfficientNet) to extract the texture and semantic features of the image. In order to fuse the features of the two modalities, a feature fusion module (such as a fusion method based on an attention mechanism or feature splicing) is designed in the network. When the network is initialized, the image branch is initialized using a pre-trained model (such as ImageNet pre-trained weights), while the point cloud branch is randomly initialized according to task requirements or uses the weights of an existing point cloud model.

[0080] 2. Feature extraction and training data preparation: Input the labeled point cloud and labeled image data into the point cloud branch and image branch of the 3DCNN network respectively to extract multi-scale point cloud features and image features. Through the feature fusion module, the features of the two modalities are fused to generate a joint feature representation. Subsequently, the joint features are input into the classifier and the prediction head. The classifier is used to predict the category of the obstacle, and the prediction head is used to regress the position and size of the obstacle (such as a bounding box or a 3D bounding box). During the training process, the category labels and position labels in the labeled data are used as supervisory signals to construct a loss function (such as cross entropy loss for classification and smooth L1 loss for regression), and the network is trained end-to-end.

[0081] 3. Optimization and output of classifier and prediction head. During the training process, the feature extraction results of the labeled point cloud and labeled image data are used to optimize the parameters of the classifier and prediction head through the back propagation algorithm. The classifier processes the fused features through a multi-layer fully connected network and outputs the category probability distribution of each obstacle; the prediction head processes the fused features through a regression network and outputs information such as the three-dimensional position, size and direction of the obstacle. After training, the classifier and prediction head can efficiently identify and locate obstacles on the input point cloud and image data, and finally form a complete obstacle detection model for real-time obstacle recognition tasks in autonomous driving scenarios.

[0082] Preferably, after extracting features from the annotated point cloud and the annotated image using a preset 3D CNN network and training based on the extraction results to obtain a classifier and a prediction head, the method further includes:

[0083] Normalizing the unlabeled image and the unlabeled point cloud to obtain a normalized unlabeled image and an unlabeled point cloud;

[0084] The normalized unlabeled image and unlabeled point cloud are temporally aligned to obtain the aligned unlabeled image and unlabeled point cloud.

[0085] Specifically, after normalization, this embodiment performs time alignment on the unlabeled image and the unlabeled point cloud to ensure that the image and point cloud data at the same time can be correctly paired. The process of time alignment generally involves the following steps: First, record the timestamp of each frame of image and point cloud data to ensure that their time information is available. Next, use an interpolation method (such as linear interpolation or cubic interpolation) to align the timestamps to ensure that there are corresponding images and point cloud data at each specific time point. For example, if the acquisition frequency of the image is higher than the acquisition frequency of the point cloud, the closest point cloud data can be found according to the timestamp, and the interpolation method can be used to generate the point cloud data at the corresponding time point. In this way, the final aligned unlabeled image and unlabeled point cloud will be able to remain consistent in time, providing an accurate data basis for subsequent feature extraction and model training.

[0086] Furthermore, in step 400 of this embodiment, the first step of data enhancement for the unlabeled point cloud is to select a suitable enhancement method to improve the diversity and robustness of the point cloud data. Common point cloud data enhancement techniques include random rotation, translation, scaling, adding noise, and randomly discarding points. Random rotation can increase the directional change of the point cloud by rotating the point cloud in three-dimensional space; translation simulates different observation angles by adding random offsets to the coordinates of each point. The scaling operation simulates the observation effects at different distances by multiplying the coordinates of the point cloud. In addition, adding Gaussian noise can simulate the measurement error of the sensor, while randomly discarding points can simulate the sparsity of point cloud data and enhance the model's adaptability to incomplete data.

[0087] When implementing data enhancement, the original unlabeled point cloud is first preprocessed to ensure that the format and coordinate range of the point cloud data are suitable for subsequent enhancement operations. Then, the selected enhancement methods are applied in sequence. For example, when performing random rotation, a random rotation matrix can be generated and applied to each point of the point cloud; when translating, a random translation vector is generated and added to the coordinates of each point. For scaling, a random scaling factor can be selected and applied to the coordinates of all points. When adding noise, Gaussian noise that matches the size of the point cloud can be generated and added to the coordinates of each point. Finally, the enhanced point cloud will form a new enhanced point cloud dataset for subsequent model training.

[0088] After completing the data enhancement, this embodiment needs to evaluate the effect of the enhanced point cloud to ensure that the enhancement operation does not introduce too much noise or distortion. Visualization tools can be used to compare the point cloud before and after enhancement to check the distribution, density, and shape changes of the point cloud to ensure that the enhanced point cloud still retains the characteristics and structure of the original data. In addition, the enhanced point cloud will be used to train the deep learning model to improve the generalization ability and robustness of the model. By introducing diverse point cloud data, the model can better adapt to obstacle recognition tasks in different scenarios and conditions, thereby improving the overall performance and safety of the autonomous driving system.

[0089] Preferably, denoising, color difference balancing and image enhancement are performed on the unlabeled image to obtain an enhanced image, including:

[0090] Performing filtering on the unlabeled image to obtain a filtered image;

[0091] Constructing an image enhancement window, and using the image enhancement window to traverse the filtered image for denoising, to obtain a denoised image;

[0092] Perform histogram equalization on the denoised image to obtain a preliminarily enhanced image;

[0093] Performing color difference balance on the initially enhanced image to obtain a color difference balanced image;

[0094] Converting the color-balanced image into an HSV image;

[0095] The HSV image is enhanced according to a preset enhancement coefficient to obtain the final enhanced image.

[0096] Preferably, filtering the unlabeled image to obtain a filtered image includes:

[0097] Performing wavelet decomposition on the unlabeled image at multiple scales to obtain corresponding wavelet coefficients;

[0098] A filtering threshold is constructed according to the size and decomposition scale of the unlabeled image; the formula of the filtering threshold is: Among them, λ j represents the filtering threshold at the jth decomposition scale, σ j represents the standard deviation of the noise at the jth decomposition scale, N j represents the signal length at the jth decomposition scale, where j represents the decomposition scale;

[0099] The image filtering model is constructed using the filtering threshold; the formula of the image filtering model is: Wherein, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, ω j,k represents the kth wavelet coefficient at the jth decomposition scale, represents the wavelet coefficients after filtering;

[0100] The unlabeled image is filtered using the image filtering model to obtain the filtered image.

[0101] Furthermore, it can be seen from the above filter threshold formula that as the decomposition scale increases, the threshold of each layer decreases accordingly, which is in line with the law of wavelet denoising, and thus can improve the performance of wavelet denoising. This embodiment uses an image filtering model to perform self-adjusting threshold transformation on the wavelet coefficients at each scale, which can highlight the detailed features of the image at different decomposition levels, suppress the noise of the image, and enhance the layering of the image.

[0102] Preferably, constructing an image enhancement window, and using the image enhancement window to traverse the filtered image for denoising to obtain a denoised image, comprises:

[0103] Use the image recognition model to detect the filtered image and obtain the proportion value under each image enhancement window;

[0104] When the proportion value in the image enhancement window is greater than a preset threshold, median denoising is performed on the image in the corresponding image enhancement window to obtain an image after median denoising;

[0105] The image enhancement window is slid until the entire filtered image is traversed to obtain the denoised image.

[0106] Preferably, performing color balance on the preliminarily enhanced image to obtain a color-balanced image includes:

[0107] Calculate the pixel mean of the three RGB channels of the initially enhanced image; wherein the pixel mean calculation formula is: Where D represents the pixel mean, represents the pixel mean of the initially enhanced image in the red channel, represents the pixel mean value of the green channel of the initially enhanced image, Represents the pixel mean of the initially enhanced image in the blue channel;

[0108] The balance coefficient of each color channel is calculated according to the pixel mean; the formula of the balance coefficient is: Among them, K R Indicates the balance coefficient of the red channel, K G Indicates the balance coefficient of the green channel, K B represents the balance coefficient of the blue channel, R′ represents the pixel value of the red channel after balance, G′ represents the pixel value of the green channel after balance, B′ represents the pixel value of the blue channel after balance, R represents the pixel value of the preliminary enhanced image in the red channel, G represents the pixel value of the preliminary enhanced image in the green channel, and B represents the pixel value of the preliminary enhanced image in the blue channel;

[0109] The color difference of the initially enhanced image is balanced by using the balance coefficient to obtain a color difference balanced image.

[0110] Optionally, in the state of automatic driving, when the external lighting conditions change, the surface color of the object remains unchanged, but the color of the object captured by the imaging device will deviate with the change of lighting, gradually deviating from the true color of the object. This embodiment uses the color difference balance algorithm to make up for this defect of the imaging device and correct the color difference of the image.

[0111] Preferably, converting the color-balanced image into an HSV image comprises:

[0112] Each color channel of the image after color difference balance is normalized to obtain a normalized result; the extreme value in the normalized result is selected, and a color space conversion formula is constructed based on the normalized result and its extreme value; the calculation formula of the extreme value is: Among them, Cmax Indicates the maximum value in the normalized result, C min represents the minimum value in the normalized result; the color space conversion formula is: Among them, H represents the hue component, S represents the saturation component, and V represents the lightness component;

[0113] The color space conversion formula is used to convert the color difference balanced image into an HSV image.

[0114] Preferably, the HSV image is enhanced according to a preset enhancement coefficient to obtain the final enhanced image, including:

[0115] Calculate the mean brightness and saturation of each pixel in the HSV image;

[0116] Calculate the brightness variance and saturation variance of each pixel point according to the brightness mean and saturation mean of each pixel point;

[0117] An enhancement coefficient is constructed according to the brightness variance and the saturation variance; the formula of the enhancement coefficient is: Where W represents the number of pixels on the HSV image, υ(i,j) represents the brightness value at point (i,j), represents the mean brightness of all pixels in the HSV image, s(i,j) represents the saturation at point (i,j), and s W (x,y) represents the mean saturation of all pixels in the HSV image, σ υ (x,y) represents the brightness variance of all pixels in the HSV image, σ s (x,y) represents the saturation variance of all pixels in the HSV image;

[0118] constructing an enhancement model using the enhancement coefficient;

[0119] The HSV image is enhanced using the enhancement model.

[0120] Further, in step 500, the present embodiment designs and initializes a feature extraction network. For enhanced images, a convolutional neural network (CNN) is usually used as a feature extractor, such as ResNet, VGG or EfficientNet. These networks can effectively extract local and global features in images through multi-layer convolution and pooling operations. For enhanced point clouds, the feature extraction network can adopt a network structure specifically for point cloud data, such as PointNet, PointNet++ or VoxelNet. These networks can process irregular point cloud data and extract the geometric features and spatial relationships of point clouds. During initialization, weights pre-trained on large-scale datasets (such as ImageNet or ShapeNet) can be used to accelerate convergence and improve the effect of feature extraction.

[0121] In the feature extraction process, the enhanced image is first input into the designed image feature extraction network. The network extracts features layer by layer through a series of convolutional layers, activation functions (such as ReLU), and pooling layers. In each convolutional layer, the local features of the image are captured and gradually form higher-level abstract features. In order to obtain multi-scale features, the network can extract features at different levels, such as extracting detail features at shallow layers and semantic features at deep layers. Finally, after processing by global average pooling (GAP) or fully connected layers, the resulting feature vector will contain rich multi-scale information and can effectively represent the content and structure of the image.

[0122] Similar to the feature extraction of enhanced images, enhanced point cloud data is also input into a specially designed point cloud feature extraction network. For networks such as PointNet, feature learning is first performed on the point cloud, and the global feature representation of the point cloud is extracted by performing global feature aggregation on the features of each point. PointNet++ uses a hierarchical feature extraction method to combine local and global features to further enhance the expressiveness of features. After multi-layer feature extraction, the point cloud features will be integrated into a multi-scale feature vector that can effectively capture the geometric shape and spatial distribution information of the point cloud.

[0123] Furthermore, in step 600, the enhanced image data and the enhanced point cloud data are respectively passed through the feature extraction network to obtain multi-scale image features and multi-scale point cloud features. These feature vectors can be integrated through the feature fusion module for subsequent obstacle recognition tasks. Feature fusion can adopt simple splicing, weighted averaging or more complex attention mechanisms to ensure that image features and point cloud features can effectively complement each other to form a unified feature representation. In this way, the model can simultaneously utilize the texture information of the image and the geometric information of the point cloud to improve the accuracy and robustness of obstacle recognition.

[0124] Finally, the multi-scale image features and multi-scale point cloud features extracted in this embodiment will be used for subsequent obstacle recognition tasks. These features can be input into the classifier and prediction head to classify and locate obstacles. By combining the features of the image and point cloud, the model can more comprehensively understand the obstacles in the environment and improve the accuracy of recognition. In addition, the results of feature extraction can also be used to generate environmental maps, enhance the intelligence level of the autonomous driving system, and ensure safe driving in complex scenarios.

[0125] Preferably, the formula of the fusion feature is:

[0126] F fusion =α·F image +β·F point

[0127] Among them, the weight coefficients α and β are dynamically calculated through the attention mechanism; the calculation formula of the weight coefficient α is: α=σ(W weight，image ·GAP(F image )+b image ); The calculation formula of weight coefficient β is: β=σ(W weight，point ·GAP(F point )+b point ), where F image Represents the multi-scale image features extracted from the image branch. The multi-scale image features are high-dimensional tensors with dimensions (C image ,H image ,W image ), where C image Indicates the number of channels, H image and W image Respectively represent the height and width of the feature map; F point Represents the multi-scale point cloud features extracted from the point cloud branch. The multi-scale point cloud features are high-dimensional tensors with dimensions (C point ,N point ), where C point Indicates the number of channels, N point Indicates the number of points of point cloud features; GAP() represents the global average pooling operation; b image and b point are all bias terms; σ is the Sigmoid activation function; W weight，image and W weight，point Respectively represent the weight matrices corresponding to image features and point cloud features; W weight，image The dimension is (1, C image ), W weight，point The dimension is (1, C point ).

[0128] Specifically, GAP represents the global average pooling operation, which is used to reduce the dimension of the high-dimensional feature tensor to a global description vector. image After the GAP operation, a global vector of channel dimension is obtained, the dimension is (C image ). For F point After the GAP operation, a global vector of channel dimension is obtained, the dimension is (C point ).

[0129] Optionally, the weight matrix of this embodiment is used to perform linear transformation on the global description vector of the image features and point cloud features to improve the feature expression capability. The bias term is used to adjust the distribution of feature weights.

[0130] Furthermore, this embodiment dynamically calculates the weight coefficients α and β through the attention mechanism, adaptively adjusts the fusion ratio of image features and point cloud features according to the global information of the input features, and fully explores the complementarity of the two-modal data. The global description information of the features is extracted using global average pooling (GAP), avoiding the problem of information loss caused by relying solely on local features. The weight calculation only involves GAP and linear transformation, with low computational complexity, and is suitable for autonomous driving scenarios with high real-time requirements. In harsh environments (such as rainy and foggy weather or sparse point clouds), the attention mechanism can dynamically adjust the weights according to the quality of the features, making the system more robust to the lack or degradation of unimodal data.

[0131] Furthermore, in step 800 of this embodiment, after feature fusion is completed, the fused feature vector is input into the trained classifier and prediction head. The classifier is usually composed of multiple fully connected layers, which is responsible for processing the fused features to output the probability distribution of each obstacle category. In specific implementation, the fused features are first subjected to a series of linear transformations and activation functions (such as ReLU), and then the probability of each category is calculated through the softmax layer to obtain the classification result of each obstacle. At the same time, the prediction head is responsible for regressing the fused features and outputting the location information and size information of the obstacle. It is usually trained using smooth L1 loss or mean square error loss to ensure the accuracy of the regression results.

[0132] Specifically, the results of obstacle recognition include multiple key information, mainly the following: first, the category label, which indicates the type of obstacle recognized (such as pedestrians, vehicles, traffic signs, etc.), usually expressed in integer form; second, the confidence score, which indicates the confidence of the model in predicting each category, ranging from 0 to 1. The higher the value, the stronger the model's confidence in the category; finally, the location information, usually expressed in the form of a bounding box, which contains the location and size information of the obstacle in the image or point cloud, usually described by four coordinates (for 2D images) or six parameters (for 3D point clouds).

[0133] Optionally, after obtaining the obstacle recognition results, this embodiment needs to perform post-processing to improve the availability and accuracy of the results. The post-processing step includes non-maximum suppression (NMS), which is used to remove bounding boxes with high overlap to ensure that each obstacle is only recognized once. Ultimately, the recognition results will be integrated into the autonomous driving system to provide real-time environmental perception information to support decision-making and path planning. For example, this embodiment can adjust the driving path according to the recognized positions of pedestrians and vehicles to ensure safe driving. In addition, the recognition results can also be used to generate environmental maps to enhance the intelligence level of the autonomous driving system.

[0134] Corresponding to the above method, this embodiment further provides an autonomous driving obstacle recognition system based on deep learning, including:

[0135] A data acquisition module, used to acquire sample image data and sample point cloud data; the sample image data includes annotated images and unannotated images; the sample point cloud data includes annotated point clouds and unannotated point clouds;

[0136] A labeling feature extraction module, used to extract features from the labeling point cloud and the labeling image using a preset 3D CNN network, and to perform training based on the extraction results to obtain a classifier and a prediction head;

[0137] An image processing module, used for performing denoising, color difference balancing and image enhancement on the unlabeled image to obtain an enhanced image;

[0138] A point cloud processing module, used for performing data enhancement on the unlabeled point cloud to obtain an enhanced point cloud;

[0139] A multi-source feature extraction module, used to extract features from the enhanced image and the enhanced point cloud respectively to obtain multi-scale image features and multi-scale point cloud features;

[0140] The feature fusion module is used for the feature fusion method based on the attention mechanism to dynamically adjust the weights of multi-scale image features and multi-scale point cloud features to obtain fused features;

[0141] The recognition module is used to input the fused features into the trained classifier and prediction head to obtain the obstacle recognition results.

[0142] The beneficial effects of the present invention are as follows:

[0143] (1) This invention combines the multimodal characteristics of image data and point cloud data, uses the attention mechanism to dynamically adjust the weights of the two features, and fully exploits the complementary information of images and point clouds. Image data provides rich texture and color information, and point cloud data provides accurate spatial geometry information. The fusion of the two can significantly improve the accuracy and robustness of obstacle recognition.

[0144] (2) The attention mechanism of the present invention can dynamically adjust the weight distribution of image features and point cloud features according to the quality of the input features. In harsh environments (such as rainy and foggy weather, changing lighting, or sparse point clouds), the attention mechanism can adaptively reduce the weight of degraded modes and enhance the model's adaptability to missing or degraded unimodal data.

[0145] (3) The present invention performs multi-scale feature extraction on image and point cloud data, which can capture obstacle features at different scales and adapt to obstacles of different sizes and shapes. Multi-scale feature extraction enhances the model's ability to recognize diverse obstacles in complex scenes.

[0146] (4) The present invention performs denoising, color balance and image enhancement on unlabeled images to improve image quality and reduce the interference of noise on feature extraction. It also performs data enhancement on unlabeled point clouds to increase the diversity of point cloud data and improve the generalization ability of the model for different scenes.

[0147] (5) The present invention fully utilizes unlabeled data and reduces the dependence on labeled data by enhancing and extracting features from unlabeled images and point cloud data. It also reduces the cost of data labeling and improves the training efficiency of the model.

[0148] (6) The present invention uses a preset 3D CNN network to extract features from the labeled data and trains the classifier and prediction head to form an end-to-end obstacle recognition process. The joint training of the classifier and prediction head can more efficiently complete the obstacle classification and positioning tasks.

[0149] (7) Through multimodal fusion of images and point clouds, as well as data enhancement and multi-scale feature extraction, the model can adapt to complex driving scenarios, including dynamic obstacles, occlusions, lighting changes, etc. The present invention also improves the reliability and safety of the autonomous driving system in actual scenarios.

[0150] (8) The present invention adopts a lightweight attention mechanism and an efficient feature extraction network to ensure the real-time performance of obstacle recognition. It is suitable for tasks with high real-time requirements in autonomous driving scenarios.

[0151] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0152] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for autonomous driving obstacle recognition based on deep learning, characterized in that: include: Collecting sample image data and sample point cloud data; the sample image data includes annotated images and unannotated images; The sample point cloud data includes annotated point clouds and unannotated point clouds; Using a preset 3D CNN network to extract features from the annotated point cloud and the annotated image, and training based on the extraction results to obtain a classifier and a prediction head; Performing denoising, color difference balancing and image enhancement on the unlabeled image to obtain an enhanced image; Performing data enhancement on the unlabeled point cloud to obtain an enhanced point cloud; Performing feature extraction on the enhanced image and the enhanced point cloud respectively to obtain multi-scale image features and multi-scale point cloud features; A feature fusion method based on the attention mechanism dynamically adjusts the weights of multi-scale image features and multi-scale point cloud features to obtain fused features; The fused features are input into the trained classifier and prediction head to obtain the obstacle recognition results.

2. The deep learning-based autonomous driving obstacle recognition method according to claim 1, characterized in that: After extracting features from the annotated point cloud and the annotated image using a preset 3D CNN network and training based on the extraction results to obtain a classifier and a prediction head, the method further includes: Normalizing the unlabeled image and the unlabeled point cloud to obtain a normalized unlabeled image and an unlabeled point cloud; The normalized unlabeled image and unlabeled point cloud are temporally aligned to obtain the aligned unlabeled image and unlabeled point cloud.

3. The deep learning-based autonomous driving obstacle recognition method according to claim 1, characterized in that: Denoising, color difference balancing and image enhancement are performed on the unlabeled image to obtain an enhanced image, including: Performing filtering on the unlabeled image to obtain a filtered image; Constructing an image enhancement window, and using the image enhancement window to traverse the filtered image for denoising, to obtain a denoised image; Perform histogram equalization on the denoised image to obtain a preliminarily enhanced image; Performing color difference balance on the initially enhanced image to obtain a color difference balanced image; Converting the color-balanced image into an HSV image; The HSV image is enhanced according to a preset enhancement coefficient to obtain the final enhanced image.

4. The deep learning-based autonomous driving obstacle recognition method according to claim 3, characterized in that: The unlabeled image is filtered to obtain a filtered image, including: Performing wavelet decomposition on the unlabeled image at multiple scales to obtain corresponding wavelet coefficients; A filtering threshold is constructed according to the size and decomposition scale of the unlabeled image; the formula of the filtering threshold is: Among them, λ j represents the filtering threshold at the jth decomposition scale, σ j represents the standard deviation of the noise at the jth decomposition scale, N j represents the signal length at the jth decomposition scale, where j represents the decomposition scale; The image filtering model is constructed using the filtering threshold; the formula of the image filtering model is: Wherein, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, ω j,k represents the kth wavelet coefficient at the jth decomposition scale, represents the wavelet coefficients after filtering; The unlabeled image is filtered using the image filtering model to obtain the filtered image.

5. The deep learning-based autonomous driving obstacle recognition method according to claim 3, characterized in that: Constructing an image enhancement window, and using the image enhancement window to traverse the filtered image for denoising, to obtain a denoised image, including: Use the image recognition model to detect the filtered image and obtain the proportion value under each image enhancement window; When the proportion value in the image enhancement window is greater than a preset threshold, median denoising is performed on the image in the corresponding image enhancement window to obtain an image after median denoising; The image enhancement window is slid until the entire filtered image is traversed to obtain the denoised image.

6. The deep learning-based autonomous driving obstacle recognition method according to claim 3, characterized in that: Performing color difference balance on the initially enhanced image to obtain a color difference balanced image includes: Calculate the pixel mean of the three RGB channels of the initially enhanced image; wherein the pixel mean calculation formula is: Where D represents the pixel mean, represents the pixel mean of the initially enhanced image in the red channel, represents the pixel mean value of the green channel of the initially enhanced image, Represents the pixel mean of the initially enhanced image in the blue channel; The balance coefficient of each color channel is calculated according to the pixel mean; the formula of the balance coefficient is: Among them, K R Indicates the balance coefficient of the red channel, K G Indicates the balance coefficient of the green channel, K B represents the balance coefficient of the blue channel, R′ represents the pixel value of the red channel after balance, G′ represents the pixel value of the green channel after balance, B′ represents the pixel value of the blue channel after balance, R represents the pixel value of the preliminary enhanced image in the red channel, G represents the pixel value of the preliminary enhanced image in the green channel, and B represents the pixel value of the preliminary enhanced image in the blue channel; The color difference of the initially enhanced image is balanced by using the balance coefficient to obtain a color difference balanced image.

7. The deep learning-based autonomous driving obstacle recognition method according to claim 6, characterized in that: Converting the color-balanced image into an HSV image includes: Normalizing each color channel of the image after color difference balance to obtain a normalized result; Select the extreme value in the normalized result, and construct the color space conversion formula based on the normalized result and its extreme value; the calculation formula of the extreme value is: Among them, C max Indicates the maximum value in the normalized result, C min represents the minimum value in the normalized result; the color space conversion formula is: Among them, H represents the hue component, S represents the saturation component, V=C max V represents the brightness component; The color space conversion formula is used to convert the color difference balanced image into an HSV image.

8. The deep learning-based autonomous driving obstacle recognition method according to claim 6, characterized in that: The HSV image is enhanced according to a preset enhancement coefficient to obtain the final enhanced image, including: Calculate the mean brightness and saturation of each pixel in the HSV image; Calculate the brightness variance and saturation variance of each pixel point according to the brightness mean and saturation mean of each pixel point; An enhancement coefficient is constructed according to the brightness variance and the saturation variance; the formula of the enhancement coefficient is: Where W represents the number of pixels on the HSV image, υ(i,j) represents the brightness value at point (i,j), represents the mean brightness of all pixels in the HSV image, s(i,j) represents the saturation at point (i,j), and s W (x,y) represents the mean saturation of all pixels in the HSV image, σ υ (x,y) represents the brightness variance of all pixels in the HSV image, σ s (x,y) represents the saturation variance of all pixels in the HSV image; constructing an enhancement model using the enhancement coefficient; The HSV image is enhanced using the enhancement model.

9. The deep learning-based autonomous driving obstacle recognition method according to claim 1, characterized in that: The formula of the fusion feature is: F fusion =α·F image +β·F point Among them, the weight coefficients α and β are dynamically calculated through the attention mechanism; the calculation formula of the weight coefficient α is: α=σ(W weight，image ·GAP(F image )+b image ); The calculation formula of weight coefficient β is: β=σ(W weight，point ·GAP(F point )+b point ), where F image Represents the multi-scale image features extracted from the image branch. The multi-scale image features are high-dimensional tensors with dimensions (C image ,H image ,W image ), where C image Indicates the number of channels, H image and W image Respectively represent the height and width of the feature map; F point Represents the multi-scale point cloud features extracted from the point cloud branch. The multi-scale point cloud features are high-dimensional tensors with dimensions (C point ,N point ), where C point Indicates the number of channels, N point Indicates the number of points of point cloud features; GAP() represents the global average pooling operation; b image and b point are all bias terms; σ is the Sigmoid activation function; W weight，image and W weight，point Respectively represent the weight matrices corresponding to image features and point cloud features; W weight，image The dimension is (1, C image ), W weight，point The dimension is (1, C point ).

10. An autonomous driving obstacle recognition system based on deep learning, characterized in that: include: A data acquisition module, used to acquire sample image data and sample point cloud data; the sample image data includes annotated images and unannotated images; the sample point cloud data includes annotated point clouds and unannotated point clouds; A labeling feature extraction module, used to extract features from the labeling point cloud and the labeling image using a preset 3D CNN network, and to perform training based on the extraction results to obtain a classifier and a prediction head; An image processing module, used for performing denoising, color difference balancing and image enhancement on the unlabeled image to obtain an enhanced image; A point cloud processing module, used for performing data enhancement on the unlabeled point cloud to obtain an enhanced point cloud; A multi-source feature extraction module, used to extract features from the enhanced image and the enhanced point cloud respectively to obtain multi-scale image features and multi-scale point cloud features; The feature fusion module is used for the feature fusion method based on the attention mechanism to dynamically adjust the weights of multi-scale image features and multi-scale point cloud features to obtain fused features; The recognition module is used to input the fused features into the trained classifier and prediction head to obtain the obstacle recognition results.

Images