Environment perception enhancement method and system based on intelligent driving

Abstract

The invention relates to the technical field of intelligent driving, in particular to an environment perception enhancement method and system based on intelligent driving, and the method comprises the following steps: obtaining enhanced voxel features used for sensing the form of an obstacle and positioning in a driving environment in three-dimensional point cloud data from the three-dimensional point cloud data of the driving environment; enhanced pixel features used for sensing the texture and color of an obstacle in the driving environment in the visual image are obtained from the visual image data of the driving environment; and performing feature fusion on the enhanced voxel features and the enhanced pixel features to obtain multi-source environment features, and performing obstacle detection on the multi-source environment features to obtain target positions and target categories of obstacles in the driving environment. According to the invention, the effective features for sensing the environmental obstacles are extracted from the multi-source information, redundant features are eliminated, the efficiency of sensing the obstacles by using the multi-source information is improved, and the timeliness requirement of intelligent driving on environmental sensing is met.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to an environment perception enhancement method and system based on intelligent driving. Background Art

[0002] The most commonly used device for environmental perception in intelligent driving is the camera, which is closest to human eyes and can see colored signs and objects, understand fonts, and distinguish traffic lights. However, it also has many disadvantages. For example, vision is severely reduced at night or in bad weather, and it is not good at long-distance observation. The second is LiDAR. The most common one is on the roof, like a hat that rotates continuously. The principle is that by calculating the reflection time and wavelength of the laser beam, a 3D map of surrounding obstacles can be drawn, but the shortcoming is that it cannot recognize images and colors.

[0003] In the existing technology, lidar and cameras are combined in the environmental perception of intelligent driving to achieve environmental perception of multi-source information and improve the accuracy of environmental perception. However, using multi-source information for environmental perception will inevitably increase the redundancy of information and reduce the efficiency of environmental perception. An excessively long environmental perception process will cause road traffic congestion. Therefore, it is difficult to meet the timeliness requirements of intelligent driving for environmental perception. Summary of the invention

[0004] The purpose of the present invention is to provide an environmental perception enhancement method and system based on intelligent driving, so as to solve the technical problem in the prior art that using multi-source information for environmental perception will inevitably increase the redundancy of information, reduce the efficiency of environmental perception, and make it difficult to meet the timeliness requirements of intelligent driving for environmental perception.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] A method for enhancing environmental perception based on intelligent driving comprises the following steps:

[0007] The laser radar mounted on the vehicle collects radar scanning data of the driving environment, and constructs three-dimensional point cloud data of the driving environment based on the radar data of the driving environment;

[0008] Collect visual image data of the driving environment through the industrial camera mounted on the vehicle;

[0009] Extracting voxel features from the three-dimensional point cloud data of the driving environment, and identifying the importance of the voxel features, to obtain enhanced voxel features in the three-dimensional point cloud data for sensing the shape and positioning of obstacles in the driving environment;

[0010] Extract multi-scale features from visual image data of the driving environment, and use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment;

[0011] The enhanced voxel features and enhanced pixel features are feature-fused to obtain multi-source environment features, and obstacle detection is performed on the multi-source environment features using a classifier and a regressor to obtain target positions and target categories of obstacles in the driving environment.

[0012] As a preferred solution of the present invention, the method for constructing three-dimensional point cloud data of the driving environment includes:

[0013] Performing three-dimensional coordinate conversion on the laser distance measurement value, horizontal angle value, vertical angle value and horizontal offset angle of the laser channel in the radar scanning data to obtain three-dimensional coordinate data of each object in the driving environment;

[0014] The conversion formula of the three-dimensional coordinate data is:

[0015]

[0016] In the formula, x kj is the x-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, and y kj is the y-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, z kj is the z-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, r kj is the laser ranging value of the jth channel of the laser radar in the driving environment scanning the kth object, α kj is the horizontal angle value of the jth channel of the laser radar in the driving environment scanning the kth object, δ kj is the horizontal offset angle value of the kth object scanned by the jth channel of the laser radar in the driving environment, β kj The vertical angle value of the kth object scanned by the jth channel of the laser radar in the driving environment;

[0017] Among them, the three-dimensional coordinate data of the kth object in the driving environment is {(x kj ,y kj ,z kj )|j∈[1,n]}, where (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, and n is the number of channels of the laser radar; the three-dimensional coordinate data of each object is combined to form the three-dimensional point cloud data of the driving environment, and the three-dimensional point cloud data of the driving environment is: {(xkj ,y kj ,z kj )|j∈[1,n],k∈[1,m]}, where m is the total number of objects in the driving environment.

[0018] As a preferred solution of the present invention, the method for extracting voxel features includes:

[0019] The three-dimensional point cloud data of the driving environment is converted into voxel representation using the voxel grid (d, h, w), and the voxel representation of the driving environment is obtained as follows:

[0020]

[0021] In the formula, (D j ,H j ,W j ) is the voxel coordinate corresponding to the jth channel of the laser radar in the driving environment, D j is the x-axis coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, H j is the y-axis coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, W j is the z-axis dimension coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, d is the x-axis dimension of the voxel grid, h is the y-axis dimension of the voxel grid, w is the z-axis dimension of the voxel grid, D is the dimension of the voxel on the x-axis, H is the dimension of the voxel on the y-axis, and W is the dimension of the voxel on the z-axis. is the total size of all objects in the driving environment on the x-axis, is the total size of all objects in the driving environment on the y-axis, It is the total size of all objects in the driving environment on the z-axis;

[0022] Key voxels are identified in each voxel of the driving environment, and voxels whose key degrees exceed a preset threshold are marked as key voxels. The key voxel identification formula is: Where, L j,DHW is the criticality of the voxels in the D, H, and W dimensions corresponding to the j-th channel of the laser radar in the driving environment, (x jo ,y jo ,z jo ) is the oth 3D point cloud data in the D, H, W dimensional voxel, (x jp ,y jp ,z jp ) is the pth 3D point cloud data in the D, H, W dimensional voxel, ||(x jo ,y jo ,z jo )-(x jp,y jp ,z jp )|| 2 2 is (x jo ,y jo ,z jo ) and (x jp ,y jp ,z jp ), N is the total number of 3D point cloud data contained in the voxel grid;

[0023] Inputting key voxels of the driving environment into a fully connected layer to extract key voxel features of the driving environment, and inputting each voxel of the driving environment into a fully connected layer to extract all voxel features of the driving environment;

[0024] The entire voxel feature and the key voxel feature are fused with corresponding coordinates to obtain voxel features with prominent key information.

[0025] As a preferred solution of the present invention, the importance identification method of voxel features includes:

[0026] The voxel feature is subjected to a sigmoid function to obtain an importance probability distribution map of the voxel feature;

[0027] According to the importance probability distribution map, voxel signs whose importance probability exceeds a preset threshold are extracted as enhanced voxel features.

[0028] As a preferred solution of the present invention, a feature pyramid is used to extract multi-scale features in a visual image of a driving environment.

[0029] As a preferred embodiment of the present invention, the projection screening method of multi-scale features comprises:

[0030] Project the center point of the enhanced voxel feature onto the image feature map, and select the corresponding multi-scale feature according to the projection position of the voxel center point on the feature map;

[0031] The selected multi-scale features are recombined into features of the same size as the enhanced voxel features to obtain enhanced pixel features.

[0032] As a preferred solution of the present invention, the enhanced voxel feature and the enhanced pixel feature are multiplied at corresponding positions and fused to obtain the multi-source environment feature.

[0033] As a preferred solution of the present invention, the loss functions of the classifier and regressor for multi-source environmental feature detection include classifier loss and regressor loss, wherein the classifier loss is a softmax function loss, and the regressor loss is a SmoothL1 function loss.

[0034] As a preferred solution of the present invention, the multi-source environmental features are subjected to maximum pooling processing before entering the classifier and the regressor.

[0035] As a preferred solution of the present invention, the present invention provides an environment perception enhancement system based on intelligent driving, which is applied to an environment perception enhancement method based on intelligent driving. The system includes:

[0036] LiDAR, which is used to collect radar scan data of the driving environment;

[0037] Industrial cameras, used to collect visual image data of the driving environment;

[0038] A data preprocessing unit, used for constructing three-dimensional point cloud data of the driving environment based on radar data of the driving environment;

[0039] A feature processing unit, used to extract voxel features from the three-dimensional point cloud data of the driving environment, and identify the importance of the voxel features to obtain enhanced voxel features in the three-dimensional point cloud data for sensing the shape and positioning of obstacles in the driving environment;

[0040] It is used to extract multi-scale features from visual image data of the driving environment, and use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment;

[0041] And, used for fusing the enhanced voxel features and the enhanced pixel features to obtain multi-source environment features;

[0042] The obstacle detection unit is used to detect obstacles based on multi-source environmental features using a classifier and a regressor to obtain the target position and target category of obstacles in the driving environment.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention has performed feature enhancement in the feature extraction of radar scanning data and visual image data, so as to extract effective features for environmental obstacle perception from multi-source information, eliminate redundant features, improve the efficiency of obstacle perception using multi-source information, and meet the timeliness requirements of intelligent driving for environmental perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0046] Figure 1 A flow chart of an environment perception enhancement method based on intelligent driving provided by an embodiment of the present invention;

[0047] Figure 2 A block diagram of an environment perception enhancement system based on intelligent driving provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a multi-scale convolution kernel provided in an embodiment of the present invention;

[0049] Figure 4 A structural diagram of a model for environmental perception enhancement provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, the present invention provides an environment perception enhancement method based on intelligent driving, comprising the following steps:

[0052] The laser radar mounted on the vehicle collects radar scanning data of the driving environment, and constructs three-dimensional point cloud data of the driving environment based on the radar data of the driving environment;

[0053] Collect visual image data of the driving environment through the industrial camera mounted on the vehicle;

[0054] Extract voxel features from the three-dimensional point cloud data of the driving environment, identify the importance of the voxel features, and obtain enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and positioning of obstacles in the driving environment;

[0055] Extract multi-scale features from visual image data of the driving environment, and use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment;

[0056] The enhanced voxel features and enhanced pixel features are fused to obtain multi-source environmental features. Classifiers and regressors are used to detect obstacles on the multi-source environmental features to obtain the target position and target category of obstacles in the driving environment.

[0057] The present invention uses laser radar and industrial cameras to obtain radar scanning data for sensing the shape and position of obstacles in the driving environment, as well as visual image data of the color and texture of obstacles, to achieve environmental perception through multi-source information fusion. Compared with target detection using a single information source, multi-source information enhances the identifiable feature quantity of obstacle perception, thereby improving the accuracy of environmental perception. Therefore, multi-source information enhances the environmental perception accuracy of intelligent driving.

[0058] Multi-source information enhances the identifiable feature quantity of obstacle perception in the driving environment, that is, the three-dimensional point cloud data representing the shape and position of the obstacle is combined with the two-dimensional visual image data, so as to realize high-precision fusion detection by using redundant perception information. The addition of redundant perception information will inevitably lead to low efficiency of fusion detection. Therefore, the present invention provides a feature enhancement mode, which extracts effective features for obstacle perception from the three-dimensional point cloud data of the obstacle shape and position and the two-dimensional visual image data of the obstacle color texture, eliminates redundant features, enhances the features for identifying the obstacle category and positioning, improves the obstacle recognition efficiency, and finally realizes the high efficiency of obstacle perception using multi-source information, and meets the timeliness requirements of intelligent driving for environmental perception.

[0059] Specifically, the present invention first reduces the dimension of the three-dimensional point cloud data of the driving environment containing the obstacle morphological position information into voxel form, thereby reducing the obstacle morphological position information from a high-dimensional representation to a low-dimensional representation, and converting the data processing object from a point cloud to a whole block of voxels composed of multiple point clouds, thereby reducing the data processing amount and making it easier to quickly and prominently mine the obstacle morphological position information in the three-dimensional point cloud data in the driving environment, that is, mining the obstacle information in units of whole blocks of voxels rather than mining in a single point cloud, thereby improving the efficiency of mining the obstacle morphological position information.

[0060] In order to realize the mining of obstacle morphological position information with voxels as processing units, the present invention identifies key voxels in each voxel of the driving environment, wherein the key voxel is the voxel corresponding to the morphological position information of the environment with high discreteness. The morphological position information presented by the environment in the voxel is highly discrete, indicating that the amount of information that can be mined contained in the voxel is large, and the corresponding available feature quantity is also large. The greater the possibility of containing obstacle morphological position information, the greater the correlation with the identification of obstacle morphological position information, and the more worthy the included features in the voxel are to be enhanced. Therefore, the present invention extracts features from key voxels, and then superimposes the voxel features of the key voxels on the corresponding positions of the original full voxel features, so as to highlight the features related to obstacle recognition in the full voxel features, and widen the gap between background information and information related to obstacle recognition.

[0061] The present invention further utilizes the importance recognition of voxel features, and extracts the features related to obstacle recognition from all voxel features including the features related to obstacle recognition as enhanced voxel features for subsequent obstacle recognition, so that the classifier and regressor used for subsequent obstacle detection only process the enhanced voxel features, or only process the morphological position information most relevant to obstacle recognition, that is, to achieve feature enhancement of mining the morphological position information of obstacles.

[0062] After completing the feature enhancement of the voxel feature, the present invention projects the enhancement of the voxel feature onto the pixel feature of the visual image data, so that the multi-scale feature obtained from the visual image data only contains the pixel feature related to the enhanced voxel feature, so that the pixel feature most relevant to obstacle recognition is obtained in the multi-scale feature as the enhanced pixel feature, that is, only the color texture information of the obstacle is retained on the visual image data for subsequent obstacle recognition, so that the classifier and regressor used for subsequent obstacle detection only process the enhanced pixel feature, or only process the color texture information most relevant to obstacle recognition, that is, the feature enhancement of mining the color texture information of the obstacle is realized.

[0063] After completing the feature enhancement on the three-dimensional point cloud data and the feature enhancement on the two-dimensional image data, the present invention adopts the feature multiplication method to fuse the enhanced voxel features and the enhanced pixel features. By utilizing the feature multiplication fusion method, the important features related to obstacle recognition are further highlighted in the multi-source information, and irrelevant features are suppressed, that is, the features of the multi-source information are further enhanced.

[0064] In summary, the present invention enhances features step by step so that target detection of obstacles only processes valid information in the color texture and shape position of the obstacles. In other words, only valid information is used in the redundant multi-source perception information, and there is no need to process redundant information. On the basis of retaining the obstacle perception accuracy of the multi-source perception information, the obstacle perception efficiency is improved, which meets the requirements of intelligent driving for perception accuracy and efficiency.

[0065] The present invention uses the radar scanning data of the laser radar to perceive the shape and position of obstacles in the driving environment, and constructs the driving environment in the form of a three-dimensional point cloud, as follows:

[0066] The method for constructing the three-dimensional point cloud data of the driving environment includes:

[0067] The laser range value, horizontal angle value, vertical angle value and horizontal offset angle of the laser channel in the radar scanning data are converted into three-dimensional coordinates to obtain the three-dimensional coordinate data of each object in the driving environment;

[0068] The conversion formula for three-dimensional coordinate data is:

[0069]

[0070] In the formula, x kj is the x-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, and y kj is the y-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, z kj is the z-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, r kj is the laser ranging value of the jth channel of the laser radar in the driving environment scanning the kth object, α kj is the horizontal angle value of the jth channel of the laser radar in the driving environment scanning the kth object, δ kj is the horizontal offset angle value of the kth object scanned by the jth channel of the laser radar in the driving environment, β kj The vertical angle value of the kth object scanned by the jth channel of the laser radar in the driving environment;

[0071] Among them, the three-dimensional coordinate data of the kth object in the driving environment is {(x kj ,y kj ,z kj )|j∈[1,n]}, where (x kj ,y kj ,z kj ) is the 3D coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, and n is the number of channels of the laser radar; the 3D coordinate data of each object is combined to form the 3D point cloud data of the driving environment. The 3D point cloud data of the driving environment is: {(x kj ,y kj ,z kj )|j∈[1,n],k∈[1,m]}, where m is the total number of objects in the driving environment.

[0072] The present invention firstly reduces the dimension of the three-dimensional point cloud data of the driving environment containing the obstacle morphological position information into the form of voxels, thereby reducing the obstacle morphological position information from a high-dimensional representation to a low-dimensional representation, and converting the data processing object from a point cloud to a whole block of voxels composed of multiple point clouds, thereby reducing the amount of data processing, making it easier and faster to highlight the obstacle morphological position information in the three-dimensional point cloud data of the driving environment, that is, mining the obstacle information in units of a whole block of voxels rather than mining in a single point cloud, thereby improving the efficiency of mining the obstacle morphological position information, as follows:

[0073] The voxel feature extraction methods include:

[0074] The three-dimensional point cloud data of the driving environment is converted into voxel representation using the voxel grid (d, h, w), and the voxel representation of the driving environment is obtained as follows:

[0075]

[0076] In the formula, (D j ,H j ,W j ) is the voxel coordinate corresponding to the jth channel of the laser radar in the driving environment, D j is the x-axis coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, H j is the y-axis coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, W j is the z-axis dimension coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, d is the x-axis dimension of the voxel grid, h is the y-axis dimension of the voxel grid, w is the z-axis dimension of the voxel grid, D is the dimension of the voxel on the x-axis, H is the dimension of the voxel on the y-axis, and W is the dimension of the voxel on the z-axis. is the total size of all objects in the driving environment on the x-axis, is the total size of all objects in the driving environment on the y-axis, It is the total size of all objects in the driving environment on the z-axis;

[0077] Key voxels are identified in each voxel of the driving environment, and voxels whose key degree exceeds the preset threshold are marked as key voxels. The key voxel identification formula is: Where, L j,DHW is the criticality of the voxels in the D, H, and W dimensions corresponding to the j-th channel of the laser radar in the driving environment, (x jo ,y jo ,z jo ) is the oth 3D point cloud data in the D, H, W dimensional voxel, (x jp ,y jp ,z jp ) is the pth 3D point cloud data in the D, H, W dimensional voxel, is (x jo ,y jo ,z jo ) and (x jp ,y jp ,z jp ), N is the total number of 3D point cloud data contained in the voxel grid;

[0078] Inputting key voxels of the driving environment into a fully connected layer to extract key voxel features of the driving environment, and inputting each voxel of the driving environment into a fully connected layer to extract all voxel features of the driving environment;

[0079] The overall voxel features are fused with the key voxel features at corresponding positions to obtain the voxel features that highlight the key information.

[0080] In order to realize the mining of obstacle morphological position information with voxels as processing units, the present invention identifies key voxels in each voxel of the driving environment, wherein the key voxel is the voxel corresponding to the morphological position information of the environment with high discreteness. The morphological position information presented by the environment in the voxel is highly discrete, indicating that the amount of information that can be mined contained in the voxel is large, and the corresponding available feature quantity is also large. The greater the possibility of containing obstacle morphological position information, the greater the correlation with the identification of obstacle morphological position information, and the more worthy the included features in the voxel are to be enhanced. Therefore, the present invention extracts features from key voxels, and then superimposes the voxel features of the key voxels on the corresponding positions of the original full voxel features, so as to highlight the features related to obstacle recognition in the full voxel features, and widen the gap between background information and information related to obstacle recognition.

[0081] The present invention further utilizes the importance recognition of voxel features, and extracts the features related to obstacle recognition from all voxel features including features related to obstacle recognition as enhanced voxel features for subsequent obstacle recognition, so that the classifier and regressor used for subsequent obstacle detection only process the enhanced voxel features, or only process the morphological position information most relevant to obstacle recognition, that is, to achieve feature enhancement of mining obstacle morphological position information, as follows:

[0082] Methods for identifying the importance of voxel features include:

[0083] The voxel features are passed through the sigmoid function to obtain the importance probability distribution map of the voxel features;

[0084] According to the importance probability distribution map, voxel signs whose importance probability exceeds a preset threshold are extracted as enhanced voxel features.

[0085] The feature pyramid is used to extract multi-scale features in the visual image of the driving environment, where the convolution kernel for multi-scale feature extraction is: F = (k-1)*d+1, where k is the size of the conventional convolution kernel, F is the size of the dilated convolution kernel, and d is the dilation rate. Figure 3 As shown in the figure, when the conventional convolution kernel with k=3, d=1 is expanded with a dilation rate of 2, it is transformed into an expanded convolution kernel with F=5, d=2, which can expand the receptive field and capture multi-scale context information.

[0086] Projection screening methods for multi-scale features include:

[0087] Project the center point of the enhanced voxel feature onto the image feature map, and select the corresponding multi-scale feature according to the projection position of the voxel center point on the feature map;

[0088] The selected multi-scale features are recombined into features of the same size as the enhanced voxel features to obtain enhanced pixel features.

[0089] The enhanced voxel features and enhanced pixel features are multiplied at corresponding positions and fused to obtain multi-source environment features.

[0090] The loss functions of the classifier and regressor for multi-source environmental feature detection include classifier loss and regressor loss, wherein the classifier loss is a softmax function loss, and the regressor loss is a SmoothL1 function loss.

[0091] The softmax function loss is: In the formula, s i is the i-th vector value of the fully connected layer in the classifier, and n is the total number of classification categories;

[0092] The loss of SmoothL1 function is: In the formula, P is the predicted value, P r is the true value, α is the smoothing area control parameter, α=3.0.

[0093] like Figure 4 As shown, the intelligent driving environment perception model structure of the present invention includes: a feature extraction module, a feature enhancement module, a feature fusion module and a classification regression module;

[0094] The feature extraction module is composed of a convolutional layer and a fully connected layer, wherein the voxel representation of the driving environment is input into the feature extraction module to obtain voxel features; the visual image data of the driving environment is input into the feature extraction module to obtain multi-scale features;

[0095] The voxel features of the driving environment are enhanced by the feature enhancement module to obtain enhanced voxel features, and the multi-scale features are enhanced by the feature enhancement module to obtain enhanced pixel features;

[0096] Enhanced pixel features and enhanced voxel features are used to obtain multi-source environmental features through a feature fusion module;

[0097] The classification and regression module consists of a classifier and a regressor, where the regressor outputs the detection anchor box position (target position) and the classifier outputs the classification category of the obstacle (target category).

[0098] Multi-source environmental features are processed by maximum pooling before entering the classifier and regressor.

[0099] like Figure 2 As shown, the present invention provides an environment perception enhancement system based on intelligent driving, which is applied to an environment perception enhancement method based on intelligent driving. The system includes:

[0100] LiDAR, which is used to collect radar scan data of the driving environment;

[0101] Industrial cameras, used to collect visual image data of the driving environment;

[0102] A data preprocessing unit, used for constructing three-dimensional point cloud data of the driving environment based on radar data of the driving environment;

[0103] A feature processing unit is used to extract voxel features from the three-dimensional point cloud data of the driving environment, and identify the importance of the voxel features to obtain enhanced voxel features in the three-dimensional point cloud data for sensing the shape and positioning of obstacles in the driving environment;

[0104] It is used to extract multi-scale features from visual image data of the driving environment, and use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment;

[0105] and, for fusing the enhanced voxel features and the enhanced pixel features to obtain multi-source environment features;

[0106] The obstacle detection unit is used to detect obstacles based on multi-source environmental features using a classifier and a regressor to obtain the target position and target category of obstacles in the driving environment.

[0107] The present invention has performed feature enhancement in the feature extraction of radar scanning data and visual image data, so as to extract effective features for environmental obstacle perception from multi-source information, eliminate redundant features, improve the efficiency of obstacle perception using multi-source information, and meet the timeliness requirements of intelligent driving for environmental perception.

[0108] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for enhancing environmental perception based on intelligent driving, characterized in that: The following steps are involved: The laser radar mounted on the vehicle collects radar scanning data of the driving environment, and constructs three-dimensional point cloud data of the driving environment based on the radar data of the driving environment; Collect visual image data of the driving environment through industrial cameras mounted on vehicles; Extracting voxel features from the three-dimensional point cloud data of the driving environment, and identifying the importance of the voxel features, to obtain enhanced voxel features in the three-dimensional point cloud data for sensing the shape and positioning of obstacles in the driving environment; Extract multi-scale features from visual image data of the driving environment, and use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment; The enhanced voxel features and enhanced pixel features are feature-fused to obtain multi-source environment features, and obstacle detection is performed on the multi-source environment features using a classifier and a regressor to obtain target positions and target categories of obstacles in the driving environment.

2. The method for enhancing environmental perception based on intelligent driving according to claim 1, characterized in that: The method for constructing three-dimensional point cloud data of the driving environment includes: Performing three-dimensional coordinate conversion on the laser distance measurement value, horizontal angle value, vertical angle value and horizontal offset angle of the laser channel in the radar scanning data to obtain three-dimensional coordinate data of each object in the driving environment; The conversion formula of the three-dimensional coordinate data is: In the formula, x kj is the x-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, and y kj is the y-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, z kj is the z-axis coordinate of the k-th object corresponding to the j-th channel of the laser radar in the driving environment, r kj is the laser ranging value of the jth channel of the laser radar in the driving environment scanning the kth object, α kj is the horizontal angle value of the jth channel of the laser radar in the driving environment scanning the kth object, δ kj is the horizontal offset angle value of the kth object scanned by the jth channel of the laser radar in the driving environment, β kj The vertical angle value of the kth object scanned by the jth channel of the laser radar in the driving environment; Among them, the three-dimensional coordinate data of the kth object in the driving environment is {(x kj ,y kj ,z kj )|j∈[1,n]}, where (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, and n is the number of channels of the laser radar; the three-dimensional coordinate data of each object is combined to form the three-dimensional point cloud data of the driving environment, and the three-dimensional point cloud data of the driving environment is: {(x kj ,y kj ,z kj )|j∈[1,n],k∈[1,m]}, where m is the total number of objects in the driving environment.

3. The method for enhancing environmental perception based on intelligent driving according to claim 2, characterized in that: The voxel feature extraction method includes: The three-dimensional point cloud data of the driving environment is converted into voxel representation using the voxel grid (d, h, w), and the voxel representation of the driving environment is obtained as follows: In the formula, (D j ,H j ,W j ) is the voxel coordinate corresponding to the jth channel of the laser radar in the driving environment, D j is the x-axis coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, H j is the y-axis coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, W j is the z-axis dimension coordinate of the voxel corresponding to the j-th channel of the laser radar in the driving environment, d is the x-axis dimension of the voxel grid, h is the y-axis dimension of the voxel grid, w is the z-axis dimension of the voxel grid, D is the dimension of the voxel on the x-axis, H is the dimension of the voxel on the y-axis, and W is the dimension of the voxel on the z-axis. is the total size of all objects in the driving environment on the x-axis, is the total size of all objects in the driving environment on the y-axis, It is the total size of all objects in the driving environment on the z-axis; Key voxels are identified in each voxel of the driving environment, and voxels whose key degrees exceed a preset threshold are marked as key voxels. The key voxel identification formula is: Where, L j,DHW is the criticality of the voxels in the D, H, and W dimensions corresponding to the j-th channel of the laser radar in the driving environment, (x jo ,y jo ,z jo ) is the oth 3D point cloud data in the D, H, W dimensional voxel, (x jp ,y jp ,z jp ) is the pth 3D point cloud data in the D, H, W dimensional voxel, is (x jo ,y jo ,z jo ) and (x jp ,y jp ,z jp ), N is the total number of 3D point cloud data contained in the voxel grid; Inputting key voxels of the driving environment into a fully connected layer to extract key voxel features of the driving environment, and inputting each voxel of the driving environment into a fully connected layer to extract all voxel features of the driving environment; The entire voxel feature and the key voxel feature are fused with corresponding coordinates to obtain voxel features with prominent key information.

4. The method for enhancing environmental perception based on intelligent driving according to claim 3, characterized in that: The importance identification method of the voxel feature comprises: The voxel feature is subjected to a sigmoid function to obtain an importance probability distribution map of the voxel feature; According to the importance probability distribution map, voxel signs whose importance probability exceeds a preset threshold are extracted as enhanced voxel features.

5. The method for enhancing environmental perception based on intelligent driving according to claim 4, characterized in that: Feature pyramid is used to extract multi-scale features in visual images of driving environments.

6. The method for enhancing environmental perception based on intelligent driving according to claim 5, characterized in that: The projection screening method of the multi-scale feature comprises: Project the center point of the enhanced voxel feature onto the image feature map, and select the corresponding multi-scale feature according to the projection position of the voxel center point on the feature map; The selected multi-scale features are recombined into features of the same size as the enhanced voxel features to obtain enhanced pixel features.

7. The method for enhancing environmental perception based on intelligent driving according to claim 6, characterized in that: The enhanced voxel feature and the enhanced pixel feature are multiplied at corresponding positions and fused to obtain the multi-source environment feature.

8. The method for enhancing environmental perception based on intelligent driving according to claim 7, characterized in that: The loss functions of the classifier and regressor for multi-source environmental feature detection include classifier loss and regressor loss, wherein the classifier loss is a softmax function loss, and the regressor loss is a SmoothL1 function loss.

9. The method for enhancing environmental perception based on intelligent driving according to claim 8, characterized in that: The multi-source environmental features are subjected to maximum pooling processing before entering the classifier and the regressor.

10. An environment perception enhancement system based on intelligent driving, characterized in that: The method for enhancing environmental perception based on intelligent driving as described in any one of claims 1 to 9 comprises: LiDAR, which is used to collect radar scan data of the driving environment; Industrial cameras, used to collect visual image data of the driving environment; A data preprocessing unit, used for constructing three-dimensional point cloud data of the driving environment based on radar data of the driving environment; A feature processing unit, used to extract voxel features from the three-dimensional point cloud data of the driving environment, and identify the importance of the voxel features to obtain enhanced voxel features in the three-dimensional point cloud data for sensing the shape and positioning of obstacles in the driving environment; It is used to extract multi-scale features from visual image data of the driving environment, and use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment; And, used for fusing the enhanced voxel features and the enhanced pixel features to obtain multi-source environment features; The obstacle detection unit is used to detect obstacles based on multi-source environmental features using a classifier and a regressor to obtain the target position and target category of obstacles in the driving environment.

Images