An intelligent driving-based environment perception enhancement method and system

Abstract

The present application relates to the technical field of intelligent driving, and particularly relates to an environment perception enhancement method and system based on intelligent driving, comprising the following steps: obtaining, in three-dimensional point cloud data of the driving environment, enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and location of obstacles in the driving environment; obtaining, in visual image data of the driving environment, enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment; fusing the enhanced voxel features and the enhanced pixel features to obtain multi-source environment features, and detecting obstacles based on the multi-source environment features to obtain the target position and target category of the obstacles in the driving environment. The present application extracts effective features for environment obstacle perception from multi-source information, eliminates redundant features, improves the efficiency of obstacle perception using multi-source information, and meets the timeliness requirements of intelligent driving for environment perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to an environment perception enhancement method and system based on intelligent driving. BACKGROUND

[0002] The most commonly used environment perception in intelligent driving is the camera, which is closest to the human eye and can see color signs and objects, understand fonts, and distinguish red and green lights. However, it also has many shortcomings, such as a significant decrease in vision at night or in bad weather, and poor long-distance observation. The second is laser radar. It is commonly seen on the roof of the car, like a hat that keeps rotating. The principle is to calculate the reflection time and wavelength of the laser beam to complete the 3D drawing of the surrounding obstacles, and the shortcoming is that it cannot recognize images and colors.

[0003] In the prior art, laser radar and cameras are combined in intelligent driving environment perception to achieve multi-source information environment perception and improve the accuracy of environment perception. However, using multi-source information for environment perception will inevitably increase information redundancy and reduce the efficiency of environment perception. A long environment perception process can cause road traffic congestion, so it is difficult to meet the timeliness requirements of intelligent driving for environment perception. SUMMARY

[0004] The purpose of the present application is to provide an environment perception enhancement method and system based on intelligent driving to solve the technical problem that using multi-source information for environment perception in the prior art will inevitably increase information redundancy, reduce the efficiency of environment perception, and make it difficult to meet the timeliness requirements of intelligent driving for environment perception.

[0005] To solve the above technical problems, the present application specifically provides the following technical solutions:

[0006] An environment perception enhancement method based on intelligent driving, comprising the following steps:

[0007] Collecting radar scan data of the driving environment by a laser radar mounted on the vehicle, and constructing three-dimensional point cloud data of the driving environment based on the radar data of the driving environment;

[0008] Collecting visual image data of the driving environment by an industrial camera mounted on the vehicle;

[0009] Extracting voxel features from the three-dimensional point cloud data of the driving environment, and identifying the importance of the voxel features to obtain enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and positioning of obstacles in the driving environment;

[0010] The multi-scale features are extracted from the visual image data of the driving environment, and the multi-scale features are projected and screened by using the enhanced voxel features to obtain enhanced pixel features for perceiving the texture and color of the obstacles in the driving environment.

[0011] The enhanced voxel features and the enhanced pixel features are fused to obtain multi-source environment features, and a classifier and a regressor are used for obstacle detection on the multi-source environment features to obtain the target position and the target category of the obstacles in the driving environment.

[0012] As a preferred scheme of the present application, the method for constructing the three-dimensional point cloud data of the driving environment comprises:

[0013] The laser ranging value, the horizontal angle value, the vertical angle value and the horizontal offset angle of the laser channel in the radar scanning data are converted into three-dimensional coordinates to obtain the three-dimensional coordinate data of each object in the driving environment;

[0014] The conversion formula of the three-dimensional coordinate data is:

[0015]

[0016] In the formula, x kj is the x-axis coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, y kj is the y-axis coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, z kj is the z-axis coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, r kj is the laser ranging value of the kth object scanned by the jth channel of the laser radar in the driving environment, a kj is the horizontal angle value of the kth object scanned by the jth channel of the laser radar in the driving environment, d kj is the horizontal offset angle value of the kth object scanned by the jth channel of the laser radar in the driving environment, b kj is the vertical angle value of the kth object scanned by the jth channel of the laser radar in the driving environment.

[0017] In the formula, the three-dimensional coordinate data of the kth object in the driving environment is {(x kj ,y kj ,z kj )|j∈[1,n]}, in which (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the kth object corresponding to the jth channel of the laser radar in the driving environment, and n is the number of channels of the laser radar; the three-dimensional coordinate data of each object is combined to form the three-dimensional point cloud data of the driving environment, and the three-dimensional point cloud data of the driving environment is:kj ,y kj ,z kj )|j∈[1,n],k∈[1,m]}, where m is the total number of objects in the driving environment.

[0018] As a preferred embodiment of the present invention, the method for extracting voxel features includes:

[0019] The 3D point cloud data of the driving environment is converted into a voxel representation using a voxel mesh (d,h,w), resulting in the voxel representation of the driving environment as follows:

[0020]

[0021] In the formula, (D j H j W j Let be the voxel coordinates corresponding to the j-th channel of the LiDAR in the driving environment, and D be the voxel coordinates. j H represents the x-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment. j W represents the y-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment. j Let be the z-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment, d be the x-axis dimension of the voxel mesh, h be the y-axis dimension of the voxel mesh, w be the z-axis dimension of the voxel mesh, D be the dimension of the voxel on the x-axis, H be the dimension of the voxel on the y-axis, and W be the dimension of the voxel on the z-axis. This represents the total dimension formed by all objects in the driving environment along the x-axis. This represents the total dimension formed by all objects in the driving environment along the y-axis. This represents the total dimension formed by all objects in the driving environment along the z-axis.

[0022] Key voxels are identified among various voxels in the driving environment, and voxels whose keyness exceeds a preset threshold are marked as key voxels. The key voxel identification formula is as follows: In the formula, L j,DHW For the criticality of the voxel in dimensions D, H, and W corresponding to the j-th channel of the LiDAR in the driving environment, (x jo ,y jo ,z jo (x) represents the o-th 3D point cloud data in the voxels of dimensions D, H, and W. jp ,y jp ,z jp Let be the p-th 3D point cloud data in the voxel of dimensions D, H, and W, ||(x) jo ,y jo ,z jo )-(x jp,y jp ,z jp )|| 2 2 is (x jo ,y jo ,z jo ) and (x jp ,y jp ,z jp The L2 norm between ) and N, where N is the total number of 3D point cloud data contained in the voxel mesh;

[0023] The key voxels of the driving environment are input into the fully connected layer to extract the key voxel features of the driving environment, and each voxel of the driving environment is input into the fully connected layer to extract the voxel features of the driving environment.

[0024] The voxel features and key voxel features are fused according to their corresponding coordinates to obtain voxel features that highlight key information.

[0025] As a preferred embodiment of the present invention, the method for identifying the importance of voxel features includes:

[0026] The importance probability distribution map of the voxel features is obtained by using the sigmoid function;

[0027] Based on the importance probability distribution map, voxel features with importance probabilities exceeding a preset threshold are extracted as enhanced voxel features.

[0028] As a preferred embodiment of the present invention, feature pyramids are used to extract multi-scale features from visual images of the driving environment.

[0029] As a preferred embodiment of the present invention, the projection screening method for multi-scale features includes:

[0030] The center point of the enhanced voxel feature is projected onto the image feature map, and the corresponding multi-scale feature is selected according to the projection position of the voxel center point on the feature map.

[0031] The selected multi-scale features are recombined into features of the same size as the enhanced voxel features to obtain enhanced pixel features.

[0032] As a preferred embodiment of the present invention, the enhanced voxel features and enhanced pixel features are multiplied at corresponding positions and fused to obtain the multi-source environmental features.

[0033] As a preferred embodiment of the present invention, the loss functions of the classifier and regressor for multi-source environmental feature detection include classifier loss and regressor loss, wherein the classifier loss is a softmax function loss and the regressor loss is a SmoothL1 function loss.

[0034] As a preferred embodiment of the present invention, the multi-source environmental features are subjected to max pooling before entering the classifier and regressor.

[0035] As a preferred embodiment of the present invention, the present invention provides an environment perception enhancement system based on intelligent driving, applied to an environment perception enhancement method based on intelligent driving, the system comprising:

[0036] LiDAR is used to collect radar scan data of the driving environment;

[0037] Industrial cameras are used to acquire visual image data of the driving environment;

[0038] The data preprocessing unit is used to construct three-dimensional point cloud data of the driving environment based on radar data of the driving environment;

[0039] The feature processing unit is used to extract voxel features from the three-dimensional point cloud data of the driving environment and to identify the importance of the voxel features to obtain enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and positioning of obstacles in the driving environment.

[0040] This is used to extract multi-scale features from visual image data in a driving environment, and to use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving obstacle texture and color in a driving environment.

[0041] And, for fusing the enhanced voxel features and enhanced pixel features to obtain multi-source environmental features;

[0042] The obstacle detection unit is used to detect obstacles based on multi-source environmental features using classifiers and regressors, and to obtain the target location and target category of obstacles in the driving environment.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] This invention enhances the features of both radar scan data and visual image data, enabling the extraction of effective features for environmental obstacle perception from multi-source information, eliminating redundant features, improving the efficiency of obstacle perception using multi-source information, and meeting the timeliness requirements of intelligent driving for environmental perception. Attached Figure Description

[0045] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0046] Figure 1 A flowchart of an environmental perception enhancement method based on intelligent driving provided in an embodiment of the present invention;

[0047] Figure 2 This is a block diagram of an environmental perception enhancement system based on intelligent driving provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of a multi-scale convolution kernel provided in an embodiment of the present invention;

[0049] Figure 4 This is a structural diagram of the environmental perception enhancement model provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, the present invention provides an environmental perception enhancement method based on intelligent driving, comprising the following steps:

[0052] The vehicle collects radar scan data of the driving environment using a lidar mounted on the vehicle, and constructs three-dimensional point cloud data of the driving environment based on the radar data of the driving environment.

[0053] Visual image data of the driving environment is collected by industrial cameras mounted on the vehicle.

[0054] Voxel features are extracted from 3D point cloud data of driving environment, and the importance of voxel features is identified to obtain enhanced voxel features in 3D point cloud data for perceiving obstacle shape and localization in driving environment.

[0055] Multi-scale features are extracted from visual image data of the driving environment, and enhanced voxel features are used to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment.

[0056] Enhanced voxel features and enhanced pixel features are fused to obtain multi-source environmental features. Classifiers and regressors are used to detect obstacles in the multi-source environmental features to obtain the target location and target category of obstacles in the driving environment.

[0057] This invention utilizes LiDAR and industrial cameras to acquire radar scan data for perceiving the shape and location of obstacles in a driving environment, as well as visual image data of obstacle color and texture. This achieves environmental perception through multi-source information fusion. Compared to target detection from a single information source, multi-source information enhances the identifiable features of obstacle perception, thereby improving the accuracy of environmental perception. Therefore, multi-source information enhances the environmental perception accuracy of intelligent driving.

[0058] Multi-source information enhances the identifiable features of obstacle perception in the driving environment. Specifically, it combines 3D point cloud data representing the shape and location of obstacles with 2D visual image data, achieving high-precision fusion detection using redundant perception information. However, the addition of redundant perception information inevitably leads to low efficiency in fusion detection. Therefore, this invention provides a feature enhancement mode that extracts effective features for obstacle perception from 3D point cloud data representing the shape and location of obstacles and 2D visual image data representing the color and texture of obstacles. Redundant features are eliminated, enhancing the features for identifying obstacle categories and locations, improving obstacle recognition efficiency, and ultimately achieving high efficiency in obstacle perception using multi-source information, meeting the timeliness requirements of intelligent driving for environmental perception.

[0059] Specifically, this invention first reduces the dimensionality of the 3D point cloud data of the driving environment containing obstacle shape and location information to voxel form, thereby reducing the obstacle shape and location information from a high-dimensional representation to a low-dimensional representation. The data processing object is transformed from a point cloud into a whole voxel composed of multiple point clouds, which reduces the amount of data processing and makes it easier to quickly extract obstacle shape and location information from the 3D point cloud data of the driving environment. That is, obstacle information is mined on a whole voxel basis, rather than on a single point cloud, which improves the efficiency of mining obstacle shape and location information.

[0060] To achieve obstacle morphology and location information mining at the voxel level, this invention identifies key voxels among the voxels in the driving environment. Key voxels are those containing highly discrete environmental morphology and location information. The high discreteness of the environmental morphology and location information within a voxel indicates a large amount of mineable information and corresponding usable features. The greater the likelihood that a voxel contains obstacle morphology and location information and the stronger its correlation with obstacle morphology and location identification, the more valuable its contained features are to be enhanced. Therefore, this invention extracts features from key voxels and then superimposes these voxel features onto the corresponding positions of all voxel features in the original system. This highlights features related to obstacle identification within the overall voxel features, widening the gap between background information and obstacle identification-related information.

[0061] This invention further utilizes the importance recognition of voxel features. Among all voxel features containing features related to obstacle recognition, the features related to obstacle recognition are extracted as enhanced voxel features for subsequent obstacle recognition. This allows the classifiers and regressors used for subsequent obstacle detection to process only the enhanced voxel features, or in other words, only the morphological and positional information most relevant to obstacle recognition, thus achieving feature enhancement for mining obstacle morphological and positional information.

[0062] After completing the feature enhancement of voxel features, this invention projects the enhanced voxel features onto the pixel features of the visual image data. This ensures that the multi-scale features obtained from the visual image data only contain pixel features related to the enhanced voxel features. Thus, the pixel features most relevant to obstacle recognition are obtained from the multi-scale features as enhanced pixel features. In other words, only the color and texture information of obstacles is retained in the visual image data for subsequent obstacle recognition. This allows the classifiers and regressors used for subsequent obstacle detection to process only the enhanced pixel features, or in other words, only the color and texture information most relevant to obstacle recognition. This achieves feature enhancement for mining obstacle color and texture information.

[0063] After completing feature enhancement on 3D point cloud data and 2D image data, this invention uses feature multiplication to fuse enhanced voxel features and enhanced pixel features. By using the feature multiplication fusion method, important features for obstacle recognition are further highlighted in multi-source information, while irrelevant features are suppressed, thus further enhancing the features of multi-source information.

[0064] In summary, this invention, through step-by-step feature enhancement, enables obstacle target detection to process only the effective information in the obstacle's color, texture, shape, and position. In other words, it utilizes only the effective information from redundant multi-source perception information, without processing redundant information. While preserving the obstacle perception accuracy of multi-source perception information, it improves obstacle perception efficiency, meeting the requirements of intelligent driving for perception accuracy and efficiency.

[0065] This invention utilizes lidar to perceive the shape and location of obstacles in a driving environment using radar scanning data, and constructs the driving environment in the form of a three-dimensional point cloud, as detailed below:

[0066] Methods for constructing 3D point cloud data of the driving environment include:

[0067] Three-dimensional coordinates are calculated from the radar scan data, including the laser range value, horizontal angle value, vertical angle value, and horizontal offset angle of the laser channel, to obtain the three-dimensional coordinate data of each object in the driving environment.

[0068] The conversion formula for 3D coordinate data is:

[0069]

[0070] In the formula, x kj Let x be the x-axis coordinate of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment, and y be the x-axis coordinate of the k-th object. kj Let z be the y-axis coordinate of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment. kj Let r be the z-axis coordinate of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment. kj For the laser ranging value of the k-th object scanned by the j-th channel of the lidar in the driving environment, α kj For the horizontal angle value of the k-th object scanned by the j-th channel of the lidar in the driving environment, δ kj For the horizontal offset angle value of the k-th object scanned by the j-th channel of the lidar in the driving environment, β kj The vertical angle value of the k-th object is scanned by the j-th channel of the LiDAR in the driving environment;

[0071] Among them, the three-dimensional coordinate data of the k-th object in the driving environment is {(x kj ,y kj ,z kj )|j∈[1,n]}, where (x kj ,y kj ,z kj Let be the 3D coordinates of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment, and n be the number of LiDAR channels. The 3D coordinate data of each object are combined to form the 3D point cloud data of the driving environment. The 3D point cloud data of the driving environment is: {(x... kj ,y kj ,z kj )|j∈[1,n],k∈[1,m]}, where m is the total number of objects in the driving environment.

[0072] This invention first reduces the dimensionality of 3D point cloud data of the driving environment, which contains obstacle shape and location information, into voxel format. This reduces the high-dimensional representation of obstacle shape and location information to a low-dimensional representation, and transforms the data processing object from point clouds to a whole voxel composed of multiple point clouds. This reduces the amount of data processing and makes it easier to quickly extract obstacle shape and location information from 3D point cloud data in the driving environment. In other words, obstacle information is extracted on a whole voxel basis, rather than on a single point cloud, which improves the efficiency of extracting obstacle shape and location information. Specifically:

[0073] Methods for extracting voxel features include:

[0074] The 3D point cloud data of the driving environment is converted into a voxel representation using a voxel mesh (d,h,w), resulting in the voxel representation of the driving environment as follows:

[0075]

[0076] In the formula, (D j H j W j Let be the voxel coordinates corresponding to the j-th channel of the LiDAR in the driving environment, and D be the voxel coordinates. j H represents the x-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment. j W represents the y-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment. j Let be the z-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment, d be the x-axis dimension of the voxel mesh, h be the y-axis dimension of the voxel mesh, w be the z-axis dimension of the voxel mesh, D be the dimension of the voxel on the x-axis, H be the dimension of the voxel on the y-axis, and W be the dimension of the voxel on the z-axis. This represents the total dimension formed by all objects in the driving environment along the x-axis. This represents the total dimension formed by all objects in the driving environment along the y-axis. This represents the total dimension formed by all objects in the driving environment along the z-axis.

[0077] Key voxels are identified among the voxels in the driving environment, and voxels whose keyness exceeds a preset threshold are marked as key voxels. The key voxel identification formula is as follows: In the formula, L j,DHW For the criticality of the voxel in dimensions D, H, and W corresponding to the j-th channel of the LiDAR in the driving environment, (x jo ,y jo ,z jo (x) represents the o-th 3D point cloud data in the voxels of dimensions D, H, and W. jp ,y jp ,z jp () represents the p-th 3D point cloud data in the voxels of dimensions D, H, and W. For (x) jo ,y jo ,z jo ) and (x jp ,y jp ,z jp The L2 norm between ) and N, where N is the total number of 3D point cloud data contained in the voxel mesh;

[0078] The key voxels of the driving environment are input into the fully connected layer to extract the key voxel features of the driving environment, and each voxel of the driving environment is input into the fully connected layer to extract the voxel features of the driving environment.

[0079] By fusing the overall voxel features with the key voxel features at corresponding positions, voxel features that highlight key information are obtained.

[0080] To achieve obstacle morphology and location information mining at the voxel level, this invention identifies key voxels among the voxels in the driving environment. Key voxels are those containing highly discrete environmental morphology and location information. The high discreteness of the environmental morphology and location information within a voxel indicates a large amount of mineable information and corresponding usable features. The greater the likelihood that a voxel contains obstacle morphology and location information and the stronger its correlation with obstacle morphology and location identification, the more valuable its contained features are to be enhanced. Therefore, this invention extracts features from key voxels and then superimposes these voxel features onto the corresponding positions of all voxel features in the original system. This highlights features related to obstacle identification within the overall voxel features, widening the gap between background information and obstacle identification-related information.

[0081] This invention further utilizes the importance recognition of voxel features. From all voxel features containing features relevant to obstacle recognition, features related to obstacle recognition are extracted as enhanced voxel features for subsequent obstacle recognition. This ensures that the classifiers and regressors used for subsequent obstacle detection only process enhanced voxel features, or in other words, only process the morphological and positional information most relevant to obstacle recognition. This achieves feature enhancement for mining obstacle morphological and positional information, as detailed below:

[0082] Methods for identifying the importance of voxel features include:

[0083] The importance probability distribution of voxel features is obtained by using the sigmoid function;

[0084] Based on the importance probability distribution map, voxel features with importance probabilities exceeding a preset threshold are extracted as enhanced voxel features.

[0085] Feature pyramids are used to extract multi-scale features from visual images of driving environments. The convolutional kernel for multi-scale feature extraction is: F = (k-1)*d+1, where k is the size of the regular convolutional kernel, F is the size of the dilated convolutional kernel, and d is the dilation rate. Figure 3 As shown, when the regular convolutional kernel with k=3 and d=1 is expanded by an expansion rate of 2, it is transformed into an expanded convolutional kernel with F=5 and d=2, thereby expanding the receptive field and capturing multi-scale contextual information.

[0086] Multi-scale feature projection filtering methods include:

[0087] The center point of the enhanced voxel feature is projected onto the image feature map, and the corresponding multi-scale feature is selected according to the projection position of the voxel center point on the feature map.

[0088] The selected multi-scale features are recombined into features of the same size as the enhanced voxel features to obtain enhanced pixel features.

[0089] Enhanced voxel features and enhanced pixel features are multiplied at corresponding positions and fused to obtain multi-source environmental features.

[0090] The loss functions for classifiers and regressors in detecting multi-source environmental features include classifier loss and regressor loss. The classifier loss is the softmax function loss, and the regressor loss is the SmoothL1 function loss.

[0091] The softmax function loss is: In the formula, s i is the i-th vector value in the fully connected layer of the classifier, and n is the total number of classification categories;

[0092] The loss of the SmoothL1 function is: In the formula, P is the predicted value, P r α is the true value, and α is the smoothing region control parameter, α = 3.0.

[0093] like Figure 4 As shown, the intelligent driving environment perception model structure of the present invention includes: a feature extraction module, a feature enhancement module, a feature fusion module, and a classification and regression module;

[0094] The feature extraction module consists of convolutional layers and fully connected layers. The voxel representation of the driving environment is input into the feature extraction module to obtain voxel features; the visual image data of the driving environment is input into the feature extraction module to obtain multi-scale features.

[0095] The voxel features of the driving environment are transformed into enhanced voxel features through the feature enhancement module, and the multi-scale features are transformed into enhanced pixel features through the feature enhancement module.

[0096] Enhanced pixel features and enhanced voxel features are used to obtain multi-source environmental features via a feature fusion module;

[0097] The classification and regression module consists of a classifier and a regressor. The regressor outputs the detected anchor box position (target position), and the classifier outputs the classification category of the obstacle (target category).

[0098] Multi-source environmental features were max-pooled before entering the classifier and regressor.

[0099] like Figure 2 As shown, this invention provides an environmental perception enhancement system based on intelligent driving, applied to an environmental perception enhancement method based on intelligent driving. The system includes:

[0100] LiDAR is used to collect radar scan data of the driving environment;

[0101] Industrial cameras are used to acquire visual image data of the driving environment;

[0102] The data preprocessing unit is used to construct three-dimensional point cloud data of the driving environment based on radar data of the driving environment;

[0103] The feature processing unit is used to extract voxel features from the three-dimensional point cloud data of the driving environment and to identify the importance of the voxel features, so as to obtain enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and positioning of obstacles in the driving environment.

[0104] This is used to extract multi-scale features from visual image data in a driving environment, and to use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving obstacle texture and color in a driving environment.

[0105] And, used to fuse enhanced voxel features and enhanced pixel features to obtain multi-source environmental features;

[0106] The obstacle detection unit is used to detect obstacles based on multi-source environmental features using classifiers and regressors, and to obtain the target location and target category of obstacles in the driving environment.

[0107] This invention enhances the features of both radar scan data and visual image data, enabling the extraction of effective features for environmental obstacle perception from multi-source information, eliminating redundant features, improving the efficiency of obstacle perception using multi-source information, and meeting the timeliness requirements of intelligent driving for environmental perception.

[0108] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A method for enhancing environmental perception based on intelligent driving, characterized in that, Includes the following steps: The vehicle collects radar scan data of the driving environment using a lidar mounted on the vehicle, and constructs three-dimensional point cloud data of the driving environment based on the radar data of the driving environment. Visual image data of the driving environment is collected by industrial cameras mounted on the vehicle. Voxel features are extracted from the three-dimensional point cloud data of the driving environment, and the importance of the voxel features is identified to obtain enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and positioning of obstacles in the driving environment. Multi-scale features are extracted from visual image data of the driving environment, and enhanced voxel features are used to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving the texture and color of obstacles in the driving environment. The enhanced voxel features and enhanced pixel features are fused to obtain multi-source environmental features. Classifiers and regressors are used to detect obstacles in the multi-source environmental features to obtain the target location and target category of obstacles in the driving environment. The key voxel recognition formula is: In the formula, The criticality of the voxel in dimensions D, H, and W corresponding to the j-th channel of the LiDAR in the driving environment. For the o-th 3D point cloud data in the voxels of dimensions D, H, and W, For the p-th 3D point cloud data in the voxel of dimensions D, H, and W, for and The L2 norm between them, where N is the total number of 3D point cloud data contained in the voxel mesh.

2. The environmental perception enhancement method based on intelligent driving according to claim 1, characterized in that: The method for constructing the three-dimensional point cloud data of the driving environment includes: The laser ranging value, horizontal angle value, vertical angle value, and horizontal offset angle of the laser channel are converted into three-dimensional coordinates in the radar scanning data to obtain the three-dimensional coordinate data of each object in the driving environment. The conversion formula for the three-dimensional coordinate data is: ; In the formula, Let x be the x-axis coordinate of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment. Let be the y-axis coordinate of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment. Let be the z-axis coordinate of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment. The laser ranging value of the k-th object scanned by the j-th channel of the lidar in the driving environment is given. To scan the horizontal angle value of the k-th object using the j-th channel of the LiDAR in a driving environment. The horizontal offset angle value for the k-th object detected by the j-th channel of the LiDAR in the driving environment. The vertical angle value of the k-th object is scanned by the j-th channel of the LiDAR in the driving environment; Among them, the three-dimensional coordinate data of the k-th object in the driving environment are: In the formula, Let n be the 3D coordinates of the k-th object corresponding to the j-th channel of the LiDAR in the driving environment, and n be the number of LiDAR channels. The 3D coordinate data of each object are combined to form the 3D point cloud data of the driving environment, which is: In the formula, m is the total number of objects in the driving environment.

3. The environmental perception enhancement method based on intelligent driving according to claim 2, characterized in that: The method for extracting voxel features includes: The 3D point cloud data of the driving environment is used with voxel mesh ( Converting to voxel representation, the voxel representation of the driving environment is obtained as follows: ； In the formula, Let J be the voxel coordinates corresponding to the j-th channel of the LiDAR in the driving environment. Let x be the x-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment. Let y be the y-axis coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment. Let be the z-axis dimension coordinate of the voxel corresponding to the j-th channel of the LiDAR in the driving environment, d be the x-axis dimension of the voxel mesh, h be the y-axis dimension of the voxel mesh, w be the z-axis dimension of the voxel mesh, D be the dimension of the voxel on the x-axis, H be the dimension of the voxel on the y-axis, and W be the dimension of the voxel on the z-axis. This represents the total dimension formed by all objects in the driving environment along the x-axis. This represents the total dimension formed by all objects in the driving environment along the y-axis. This represents the total dimension formed by all objects in the driving environment along the z-axis. Key voxels are identified in each voxel of the driving environment, and voxels whose keyness exceeds a preset threshold are marked as key voxels. The key voxels of the driving environment are input into the fully connected layer to extract the key voxel features of the driving environment, and each voxel of the driving environment is input into the fully connected layer to extract the voxel features of the driving environment. The voxel features and key voxel features are fused according to their corresponding coordinates to obtain voxel features that highlight key information.

4. The environmental perception enhancement method based on intelligent driving according to claim 3, characterized in that: The method for identifying the importance of voxel features includes: The importance probability distribution map of the voxel features is obtained by using the sigmoid function; Based on the importance probability distribution map, voxel features with importance probabilities exceeding a preset threshold are extracted as enhanced voxel features.

5. The environmental perception enhancement method based on intelligent driving according to claim 4, characterized in that: Feature pyramids are used to extract multi-scale features from visual images of driving environments.

6. The environmental perception enhancement method based on intelligent driving according to claim 5, characterized in that: The projection filtering method for the multi-scale features includes: The center point of the enhanced voxel feature is projected onto the image feature map, and the corresponding multi-scale feature is selected according to the projection position of the voxel center point on the feature map. The selected multi-scale features are recombined into features of the same size as the enhanced voxel features to obtain enhanced pixel features.

7. The environmental perception enhancement method based on intelligent driving according to claim 6, characterized in that: The enhanced voxel features and enhanced pixel features are multiplied at corresponding positions and fused to obtain the multi-source environmental features.

8. The environmental perception enhancement method based on intelligent driving according to claim 7, characterized in that: The loss functions for classifiers and regressors in detecting multi-source environmental features include classifier loss and regressor loss. The classifier loss is the softmax function loss, and the regressor loss is the SmoothL1 function loss.

9. The environmental perception enhancement method based on intelligent driving according to claim 8, characterized in that: The multi-source environmental features were max-pooled before entering the classifier and regressor.

10. An environmental perception enhancement system based on intelligent driving, characterized in that, An environmental perception enhancement method based on intelligent driving, applicable to any one of claims 1-9, comprises: LiDAR is used to collect radar scan data of the driving environment; Industrial cameras are used to acquire visual image data of the driving environment; The data preprocessing unit is used to construct three-dimensional point cloud data of the driving environment based on radar data of the driving environment; The feature processing unit is used to extract voxel features from the three-dimensional point cloud data of the driving environment and to identify the importance of the voxel features to obtain enhanced voxel features in the three-dimensional point cloud data for perceiving the shape and positioning of obstacles in the driving environment. This is used to extract multi-scale features from visual image data in a driving environment, and to use enhanced voxel features to project and filter the multi-scale features to obtain enhanced pixel features in the visual image for perceiving obstacle texture and color in a driving environment. And, for fusing the enhanced voxel features and enhanced pixel features to obtain multi-source environmental features; The obstacle detection unit is used to detect obstacles based on multi-source environmental features using classifiers and regressors, and to obtain the target location and target category of obstacles in the driving environment.

Images