Three-dimensional environment sensing method and system based on multi-sensor fusion

Abstract

The invention discloses a three-dimensional environment sensing method and system based on multi-sensor fusion, and belongs to the technical field of intelligent driving, and the method comprises the steps: collecting and processing image data and point cloud data; for image data, the number of convolution kernels is dynamically adjusted, sampling rationality is optimized by improving a sampling point calculation mode, image feature extraction is carried out, then feature enhancement is carried out, and an enhanced image feature map is obtained; performing effective area division and voxel filtering processing on the point cloud data to generate a point cloud feature map; performing auxiliary conversion on the enhanced image feature map through a point cloud feature map, uniformly converting the enhanced image feature map into a BEV feature map, fusing the two BEV feature maps by adopting an attention weighting strategy, and finally obtaining a fused BEV feature map; and based on the fused BEV feature map, a final 3D target detection frame is generated in combination with thermodynamic diagram analysis and parameter regression technologies, and deep complementation of the image and the point cloud information is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a three-dimensional environmental perception method and system based on multi-sensor fusion. Background Technology

[0002] In the field of intelligent driving, environmental perception systems are one of the core modules for achieving autonomous driving. Existing perception technologies mainly include image perception based on cameras and point cloud perception based on lidar. Cameras can provide high-resolution texture and semantic information, but lack accurate depth measurement capabilities; lidar has high-precision spatial distance measurement capabilities, but point cloud data is sparse, which limits its ability, especially in the detection of long-distance and small targets. Summary of the Invention

[0003] To address the problems in the existing technology, this invention provides a three-dimensional environment perception method and system based on multi-sensor fusion.

[0004] On the one hand, a three-dimensional environment perception method based on multi-sensor fusion is provided, the method comprising:

[0005] S1: Image data and point cloud data acquisition and processing;

[0006] S2: Perform horizontal and vertical differences on the input image data to obtain the gradient map, calculate the gradient magnitude, use the global mean of the gradient magnitude as the image complexity, dynamically adjust the convolution kernel according to the image complexity, and then generate basic features through modulation, bias and offset in deformable convolution, and then perform sampling window calculation.

[0007] Spatial offset sampling is performed on the input image data to obtain a standard feature map;

[0008] Global channel statistics are performed by calling global average pooling to process the standard feature map after Dropout regularization, compressing the 2D feature map of each channel into a single value to obtain the global statistical information of the corresponding channel. Then, the channel weight generation stage is entered. First, the result of global average pooling is flattened into a 1D vector, and after processing by the Sigmoid activation function, channel weights are generated. Attention weighting is performed by broadcasting and multiplying the standard feature map with the generated channel weights, and then convolving to obtain the attention-weighted feature map.

[0009] A dynamic convolution operation is performed on the weighted feature map to obtain a convolutional feature map. Then, the convolutional feature map is multiplied element-wise with the modulation coefficients in the generated basic features to obtain a refined feature map. The refined feature map is then added element-wise with the bias term coefficients in the generated basic features to supplement the required offset, and finally the extracted image feature map is obtained.

[0010] Then, feature enhancement is performed on the image feature map to obtain the enhanced image feature map;

[0011] S3: For point cloud data, divide the effective region, then perform voxel filtering to generate point cloud feature map;

[0012] S4: The enhanced image feature map is transformed with the aid of point cloud feature map and then uniformly transformed into BEV feature map. The two BEV feature maps are then fused using an attention weighting strategy to obtain the fused BEV feature map.

[0013] S5: Based on the fused BEV feature map, combined with heatmap analysis and parameter regression techniques, the final 3D target detection box is generated.

[0014] Furthermore, the step of dynamically adjusting the convolution kernel based on image complexity specifically includes: using 12 convolution kernels when the image complexity is > 0.1, and using 9 convolution kernels when the image complexity is ≤ 0.1.

[0015] Furthermore, the sampling window calculation includes: calculating the height and width of the sampling window based on the training epoch. If the epoch is less than 100, the sampling window size is an integer reduction factor of the input image data size. When the epoch is greater than or equal to 100, the sampling window size parameter remains unchanged.

[0016] Furthermore, in S2, the feature enhancement specifically includes:

[0017] The image feature map is flattened in spatial dimension and transposed in channel and sequence dimensions to form sequence features. The transposed sequence features are then saved as identity features.

[0018] Normalize and learnable affine transformations of sequence features;

[0019] Then, dimensionality reduction and spatial morphology restoration are performed.

[0020] The recovered feature maps are subjected to three different sizes of depthwise separable convolutions to obtain feature maps at three scales. The feature maps at the three scales are then normalized by weighted summation using three learnable weights to achieve dynamic weighted fusion. Finally, dual residual connections are performed.

[0021] Intrachannel information fusion is performed on the feature map after double residual connection using 1×1 convolution;

[0022] The feature maps after information fusion are rearranged and reshaped in dimensions to restore them to sequence features.

[0023] Finally, the recovered sequence features are residually concatenated with the saved identity features and output to obtain an enhanced image feature map.

[0024] Furthermore, the spatial dimension flattening includes: merging the height × width dimension of the image feature map into a spatial sequence length dimension through a dimension reshaping operation to obtain a 3D feature, and the dimension composition becomes batch, channel, and spatial sequence length;

[0025] The channel and sequence dimension transpose includes: by transposing the dimension, the positions of the channel and the spatial sequence length are swapped to obtain the sequence features, and the dimension composition becomes batch, spatial sequence length, and channel;

[0026] The input normalization and learnable affine transformation of the sequence features include: first, performing layer normalization on the channel dimension of the sequence features, that is, standardizing the C feature values ​​at each spatial location to a mean of 0 and a variance of 1, maintaining the output shape as batch, spatial sequence length, and channel; then performing a learnable affine transformation, using two learnable parameters, scaling input and offset input, to perform scaling and offset operations on the normalized sequence features.

[0027] The dimensionality reduction and spatial morphology restoration include: firstly, performing linear dimensionality reduction through a linear dimensionality reduction layer to reduce the channel dimension of the learnable affine transformation sequence features from the original C dimension to a fixed hidden dimension, and the output shape becomes batch, spatial sequence length, and hidden dimension; then, performing spatial morphology restoration, firstly splitting the spatial sequence length dimension into two independent dimensions, height and width, through dimensional reshaping to obtain the features of batch, height, width, and hidden dimensions; and then swapping the positions of the hidden dimension and height dimension through dimensional rearrangement.

[0028] The dual residual connection includes: adding the dynamically weighted fused feature map to the original image feature map, retaining the initial low-dimensional spatial features, to obtain the feature map after the first residual processing, further enhancing the feature map after the first residual processing, and then adding the enhancement result to the identity feature to obtain the feature map after the dual residual connection.

[0029] The dimensional rearrangement and dimensional reshaping include: firstly, by rearranging the dimensions, the hidden dimensions of the feature map after information fusion are moved to the end to obtain the features of batch, height, width, and hidden dimensions; then, by reshaping the dimensions, the height × width is merged into the spatial sequence length to restore the sequence features, with the shape being batch, spatial sequence length, and hidden dimensions.

[0030] The residual connection and output includes: adding the dimension-reshaped sequence features to the saved identity features to obtain an enhanced image feature map.

[0031] Furthermore, the three different sizes of depth-separable convolutions are 3×3 convolution, 5×5 convolution, and 7×7 convolution.

[0032] Furthermore, S4 specifically includes:

[0033] Image depth distribution is predicted by combining enhanced image feature maps with point cloud feature maps, and radar occupancy is predicted by using point cloud feature maps.

[0034] The enhanced image feature map is outer-producted with the predicted depth distribution, and the enhanced image feature map is outer-producted with the predicted radar occupancy. Finally, the images are stitched together along the channel dimension to form the image view feature map.

[0035] The point cloud feature map is projected onto the image view feature map, then voxelized into image view frustums, and then point cloud features are obtained using PointNet and sparse convolutional coding. Finally, the point cloud view feature map is obtained through convolution.

[0036] The image view feature map and the point cloud view feature map are respectively pooled by BEV to obtain the image BEV feature map and the radar BEV feature map. The image BEV feature map and the radar BEV feature map are then subjected to an attention mechanism to calculate the attention weights for the two modal features respectively, and then weighted and fused to obtain the fused BEV feature map.

[0037] Furthermore, S5 specifically includes:

[0038] A heatmap is generated on the fused BEV feature map using a 2D convolutional network. Each pixel value in the heatmap represents the probability that the location is the center point of the target. For the high-probability center points in the heatmap, ROI pooling is used to extract the local features corresponding to each high-probability center point from the fused BEV feature map. All parameters required to generate the 3D target detection box are then regressed, including: size parameters, orientation parameters, and height offset.

[0039] Pixels with a probability higher than a threshold are selected from the heatmap. Their corresponding BEV coordinates are the projection center of the target on the ground. Combined with the height offset from the regression, the 3D center point of the target is obtained. Based on the 3D center point, size parameters, and orientation parameters, the coordinates of the eight vertices of the 3D target detection box are calculated through geometric transformation, and finally the 3D target detection box is obtained.

[0040] On the other hand, a three-dimensional environment perception system based on multi-sensor fusion is provided to implement the aforementioned three-dimensional environment perception method based on multi-sensor fusion. The system includes:

[0041] LiDAR sensors, installed in autonomous vehicles, are used to collect point cloud data;

[0042] Camera sensors, installed in intelligent driving vehicles, are used to collect image data;

[0043] Data processors, installed in intelligent driving vehicles, are used for processing point cloud data and image data;

[0044] The image feature extraction module is used to perform horizontal and vertical differences on the input image data to obtain gradient maps, calculate gradient magnitudes, use the global mean of gradient magnitudes as image complexity, dynamically adjust the convolution kernels according to image complexity, and then generate basic features through modulation, bias and offset in deformable convolution, followed by sampling window calculation.

[0045] Spatial offset sampling is performed on the input image data to obtain a standard feature map;

[0046] Then, global channel statistics are performed. Global average pooling is called to process the standard feature map after Dropout regularization, compressing the 2D feature map of each channel into a single value to obtain the global statistical information of the corresponding channel. Then, the channel weight generation stage is entered. First, the result of global average pooling is flattened into a 1-dimensional vector. After processing by the Sigmoid activation function, channel weights are generated. Attention weighting is performed. The standard feature map and the generated channel weights are broadcast and multiplied. After convolution, attention-weighted feature map is obtained.

[0047] Then, a dynamic convolution operation is performed on the weighted feature map to obtain a convolutional feature map. Next, the convolutional feature map is multiplied element-wise with the modulation coefficients in the generated basic features to obtain a refined feature map. Then, the refined feature map is added element-wise with the bias term coefficients in the generated basic features to supplement the required offset, and finally the extracted image feature map is obtained.

[0048] The feature enhancement module is used to enhance the features of the image feature map to obtain an enhanced image feature map;

[0049] The point cloud feature extraction module is used to divide the effective region based on point cloud data, and then perform voxel filtering to generate point cloud feature maps.

[0050] The feature fusion module is used to transform the enhanced image feature map with the assistance of the point cloud feature map and convert it into a unified BEV feature map. The two BEV feature maps are fused using an attention weighting strategy to finally obtain the fused BEV feature map.

[0051] The 3D object detection box generation module is used to generate the final 3D object detection box based on the fused BEV feature map, combined with heatmap analysis and parametric regression techniques.

[0052] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows: The three-dimensional environment perception method based on multi-sensor fusion in the present invention proposes adaptive input normalization and dynamic multi-scale convolution perception to improve the expression and transfer capabilities of visual features; it also proposes to dynamically adjust the number of convolution kernels according to the image complexity, and combine attention mechanism and improved sampling point calculation strategy to accurately capture important features of different scales and regions, realizing deep complementarity between image and point cloud information, so as to achieve higher precision and stronger robustness of three-dimensional environment perception. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an overall framework diagram of a three-dimensional environment perception method based on multi-sensor fusion provided by the present invention;

[0055] Figure 2 This is a flowchart of an image feature extraction method provided by the present invention;

[0056] Figure 3 This is a flowchart of a feature enhancement method provided by the present invention;

[0057] Figure 4 This is a flowchart of a 3D target detection box generation method provided by the present invention;

[0058] Figure 5 This is an installation diagram of a three-dimensional environment perception system based on multi-sensor fusion provided by the present invention.

[0059] Reference numerals: 1-LiDAR sensor; 2-Camera sensor; 3-User display; 4-Intelligent driving vehicle; 5-Data processor. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0062] Example 1

[0063] A three-dimensional environment perception method based on multi-sensor fusion includes the following steps:

[0064] Step (1): First, collect and process the raw image data of the environment surrounding the intelligent driving vehicle to provide necessary data information for subsequent data processing and fusion, specifically including:

[0065] The first step is to collect a large amount of image data under different types of environments (environments in different places). For each environment, sample photos should be taken from different angles and under different lighting conditions to increase the diversity of the raw data. For example, 10,000 raw images should be taken around the predetermined road.

[0066] Step 2: Convert the collected JPG images to PNG format and save them.

[0067] Step 3: Considering the common changes in objects in the test environment, random noise is used for data augmentation. This method increases the number of samples and improves the robustness of the model.

[0068] Step (2): Collect and process the raw point cloud data of the environment surrounding the intelligent driving vehicle to provide necessary data information for subsequent data processing and fusion, specifically including:

[0069] Step 1 involves collecting a large amount of point cloud data under different environments. For each environment, sample point cloud data under different environmental conditions should be collected as much as possible to improve the diversity of the raw data. For example, 10,000 frames of discontinuous raw point cloud data should be collected around a predetermined road.

[0070] Step 2 involves dividing the point cloud data into effective regions. For example, if the point cloud density is too low to accurately represent obstacle information 100 meters directly in front of the LiDAR sensor, the point cloud data beyond 100 meters should be removed, retaining the point cloud data within 20 meters behind the LiDAR sensor. Therefore, the range for the autonomous vehicle in the X direction is (-20, 100), and the point cloud data within 10 meters on either side of the autonomous vehicle's Y axis is retained, i.e., the range in the Y direction is (-10, 10), and the range in the Z direction is (-2, 5).

[0071] Step 3: Filter the point cloud data, such as by performing voxel filtering, to remove some dense and useless points from the original point cloud data, reducing the number of points without affecting the geometric features of objects in the point cloud.

[0072] Step (3): Feature extraction of image data: For image data, dynamically adjust the number of convolution kernels and optimize the sampling rationality by improving the sampling point calculation method, thereby extracting image features and obtaining image feature maps.

[0073] See Figure 2 Specifically, this includes: performing horizontal and vertical differences on the input image data to obtain a gradient map, calculating the gradient magnitude, using the global mean of the gradient magnitude as the image complexity, dynamically adjusting the convolution kernel according to the image complexity (12 convolution kernels when the image complexity > 0.1, and 9 convolution kernels when the image complexity ≤ 0.1), and then generating basic features through modulation, bias, and offset in deformable convolution.

[0074] Next, the sampling window is calculated. The height and width of the sampling window are calculated based on the training epoch. If the epoch is less than 100, the sampling window size is an integer factor that reduces the size of the input image data. When the epoch is greater than or equal to 100, the sampling window size parameter remains fixed.

[0075] Spatial offset sampling of the input image data includes: calculating adjustment coefficients in the length and width directions based on the size of the input image data and the size of the target output feature map; generating a normalized reference coordinate grid based on the size of the target output feature map (i.e., the number of sampling points); creating basic sampling coordinates based on the actual size of the input image data; applying the aforementioned adjustment coefficients in the length and width directions to the basic sampling coordinates; repeatedly sampling along the sampling dimension and concatenating them to generate an offset for each output position; the final sampling point coordinates are obtained by adding the generated offset to the neighborhood reference coordinates; for each determined final sampling point coordinate, selecting its four nearest neighboring points (top left, bottom right, etc.) and using bilinear interpolation to calculate the feature value of the sampling point; and concatenating and reshaping the feature values ​​of all sampling points according to the calculated sampling window size to obtain a standard feature map.

[0076] Global channel statistics are performed by calling global average pooling to process the standard feature map after Dropout regularization, compressing the 2D feature map of each channel into a single value to obtain the global statistical information of the corresponding channel. Then, the channel weight generation stage is entered. First, the result of global average pooling is flattened into a 1D vector and then input into the connection layer. After processing by the Sigmoid activation function, the channel weights are generated. Finally, attention weighting is performed by broadcasting and multiplying the standard feature map with the generated channel weights, and then convolving to obtain the attention-weighted feature map.

[0077] A dynamic convolution operation is performed on the weighted feature map. A convolution kernel of matching specifications is selected from the convolution list, and the weighted feature map is convolved to obtain a convolutional feature map. Then, the convolutional feature map is multiplied element-wise with the modulation coefficients in the generated basic features to obtain a refined feature map. The refined feature map is then added element-wise with the bias term coefficients in the generated basic features to supplement the required offset, and finally the extracted image feature map is obtained.

[0078] Step (4): Perform feature enhancement on the extracted image feature map, see [link to relevant documentation]. Figure 3 Specifically, it includes:

[0079] Step 1: First, process the feature map of the input image, specifically as follows:

[0080] Spatial dimension flattening: Through dimension reshaping operation, the height × width dimension of the image feature map is merged into a spatial sequence length dimension, resulting in 3D features. The dimensional composition becomes batch, channel, and spatial sequence length.

[0081] Channel and sequence dimension transpose: By transposing the dimension, the positions of the channel and spatial sequence length are swapped, and the sequence features are finally obtained. The dimension composition becomes batch, spatial sequence length, and channel.

[0082] Residual preservation: The transposed sequence features are directly saved as identity features (residual branches) for subsequent residual join operations.

[0083] Step 2: Normalize and perform learnable affine transformation on the sequence features, specifically:

[0084] Layer normalization is performed on the “channel dimension (C)” of the sequence features: the C feature values ​​at each spatial location are standardized with a mean of 0 and a variance of 1, and the output shape is maintained as batch, spatial sequence length, and channel.

[0085] Learnable affine transformation: Using two learnable parameters (scaling input and offset input), scaling and offset operations are performed on the normalized sequence features.

[0086] Step 3: Dimensionality Reduction and Spatial Shape Restoration: Linear dimensionality reduction is performed through a linear layer to reduce the channel dimension of the sequence features from the original C dimension to a fixed hidden dimension. The output shape becomes batch, spatial sequence length, and hidden dimension. Then, spatial shape restoration is performed. First, the spatial sequence length dimension (height × width) is split into two independent dimensions, height and width, through dimensional reshaping to obtain the features of batch, height, width, and hidden dimensions. Then, the positions of the channel dimension (hidden dimension) and the height dimension are swapped through dimensional rearrangement to obtain the restored feature map.

[0087] Step 4: Perform three different sizes (3×3, 5×5 and 7×7) of depthwise separable convolution on the recovered feature maps to obtain three feature maps of three scales, which correspond to three feature maps of the same shape (3×3, 5×5 and 7×7 convolution outputs) for "local details", "mid-range correlation" and "global context" information, respectively.

[0088] Normalization is achieved by weighted summation of feature maps at three scales using three learnable weights, realizing dynamic weighted fusion. Then, a double residual connection is performed, specifically: First, the dynamically weighted fused feature map is added to the original image feature map, preserving the initial low-dimensional spatial features, resulting in a feature map after the first residual processing; Second, the feature map after the first residual processing undergoes further small-scale residual enhancement, and the enhancement result is added to the identity feature, resulting in a feature map after double residual connection, enhancing the non-linear expressive power of the features; Finally, intra-channel information fusion is performed on the double residual feature map through a 1×1 convolution (projection layer) to adapt the feature dimensions to subsequent processes.

[0089] Step 5: Perform dimensional rearrangement and dimensional reshaping on the above output, specifically including:

[0090] Dimensional rearrangement: The hidden dimension (channel dimension) of the feature map after information fusion is moved to the end, resulting in features with batch, height, width, and hidden dimension.

[0091] Dimensional reshaping: The height × width is merged into the spatial sequence length, restoring the sequence features. The shape is batch, spatial sequence length, and hidden dimension.

[0092] Step 6: Residual connection and output. The sequence features after dimensionality reshaping are added to the saved identity features to obtain the enhanced image feature map.

[0093] Step (5): The enhanced image feature map is transformed using point cloud feature map assistance, and then uniformly converted into a BEV feature map. An attention weighting strategy is then used to fuse the two BEV feature maps to obtain the final fused BEV feature map. Specifically, this includes:

[0094] Step 1: Predict the image depth distribution by combining the enhanced image feature map with the point cloud feature map, predict radar occupancy by using the point cloud feature map, and perform an outer product between the enhanced image feature map and the predicted depth distribution to assign depth information to the enhanced image feature map; perform an outer product between the enhanced image feature map and the predicted radar occupancy to verify the depth estimate obtained by the enhanced image feature map, and finally stitch them together in the channel dimension to form the image view feature map. The specific process is shown in Equation (1): (1) In equation (1), C C For image view feature maps, C ITo enhance the image feature map, D I For depth distribution, O represents radar occupancy.

[0095] Step 2: Project the point cloud feature map onto the image view feature map, then voxelize it into image view frustums, then use PointNet and sparse convolutional coding to obtain point cloud features, and then obtain the point cloud view feature map through convolution.

[0096] Step 3: Perform BEV pooling on the image view feature map and the point cloud view feature map respectively to obtain the image BEV feature map and the radar BEV feature map. Apply an attention mechanism to the image BEV feature map and the radar BEV feature map, calculate the attention weights for the two modal features respectively, and then perform weighted fusion to obtain the fused BEV feature map.

[0097] The BEV pooling for the image view feature map and the point cloud view feature map includes: combining the image view feature map with the point cloud depth information, and converting the pixel coordinates and depth information into 3D coordinates in the camera coordinate system through the camera intrinsic parameter matrix. The specific process is shown in Equations (2), (3), and (4).

[0098] (3)

[0099] (4)

[0100] (5)

[0101] In equations (2), (3), and (4), ƒ x and ƒ y C represents the focal length along the x and y axes. X With C Y Let U be the principal point of the camera, V be the pixel coordinates of a point on the image view feature map, d be the depth, and X, Y, Z be the 3D coordinates of that point on the image view feature map.

[0102] The 3D coordinates of the image view feature map and the point cloud view feature map are converted to the world coordinate system using the extrinsic parameter matrix of the camera and the world coordinate system, and then BEV projection is performed to obtain the image BEV feature map and the radar BEV feature map, respectively.

[0103] Step (6): Based on the fused BEV feature map, combined with heatmap analysis and parameter regression techniques, the final 3D target detection box is generated.

[0104] See Figure 4A heatmap is generated on the fused BEV feature map using a 2D convolutional network. Each pixel value in the heatmap represents the probability that the location is the center point of the target. For the high-probability center points in the heatmap, local features corresponding to each high-probability center point are extracted from the fused BEV feature map through ROI pooling. All parameters required to generate the 3D target detection box are then regressed, including: size parameters, orientation parameters, and height offset.

[0105] Pixels with a probability higher than a threshold are selected from the heatmap. Their corresponding BEV coordinates are the projection center of the target on the ground. Combined with the height offset from the regression, the 3D center point of the target is obtained. Based on the 3D center point, size parameters, and orientation parameters, the coordinates of the eight vertices of the 3D target detection box are calculated through geometric transformation, and finally the 3D target detection box is obtained.

[0106] It is worth noting that the 3D environment perception method based on multi-sensor fusion in this invention proposes adaptive input normalization and dynamic multi-scale convolutional perception to improve the expression and transfer capabilities of visual features. It also proposes to dynamically adjust the number of convolutional kernels according to image complexity, and combine attention mechanism and improved sampling point calculation strategy to accurately capture important features of different scales and regions, so as to achieve higher accuracy and stronger robustness of 3D environment perception.

[0107] Example 2

[0108] A three-dimensional environment perception system based on multi-sensor fusion is used to implement a three-dimensional environment perception method based on multi-sensor fusion in Embodiment 1, including a lidar sensor 1, a camera sensor 2, a user display 3, an intelligent driving vehicle 4, and a data processor 5.

[0109] See Figure 5 The lidar sensor 1 is installed at the center of the roof of the intelligent driving vehicle 4, and the camera sensor 2 is installed around the lidar sensor 1 with the lidar sensor 1 as the center. The user display 3 is installed in the driver's cabin of the intelligent driving vehicle 4, mainly for displaying the working status of the entire system. The data processor 5 is installed next to the user display 3 inside the intelligent driving vehicle 4, mainly for processing and fusing data information to improve detection accuracy.

[0110] See Figure 1After the camera sensor 2 and the lidar sensor 1 detect the target, the image data acquired by the camera sensor 2 and the point cloud data acquired by the lidar sensor 1 are input into the decoder (Multi-model Encoder) in the data processor 5 for processing. The processed image data and point cloud data are then input into the image feature extraction module and the point cloud feature extraction module to extract image features and point cloud features. For the image features, the number of convolution kernels is dynamically adjusted by the DARConv module, and the sampling rationality is optimized by improving the sampling point calculation method, so as to capture key information of the image more accurately and obtain an image feature map. Then, the Nova module is used to enhance the image features to obtain an enhanced image feature map.

[0111] For point cloud features, point cloud projection processing is performed first, and then voxel filtering is used to generate point cloud feature maps.

[0112] Simultaneously, it performs two major tasks: depth prediction and radar occupancy prediction.

[0113] After extraction, the image feature map, after being transformed with the aid of the point cloud feature map, is sent to the feature fusion module along with the point cloud feature map. The feature fusion module first further processes the image feature map and the point cloud feature map separately, and then transforms them into BEV (bird's-eye view) feature maps through BEVPooling, namely image BEV feature maps and radar BEV feature maps. Then, an attention weighting strategy is used to fuse the two BEV feature maps, and finally a high-quality fused BEV feature map is obtained. Finally, based on the fused BEV feature map, the 3D object detection box generation module combines heat map analysis and parameter regression technology to generate the final 3D object detection box.

[0114] It is worth noting that the 3D environment perception system based on multi-sensor fusion of the present invention not only achieves deep complementarity between image and point cloud information, but also effectively improves the accuracy and robustness of target detection in complex scenes through dynamic optimization and attention fusion strategies, providing reliable technical support for autonomous driving.

[0115] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional environment perception method based on multi-sensor fusion, characterized in that, The method includes: S1: Image data and point cloud data acquisition and processing; S2: Perform horizontal and vertical differences on the input image data to obtain the gradient map, calculate the gradient magnitude, use the global mean of the gradient magnitude as the image complexity, dynamically adjust the convolution kernel according to the image complexity, and then generate basic features through modulation, bias and offset in deformable convolution, and then perform sampling window calculation. Spatial offset sampling is performed on the input image data to obtain a standard feature map; Global channel statistics are performed by calling global average pooling to process the standard feature map after Dropout regularization, compressing the 2D feature map of each channel into a single value to obtain the global statistical information of the corresponding channel. Then, the channel weight generation stage is entered. First, the result of global average pooling is flattened into a 1D vector, and after processing by the Sigmoid activation function, channel weights are generated. Attention weighting is performed by broadcasting and multiplying the standard feature map with the generated channel weights, and then convolving to obtain the attention-weighted feature map. A dynamic convolution operation is performed on the weighted feature map to obtain a convolutional feature map. Then, the convolutional feature map is multiplied element-wise with the modulation coefficients in the generated basic features to obtain a refined feature map. The refined feature map is then added element-wise with the bias term coefficients in the generated basic features to supplement the required offset, and finally the extracted image feature map is obtained. Then, feature enhancement is performed on the image feature map to obtain the enhanced image feature map; S3: For point cloud data, divide the effective region, then perform voxel filtering to generate point cloud feature map; S4: The enhanced image feature map is transformed with the aid of point cloud feature map and then uniformly transformed into BEV feature map. The two BEV feature maps are then fused using an attention weighting strategy to obtain the fused BEV feature map. S5: Based on the fused BEV feature map, combined with heatmap analysis and parameter regression techniques, the final 3D target detection box is generated.

2. The three-dimensional environment perception method based on multi-sensor fusion according to claim 1, characterized in that, The specific method of dynamically adjusting the convolution kernel based on image complexity includes: using 12 convolution kernels when the image complexity is >0.1, and using 9 convolution kernels when the image complexity is ≤0.

1.

3. The three-dimensional environment perception method based on multi-sensor fusion according to claim 2, characterized in that, The sampling window calculation includes: calculating the height and width of the sampling window based on the training epoch. If the epoch is less than 100, the sampling window size is an integer reduction factor of the input image data size. When the epoch is greater than or equal to 100, the sampling window size parameter remains unchanged.

4. The three-dimensional environment perception method based on multi-sensor fusion according to claim 3, characterized in that, In S2, feature enhancement specifically includes: The image feature map is flattened in spatial dimension and transposed in channel and sequence dimensions to form sequence features. The transposed sequence features are then saved as identity features. Normalize and learnable affine transformations of sequence features; Then, dimensionality reduction and spatial morphology restoration are performed. The recovered feature maps are subjected to three different sizes of depthwise separable convolutions to obtain feature maps at three scales. The feature maps at the three scales are then normalized by weighted summation using three learnable weights to achieve dynamic weighted fusion. Finally, dual residual connections are performed. Intrachannel information fusion is performed on the feature map after double residual connection using 1×1 convolution; The feature maps after information fusion are rearranged and reshaped in dimensions to restore them to sequence features. Finally, the recovered sequence features are residually concatenated with the saved identity features and output to obtain an enhanced image feature map.

5. The three-dimensional environment perception method based on multi-sensor fusion according to claim 4, characterized in that, The spatial dimension flattening includes: merging the height × width dimension of the image feature map into a spatial sequence length dimension through a dimension reshaping operation to obtain a 3D feature, and the dimension composition becomes batch, channel, and spatial sequence length; The channel and sequence dimension transpose includes: by transposing the dimension, the positions of the channel and the spatial sequence length are swapped to obtain the sequence features, and the dimension composition becomes batch, spatial sequence length, and channel; The input normalization and learnable affine transformation of the sequence features include: first, performing layer normalization on the channel dimension of the sequence features, that is, standardizing the C feature values ​​at each spatial location to a mean of 0 and a variance of 1, maintaining the output shape as batch, spatial sequence length, and channel; then performing a learnable affine transformation, using two learnable parameters, scaling input and offset input, to perform scaling and offset operations on the normalized sequence features. The dimensionality reduction and spatial morphology restoration include: firstly, performing linear dimensionality reduction through a linear dimensionality reduction layer to reduce the channel dimension of the learnable affine transformation sequence features from the original C dimension to a fixed hidden dimension, and the output shape becomes batch, spatial sequence length, and hidden dimension; then, performing spatial morphology restoration, firstly splitting the spatial sequence length dimension into two independent dimensions, height and width, through dimensional reshaping to obtain the features of batch, height, width, and hidden dimensions; and then swapping the positions of the hidden dimension and height dimension through dimensional rearrangement. The dual residual connection includes: adding the dynamically weighted fused feature map to the original image feature map, retaining the initial low-dimensional spatial features, to obtain the feature map after the first residual processing, further enhancing the feature map after the first residual processing, and then adding the enhancement result to the identity feature to obtain the feature map after the dual residual connection. The dimensional rearrangement and dimensional reshaping include: firstly, by rearranging the dimensions, the hidden dimensions of the feature map after information fusion are moved to the end to obtain the features of batch, height, width, and hidden dimensions; then, by reshaping the dimensions, the height × width is merged into the spatial sequence length to restore the sequence features, with the shape being batch, spatial sequence length, and hidden dimensions. The residual connection and output includes: adding the dimension-reshaped sequence features to the saved identity features to obtain an enhanced image feature map.

6. The three-dimensional environment perception method based on multi-sensor fusion according to claim 5, characterized in that, The three different depthwise separable convolutions are 3×3 convolution, 5×5 convolution, and 7×7 convolution.

7. The three-dimensional environment perception method based on multi-sensor fusion according to claim 6, characterized in that, S4 specifically includes: Image depth distribution is predicted by combining enhanced image feature maps with point cloud feature maps, and radar occupancy is predicted by using point cloud feature maps. The enhanced image feature map is outer-producted with the predicted depth distribution, and the enhanced image feature map is outer-producted with the predicted radar occupancy. Finally, the images are stitched together along the channel dimension to form the image view feature map. The point cloud feature map is projected onto the image view feature map, then voxelized into image view frustums, and then point cloud features are obtained using PointNet and sparse convolutional coding. Finally, the point cloud view feature map is obtained through convolution. The image view feature map and the point cloud view feature map are respectively pooled by BEV to obtain the image BEV feature map and the radar BEV feature map. The image BEV feature map and the radar BEV feature map are then subjected to an attention mechanism to calculate the attention weights for the two modal features respectively, and then weighted and fused to obtain the fused BEV feature map.

8. The three-dimensional environment perception method based on multi-sensor fusion according to claim 7, characterized in that, S5 specifically includes: A heatmap is generated on the fused BEV feature map using a 2D convolutional network. Each pixel value in the heatmap represents the probability that the location is the center point of the target. For the high-probability center points in the heatmap, ROI pooling is used to extract the local features corresponding to each high-probability center point from the fused BEV feature map. All parameters required to generate the 3D target detection box are then regressed, including: size parameters, orientation parameters, and height offset. Pixels with a probability higher than a threshold are selected from the heatmap. Their corresponding BEV coordinates are the projection center of the target on the ground. Combined with the height offset from the regression, the 3D center point of the target is obtained. Based on the 3D center point, size parameters, and orientation parameters, the coordinates of the eight vertices of the 3D target detection box are calculated through geometric transformation, and finally the 3D target detection box is obtained.

9. A three-dimensional environment perception system based on multi-sensor fusion, characterized in that, The system is used to implement the three-dimensional environment perception method based on multi-sensor fusion as described in any one of claims 1-8, the system comprising: LiDAR sensors, installed in autonomous vehicles, are used to collect point cloud data; Camera sensors, installed in intelligent driving vehicles, are used to collect image data; Data processors, installed in intelligent driving vehicles, are used for processing point cloud data and image data; The image feature extraction module is used to perform horizontal and vertical differences on the input image data to obtain gradient maps, calculate gradient magnitudes, use the global mean of gradient magnitudes as image complexity, dynamically adjust the convolution kernels according to image complexity, and then generate basic features through modulation, bias and offset in deformable convolution, followed by sampling window calculation. Spatial offset sampling is performed on the input image data to obtain a standard feature map; Then, global channel statistics are performed. Global average pooling is called to process the standard feature map after Dropout regularization, compressing the 2D feature map of each channel into a single value to obtain the global statistical information of the corresponding channel. Then, the channel weight generation stage is entered. First, the result of global average pooling is flattened into a 1-dimensional vector. After processing by the Sigmoid activation function, channel weights are generated. Attention weighting is performed. The standard feature map and the generated channel weights are broadcast and multiplied. After convolution, attention-weighted feature map is obtained. Then, a dynamic convolution operation is performed on the weighted feature map to obtain a convolutional feature map. Next, the convolutional feature map is multiplied element-wise with the modulation coefficients in the generated basic features to obtain a refined feature map. Then, the refined feature map is added element-wise with the bias term coefficients in the generated basic features to supplement the required offset, and finally the extracted image feature map is obtained. The feature enhancement module is used to enhance the features of the image feature map to obtain an enhanced image feature map; The point cloud feature extraction module is used to divide the effective region based on point cloud data, and then perform voxel filtering to generate point cloud feature maps. The feature fusion module is used to transform the enhanced image feature map with the assistance of the point cloud feature map and convert it into a unified BEV feature map. The two BEV feature maps are fused using an attention weighting strategy to finally obtain the fused BEV feature map. The 3D object detection box generation module is used to generate the final 3D object detection box based on the fused BEV feature map, combined with heatmap analysis and parametric regression techniques.

Images