Intelligent anti-collision detection method and system for heavy-duty vehicle

Abstract

This invention discloses an intelligent collision avoidance detection method for heavy vehicles, comprising the following steps: S1, acquiring original point cloud data and image data of the vehicle; S2, voxelizing the point cloud data and extracting features from the original point cloud; S3, extracting features from the voxels, compressing the height direction to obtain pseudo-image features of a bird's-eye view; S4, extracting features from the image data; S5, obtaining two pseudo-point clouds with high-dimensional features using bilinear interpolation; S6, fusing the pseudo-point cloud features and the original point cloud features; S7, obtaining point-by-point fused features under spatial attention module processing; S8, performing higher-level feature encoding on the point-by-point fused features; S9, performing fusion feature detection on the high-level information features to obtain the final result. This invention also discloses an intelligent collision avoidance detection system for heavy vehicles. This invention solves the problems of insufficient reaction time for drivers, high false alarm rate, and poor real-time performance in existing technologies, and can be widely applied in the field of intelligent vehicle manufacturing.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle manufacturing, and in particular to an intelligent collision avoidance detection method and system for heavy vehicles. Background Technology

[0002] In recent years, with the increase in car ownership, traffic accidents have caused enormous losses to people's lives and property worldwide. Analysis of numerous traffic accidents reveals that the main influencing factors can be categorized into three aspects: environmental factors, driver factors, and vehicle factors. Among these, driver factors have the greatest impact; many traffic accidents are caused by driver negligence, fatigue, drunk driving, misjudgment, and speeding. Research data indicates that accurate safety warnings can provide drivers with sufficient reaction time to avoid collisions or reduce injuries, potentially preventing at least 60% of rear-end collisions. Therefore, intelligent collision avoidance safety systems are crucial for overall vehicle safety.

[0003] Currently, in-vehicle intelligent driving warning systems mainly utilize two methods: active and passive sensors. Active sensors detect obstacles in the driving environment by actively emitting electromagnetic wave signals (such as millimeter waves and lasers) or ultrasonic signals. Their advantages include high speed and relatively high accuracy, but disadvantages include low spatial resolution, slow scanning speed, high equipment cost, and poor target recognition. Passive sensors primarily use optical cameras. Compared to active devices, optical cameras and modern image processing technology can effectively identify obstacles such as vehicles and pedestrians. They have a wide range of applications, lower installation and implementation costs, and are less dependent on weather and other environmental factors, exhibiting strong adaptability. Therefore, using only a single sensor is insufficient to meet the vehicle's high-precision perception of the external environment, nor can it simultaneously obtain information on the shape, size, and distance of obstacles. Furthermore, the use of single-source sensors suffers from problems such as false alarms, poor real-time performance, and other issues. In conclusion, the real-time performance, accuracy, reliability, and applicability of current vehicle intelligent collision avoidance systems need improvement. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings of the aforementioned background technology and provide a method and system for intelligent collision avoidance detection of heavy vehicles, which solves the problems of insufficient reaction time for drivers, high false alarm rate, and poor real-time performance in the prior art.

[0005] The technical solution adopted in this invention is:

[0006] This invention provides an intelligent collision avoidance detection method for heavy vehicles, comprising the following steps: S1, acquiring raw point cloud data and image data in front of the vehicle using a lidar and a camera respectively; S2, performing voxelization and raw point cloud feature extraction processing on the point cloud data respectively; S3, performing feature extraction processing on the voxels in the three-dimensional space obtained in step S2, and compressing them in the height direction to obtain pseudo-image features under a bird's-eye view; S4, performing feature extraction processing on the image data acquired in step S1 to obtain image features; S5, combining the bird's-eye view features and image features using a double-line... The interpolation method is used to project the data onto the original point cloud to obtain two pseudo-point clouds with high-dimensional features; S6, the features of the two pseudo-point clouds and the original point cloud are fused in the channel attention module to obtain fused features; S7, the fused features are adaptively weighted for foreground and background points in the spatial attention module to obtain point-by-point fused features with foreground enhancement; S8, the point-by-point fused features are subjected to higher-level feature encoding; S9, the features with high-level information are sent to the detection network for detection of the fused features to obtain the final vehicle detection result, so as to inform the driver of collision risks in a timely manner.

[0007] In the above technical solution, the specific process of step S2 is as follows: S21, crop the range of the point cloud and set the length, width and height dimensions of the voxel unit [l voxel w voxel h voxel S21. Divide the point cloud along the X, Y, and Z axes of the point cloud coordinate system to obtain several voxels; S22. Feed the original point cloud into a simple multilayer perceptron for feature encoding to obtain the original point cloud features.

[0008] In the above technical solution, the specific process of step S3 is as follows: S31, for the voxels containing a different number of original points, the average value is calculated to complete the feature encoding of the voxels; S32, the voxels obtained after feature encoding are extracted using 3D sparse convolution, and the four convolution blocks contained in the voxel features are downsampled by 1×, 2×, 4×, and 8× in sequence to obtain the three-dimensional feature map after downsampling by 8 times; S33, the three-dimensional feature map after 3D sparse convolution is compressed and superimposed in the height direction to obtain the pseudo-image features under the bird's-eye view.

[0009] In the above technical solution, the specific features of step S4 are as follows: the network for extracting features from the image is a lightweight DenseNet, which includes an input processing layer, two DenseBlock modules, and a Transition layer; wherein, the input processing layer includes 2D convolutional kernel pooling operations; DenseBlock is a dense connection module that includes two batch normalizations, two activation transformations, and two convolutional processes; the Transition layer is a transition layer connecting the two DenseBlocks to reduce the size of the feature map.

[0010] In the above technical solution, the specific process of step S5 is as follows: S51, establish the original point cloud and the bird's-eye view feature map F bev Camera view feature map F cv The correspondence between them is represented by the transformation matrix M. bev and M cv To represent the mapping relationship between the original point cloud of the LiDAR and the bird's-eye view feature map and the camera's view feature map, suppose there is a point p in the point cloud. i (x i ,y i ,z i The position p of a point in the point cloud within the corresponding feature map can be obtained using the following formula. i '(x i ',y i '): S52. The required point features are obtained using bilinear interpolation, and the calculation formula is as follows: Among them, Q 11 Q 12 Q 21 Q 22 It is the coordinates of a point on a known feature map and its corresponding feature value.

[0011] In the above technical solution, the specific process of step S6 is as follows: S61, the point-by-point features of the bird's-eye view, the point-by-point features of the camera view, and the original point features are concatenated on the channel to obtain the expanded point-by-point features; S62, the expanded features are fed into the three channel attention modules respectively, and each channel attention module uses the same expanded point features to adaptively calculate the importance of its respective channel dimension features in the channel dimension; S63, the feature channel weights are obtained through the Sigmoid function, and the weights are multiplied element-wise with their respective point-by-point features to obtain the attention features, and the calculation formula is as follows: Among them, F cat This indicates that the bird's-eye view features point by point F bev-p Camera view point-by-point features F cv-p and original point features F raw-pExtended features after concatenation along the channel dimension S64 represents the element-wise dot product operator; concatenate the attention features of each view along the channel dimension to obtain the feature F after channel attention fusion. fusion-p The calculation formula for each channel attention module is as follows: M c (F)=σ(MLP(MaxPool(F))+MLP(AvgPool(F))), where F is the input feature, Mc is the channel attention output feature map, MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, MLP represents the parameter-shared fully connected layer, and σ represents the sigmoid activation function.

[0012] In the above technical solution, the specific process of step S7 is as follows: The spatial attention module distinguishes the features of foreground and background points, and the calculation formula is as follows: Among them, M s The output feature map is for spatial attention. MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, Linear represents the linear transformation process, σ represents the sigmoid activation function, and ⊕ represents the feature concatenation operation.

[0013] The above technical solution also includes step S10, which involves inputting the obtained vehicle detection results into the early warning processing module and alerting the driver through the early warning device.

[0014] The present invention also discloses a heavy vehicle intelligent collision avoidance detection system, which stores a computer program that can be executed by a processor, and the computer program can be used to execute a heavy vehicle intelligent collision avoidance detection method.

[0015] The beneficial effects of this invention are as follows: By fusing LiDAR and camera data at the data level, more accurate and richer information about obstacles ahead is obtained, overcoming the shortcomings of single sensors such as high false alarm rates and poor real-time performance. This addresses, to some extent, the difficulty of a single sensor meeting the all-weather, multi-information, high-precision, and highly robust environmental perception requirements of intelligent driving. Furthermore, the rapid obstacle detection based on deep neural networks allows drivers more reaction time, making the collision avoidance braking process of heavy vehicles safer and more reliable, thus reducing the risk of collisions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an overall flowchart of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0018] Figure 2 This is a flowchart illustrating a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0019] Figure 3 This is a schematic diagram of the workflow of the point cloud bird's-eye view feature extraction network in a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0020] Figure 4 This is a schematic diagram of the workflow of the bird's-eye view feature extraction network in a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0021] Figure 5 This is a diagram of the image feature extraction backbone network structure in a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0022] Figure 6 This is a structural diagram of the channel attention point-by-point fusion module in a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0023] Figure 7 This is a spatial attention network structure diagram in a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles of the present invention;

[0024] Figure 8 This is a schematic diagram of the structure of the intelligent collision avoidance detection system for heavy vehicles of the present invention;

[0025] Figure 9 This is a schematic diagram of the physical scene during the operation of the intelligent collision avoidance detection system for heavy vehicles of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] Figure 1 A flowchart of a heavy vehicle intelligent collision avoidance detection method provided by the present invention includes the following steps:

[0028] Step 101: Use LiDAR and camera to collect raw point cloud data and image data in front of the vehicle, respectively;

[0029] The raw point cloud data includes vehicles in front of the vehicle and static obstacles, and the image data is an RGB view map in front of the vehicle, which includes vehicles in motion and other obstacles.

[0030] Step 102: Perform voxelization and original point cloud feature extraction on the point cloud data respectively;

[0031] 2.1) First, the point cloud area is cropped, then the length, width, and height dimensions of the voxel unit are set [l voxel w voxel h voxel Then, the points are divided along the X, Y, and Z axes of the point cloud coordinate system to obtain several voxels.

[0032] 2.2) The original point cloud is fed into a simple MLP (Multilayer Perceptron) for feature encoding to obtain the original point cloud features.

[0033] Step 103: Extract features from the voxels in the three-dimensional space obtained in Step 102 and compress them in the height direction to obtain pseudo-image features under the bird's-eye view.

[0034] 3.1) A voxel contains a varying number of raw points. The features of these raw points are aggregated and averaged to obtain the feature code of the voxel.

[0035] 3.2) Feature extraction is performed on the voxels obtained after feature encoding. Since the feature is a three-dimensional feature in three-dimensional space, 3D sparse convolution is used to complete the feature extraction of non-empty voxels, which can greatly reduce the computational cost. The feature network backbone of the three-dimensional voxel contains four convolutional blocks, which perform downsampling processing of voxel features by 1×, 2×, 4×, and 8× in sequence to obtain the three-dimensional feature map after downsampling by 8 times.

[0036] 3.3) The 3D feature map after 3D sparse convolution is compressed and superimposed in the height direction to obtain the pseudo image features under the bird's-eye view.

[0037] Step 104: Perform feature extraction processing on the image data acquired in Step 101 to obtain image features;

[0038] The network used for feature extraction from images is the lightweight DenseNet. DenseNet is a residual network with dense connections. The difference between DenseNet and the original DenseNet is that the lightweight DenseNet includes an input processing layer, two DenseBlock modules, and a Transition layer. The input processing layer includes 2D convolutional kernel pooling operations. The DenseBlock is a densely connected module that includes two batch normalizations, two activation transformations, and two convolutional processes. The Transition layer is a transition layer connecting the two DenseBlocks, which can reduce the size of the feature map.

[0039] Step 105: Project the bird's-eye view features and image features onto the original point cloud using bilinear interpolation to obtain two pseudo point clouds with high-dimensional features;

[0040] 5.1) Establish the original point cloud and bird's-eye view feature map F bev Camera view feature map F cv The correspondence between them is represented by the transformation matrix M. bev and M cv To represent the mapping relationship between the original point cloud of the LiDAR and the bird's-eye view feature map and the camera's view feature map, suppose there is a point p in the point cloud. i (x i ,y i ,z i The position p of a point in the point cloud within the corresponding feature map can be obtained using the following formula. i '(x i ',y i '):

[0041]

[0042] 5.2) After mapping the original points to their corresponding positions in the bird's-eye view and camera view feature maps, bilinear interpolation is used to obtain the required point-by-point features. The calculation formula is as follows:

[0043]

[0044] Among them, Q 11 Q 12 Q 21 Q 22 It is the coordinates of a point on a known feature map and its corresponding feature value.

[0045] Step 106: Fuse the two sets of pseudo-point cloud features and the original point cloud features in the channel attention module to obtain the fused features;

[0046] 6.1) The point-by-point features of the bird's-eye view, the point-by-point features of the camera view, and the original point features are stitched and superimposed in the channel dimension to obtain the expanded point-by-point features;

[0047] 6.2) First, construct three different channel attention modules. Then, input the expanded features into the channel attention module simultaneously. Finally, adaptively calculate the importance of the features in each channel dimension.

[0048] 6.3) A Sigmoid function is added after each channel attention module. After passing through the Sigmoid function, the range of each feature value is mapped to between 0 and 1, which is the weight of each channel. The larger the value, the higher the importance. Then, the features under each viewpoint are multiplied element-wise with the corresponding weights to obtain the channel attention features. The calculation formula is as follows:

[0049]

[0050] Where Fcat represents the point-by-point feature F of the bird's-eye view. bev-p Camera view point-by-point features F cv-p and original point features F raw-p Extended features after concatenation along the channel dimension This represents the element-wise dot product operator.

[0051] 6.4) Concatenate the attention features of each view along the channel dimension to obtain the feature F after channel attention fusion. fusion-p The calculation formula for each channel attention module is as follows:

[0052] M c (F)=σ(MLP(MaxPool(F))+MLP(AvgPool(F))),

[0053] Where F is the input feature, M c The feature map is output for channel attention. MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, MLP represents a fully connected layer with shared parameters, and σ represents the sigmoid activation function.

[0054] Step 107: Adaptively weight the fused features with foreground and background points in the spatial attention module to obtain point-by-point fused features with enhanced foreground points;

[0055] The spatial attention module can distinguish features at different locations (foreground and background points), and the calculation formula is as follows:

[0056]

[0057] Among them, M s The output feature map is for spatial attention. MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, Linear represents the linear transformation process, σ represents the sigmoid activation function, and ⊕ represents the feature concatenation operation.

[0058] Step 108: Perform higher-level feature encoding on the point-by-point fused features;

[0059] The network used is the lightweight PointNet++, which includes sampling layers and fully connected layers to further encode the fused features, resulting in high-dimensional point cloud fused features for final detection.

[0060] Step 109: The features with high-level information are fed into the detection network for feature fusion detection to obtain the final vehicle detection result; the detection network here adopts the 3DSSD detection head network, which can output three-dimensional detection boxes and distance information on the point cloud map.

[0061] Step 110: Input the obtained vehicle detection results into the warning processing module and alert the driver through the display and buzzer; the detection results include the position and relative distance of the vehicle in front, and the driver should brake the vehicle in time according to the warning information to prevent a collision.

[0062] like Figure 8 As shown, the system includes a data acquisition module, a computing and storage module, and a warning processing module. The data acquisition module includes a LiDAR and a camera. The computing and storage module is implemented using a development board. A pre-trained and optimized vehicle detection model with fused multimodal features is deployed in the development board's memory, and the model is run on the development board to obtain detection results. The warning processing module displays the position and relative distance of vehicles ahead on a monitor and conveys warning signals to the driver through the monitor and a buzzer. The specific steps of the system are as follows:

[0063] Step 1: Train and optimize a vehicle detection model that fuses multimodal features using the KITTI vehicle dataset;

[0064] Step 2: Deploy and store the trained multimodal feature fusion vehicle detection model in the development board's memory;

[0065] Step 3: Based on the lidar and camera in the data acquisition module, acquire point cloud and image input data respectively, and transmit the input data to the processor of the development board;

[0066] Step 4: The processor preprocesses the input raw point cloud and raw image data, and loads the detection inference code in memory into the inference calculation module.

[0067] Step 5: Input the data into the inference calculation module and run the detection model deployed in Step 2;

[0068] Step 6: The inference and calculation module outputs the detection results, and the display module outputs the position and relative distance of the vehicle in the point cloud image.

[0069] Before the system can be put into operation, it requires model training, testing, and deployment. Model training is typically performed on devices with high-performance data processing capabilities (such as computers). The computer reads existing multimodal vehicle detection datasets and uses the aforementioned vehicle detection method based on multimodal feature fusion to train and test the model. Afterward, the optimized model parameters are saved, and the model is deployed on hardware devices such as the development board of the detection system for use. When the heavy-duty vehicle intelligent collision avoidance warning system is working, it first loads the detection model deployed in memory, along with point cloud data and image data. Each frame of data is preprocessed and then input into the detection model. The inference module outputs the vehicle detection results, which are finally input into the warning processing module to alert the driver to take timely braking action to prevent collisions.

[0070] Figure 2 The flowchart is a specific embodiment of the intelligent collision avoidance detection method for heavy vehicles based on the present invention, as shown below. Figure 2 As shown, the specific process is as follows:

[0071] A) Feature Extraction from Point Cloud BEV View: A point cloud bird's-eye view projects a 3D point cloud onto a plane viewed from above. This view effectively preserves the physical dimensions, geometric information, and relative positional relationships of objects, such as... Figure 3 The diagram shows the workflow of a point cloud bird's-eye view feature extraction network. The specific steps are as follows:

[0072] Point cloud voxelization: Before voxelization, the point cloud needs to be cropped by clipping along the X, Y, and Z axes of the LiDAR coordinate system. The range along the X axis is [0m, 70.4m], the range along the Y axis is [-40m, 40m], and the range along the Z axis is [-3m, 1m]. Then, the length, width, and height dimensions of the voxel are set to [l...]. voxel w voxel h voxel In this embodiment, the size of each voxel is selected as [0.05, 0.05, 0.1]. Therefore, within this three-dimensional point cloud space, a set of voxels with a quantity of 1408×1600×40 can be divided. The maximum number of three-dimensional points in each voxel is M. Each original three-dimensional point has features in four dimensions, namely (x, y, z, i).

[0073] Voxel feature encoding: First, sum all points within each voxel by adding them element by element along the same dimension; then, regularize the number of points within each voxel to ensure that each voxel has at least one point and prevent division by zero; then, calculate the average value of the coordinates and reflection intensity within each voxel, and use this value to represent the feature of the corresponding voxel, thus obtaining the features of each voxel, with a data dimension of (N, 4).

[0074] Voxel feature extraction: 3D sparse convolution is used to extract features from non-empty voxels. The backbone of the feature extraction network for the point cloud bird's-eye view has four convolutional blocks, such as... Figure 4 As shown, the voxel features are downsampled by 1×, 2×, 4×, and 8× in sequence to obtain a three-dimensional feature map after downsampling by 8 times. Then, a convolutional output layer is passed to increase the feature dimension to 128, which helps to retain more and more detailed feature information. The final dimension is (128, [2, 200, 176]), where [2, 200, 176] represents the size of the voxel.

[0075] Bird's-eye view feature map generation: Considering that there is no stacking of 3D objects in the Z-axis direction during vehicle movement, the feature map after 3D sparse convolution is compressed and superimposed in the height direction to obtain a pseudo image in the bird's-eye view. Specifically, the features of two layers of voxels in the Z-axis direction are stitched together to obtain a pseudo image with a dimension of 128×2 and a size of 200×176. That is, the generated pseudo image is a dense tensor (N, 256, 200, 176).

[0076] B) Image Feature Extraction: In this embodiment, the lightweight DenseNet is selected as the backbone network for image feature extraction, such as... Figure 5 As shown, the network consists of an input processing layer, two DenseBlocks, and a Transition layer. Each DenseBlock contains several "dense layers," and each "dense layer" performs two batch normalizations, two activation transformations, and two convolutions. Connecting two adjacent DenseBlocks is a Transition layer, which serves as a transition and reduces the size of the feature map. It mainly contains a 1×1 convolutional layer and a 2×2 average pooling layer. In the convolutional layer, the number of feature channels is reduced to half of the input features, which can compress the model. The detailed parameters of the image feature extraction backbone network are shown in Table 1.

[0077] Table 1 Detailed parameters of the image feature extraction backbone network

[0078]

[0079] C) Original point cloud feature extraction: The original point cloud is fed into a simple MLP (Multilayer Perceptron) for feature encoding to obtain the original point cloud features.

[0080] D) Point-by-point feature fusion based on channel attention: Feature-level fusion of bird's-eye view features, image view features, and original point cloud features is performed. The specific steps are as follows:

[0081] Establish original point cloud and bird's-eye view feature map F bev Camera view feature map F cv The correspondence between them is represented by the transformation matrix M. bev and M cv To represent the mapping relationship between the original point cloud of the LiDAR and the bird's-eye view feature map and the camera's view feature map, suppose there is a point p in the point cloud. i (x i ,y i ,z i The position p of a point in the point cloud within the corresponding feature map can be obtained using the following formula. i '(x i ',y i '):

[0082]

[0083] After mapping the original points to their corresponding positions in the bird's-eye view and camera view feature maps, bilinear interpolation is used to obtain the required point-by-point features. The calculation formula is as follows:

[0084]

[0085] Among them, Q 11 Q 12 Q 21 Q 22 It is the coordinates of a point on a known feature map and its corresponding feature value.

[0086] The point-by-point features from the bird's-eye view, the camera view, and the original point features are concatenated across channels to obtain expanded point-by-point features. First, three different channel attention modules are constructed, such as... Figure 6 As shown, the expanded features are then simultaneously input into the channel attention module, and the importance of each channel dimension feature is adaptively calculated along the channel dimension. A sigmoid function is added after each channel attention module. After passing through the sigmoid function, the range of each feature value is mapped to between 0 and 1, which is the weight of each channel. The larger the value, the higher the importance. Then, the features from each perspective are multiplied element-wise with the corresponding weights to obtain the channel attention features. The calculation formula is as follows:

[0087]

[0088] Among them, F cat This indicates that the bird's-eye view features point by point F bev-p Camera view point-by-point features F cv-p and original point features F raw-p Extended features after concatenation along the channel dimension This represents the element-wise dot product operator.

[0089] The attention features of each view are concatenated along the channel dimension to obtain the feature F after channel attention fusion. fusion-p The calculation formula for each channel attention module is as follows:

[0090] M c (F)=σ(MLP(MaxPool(F))+MLP(AvgPool(F))),

[0091] Where F is the input feature, M c The feature map is output for channel attention. MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, MLP represents a fully connected layer with shared parameters, and σ represents the sigmoid activation function.

[0092] First, the input features are processed by global max pooling and global average pooling along the spatial direction, resulting in global max pooling vectors and global average pooling vectors. These two vectors represent the most significant and average statistical features in each channel of the feature map, respectively, with a dimension of 1×C. Next, the two vectors are fed into a parameter-shared multilayer perceptron to model the relationships between channels. Specifically, dimensionality reduction is performed first through a fully connected layer, then an activation function is used to activate the dimensionality-reduced vectors, and finally, a fully connected layer is used again to increase the dimensionality to restore the number of channels at the input. Then, the two feature vectors are concatenated using feature addition, specifically element-wise addition. Finally, the 1×C feature vectors are mapped using the Sigmoid function to a channel attention vector with values ​​ranging from 0 to 1. Each value in the vector represents the importance of the corresponding channel. This attention vector is then multiplied by the input features channel by channel to obtain the features processed by the channel attention network.

[0093] E) Point-by-point weighting based on spatial attention: To enable the neural network to automatically learn which regions need to be focused on and to enhance the features within those regions, its detailed network structure is as follows: Figure 7 As shown, the calculation process can be described using the following formula.

[0094]

[0095] In the formula, M s The output feature map is for spatial attention. MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, Linear represents the linear transformation process, σ represents the sigmoid activation function, and ⊕ represents the feature concatenation operation.

[0096] The fused features processed by the channel attention module are fed into the spatial attention module. First, to aggregate spatial features more efficiently and fully, the input features are subjected to max pooling and average pooling along the channel dimension, respectively, to obtain a global max pooling feature map and a global average pooling feature map of dimension N×1. This feature map represents the most salient and average statistical features of each location in the space across all channels. Next, to efficiently use the average and maximum features, the two feature maps are concatenated along the channel direction to obtain a feature map of dimension N×2. Then, a linear layer is used to replace the original 1×1 convolution to aggregate the two different statistical features, obtaining a single-channel attention feature map of dimension N×1 that can represent different importance levels at different locations. Finally, the sigmoid activation function is used to map the single-channel feature map into a spatial attention feature map with values ​​ranging from 0 to 1. The value of each element in this feature map represents the importance of the location. The more target objects in the region, the larger the value; conversely, the smaller the value, the less relevant the information at that location, or even noise interference. The spatial attention feature map is multiplied with the input feature in the spatial dimension to obtain the feature map after processing by the spatial attention module.

[0097] F) Point-by-point feature encoding and fusion feature detection: A lightweight PointNet++ network is used, which includes sampling layers and fully connected layers, to further encode the fusion features and obtain high-dimensional point cloud fusion features for final detection; features with high-level information are fed into the detection network for fusion feature detection to obtain the final vehicle detection result; the detection network uses the 3DSSD detection head network, which can output three-dimensional detection boxes and distance information on the point cloud map.

[0098] G) Result Output and Driver Warning: The obtained vehicle detection results are input into the warning processing module and the driver is alerted through the display and buzzer; the detection results include the position and relative distance of the vehicle in front, and the driver can brake the vehicle in time according to the warning information to prevent a collision.

[0099] like Figure 8As shown, the system includes a data acquisition module, a computing and storage module, and a warning processing module. The data acquisition module includes a LiDAR and a camera. The computing and storage module is implemented using a development board. A pre-trained and optimized vehicle detection model with fused multimodal features is deployed in the development board's memory, and the model is run on the development board to obtain detection results. The warning processing module displays the position and relative distance of vehicles ahead on a monitor and conveys warning signals to the driver through the monitor and a buzzer. The specific steps of the system are as follows:

[0100] Step 1: Train and optimize a vehicle detection model that fuses multimodal features using the KITTI vehicle dataset;

[0101] Step 2: Deploy and store the trained multimodal feature fusion vehicle detection model in the development board's memory;

[0102] Step 3: Based on the lidar and camera in the data acquisition module, acquire point cloud and image input data respectively, and transmit the input data to the processor of the development board;

[0103] Step 4: The processor preprocesses the input raw point cloud and raw image data, and loads the detection inference code in memory into the inference calculation module.

[0104] Step 5: Input the data into the inference calculation module and run the detection model deployed in Step 2;

[0105] Step 6: The inference and calculation module outputs the detection results, and the display module outputs the position and relative distance of the vehicle in the point cloud image.

[0106] Before the system can be put into operation, it requires model training, testing, and deployment. Model training is typically performed on devices with high-performance data processing capabilities (such as computers). The computer reads existing multimodal vehicle detection datasets and uses the aforementioned vehicle detection method based on multimodal feature fusion to train and test the model. Afterward, the optimized model parameters are saved, and the model is deployed on hardware devices such as the development board of the detection system for use. When the heavy-duty vehicle intelligent collision avoidance warning system is working, it first loads the detection model deployed in memory, along with point cloud data and image data. Each frame of data is preprocessed and then input into the detection model. The inference module outputs the vehicle detection results, which are finally input into the warning processing module to alert the driver to take timely braking action to prevent collisions.

[0107] like Figure 9As shown in the figure, the heavy vehicle body 1, a vehicle 2 traveling in front, a vehicle 3 traveling behind, and a lidar 4 and a camera 5 installed on the heavy vehicle body 1. The lidar 4 is installed on the roof of the heavy vehicle body 1 and can perform 360° scanning. The camera 5 is installed in front of the heavy vehicle body 1. The lidar 4 and the camera 5 respectively acquire the original point cloud data and image data of the vehicle 2 traveling in front, and input the data into the heavy vehicle intelligent collision avoidance detection system of the present invention to complete the detection, and finally obtain the warning information to achieve the purpose of safe obstacle avoidance.

[0108] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for intelligent collision avoidance detection of heavy vehicles, characterized in that, Includes the following steps: S1. Use lidar and camera to collect raw point cloud data and image data in front of the vehicle, respectively; S2. Perform voxelization and original point cloud feature extraction processing on the point cloud data respectively; S3. Extract features from the voxels in the three-dimensional space obtained in step S2 and compress them in the height direction to obtain pseudo-image features under the bird's-eye view. S4. Perform feature extraction processing on the image data obtained in step S1 to obtain image features; S5. Project the bird's-eye view features and image features onto the original point cloud using bilinear interpolation to obtain two pseudo point clouds with high-dimensional features. S6. The two pseudo-point cloud features and the original point cloud features are fused in the channel attention module to obtain the fused features. The specific process is as follows: S61. The point-by-point features of the bird's-eye view, the point-by-point features of the camera view, and the original point features are stitched together on the channel to obtain the expanded point-by-point features. S62. The expanded features are fed into the three channel attention modules respectively. Each channel attention module uses the same expanded point features to adaptively calculate the importance of the features in its respective channel dimension. S63. Obtain the feature channel weights using the Sigmoid function, and then multiply the weights element-wise with the respective point-by-point features to obtain the attention features. The calculation formula is as follows: ， Among them, F cat This indicates that the bird's-eye view features point by point F bev-p Camera view point-by-point features F cv-p and original point features F raw-p Extended features after concatenation along the channel dimension This represents the element-wise dot product operator. S64. Concatenate the attention features of each view along the channel dimension to obtain the feature F after channel attention fusion. fusion-p The calculation formula for each channel attention module is as follows: ， Where F is the input feature, Mc is the channel attention output feature map, MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, MLP represents the parameter-shared fully connected layer, and σ represents the sigmoid activation function. S7. The fusion features are adaptively weighted by foreground and background points in the spatial attention module to obtain point-by-point fusion features with enhanced foreground points. S8. Perform higher-level feature encoding on the point-by-point fused features; S9. Features with high-level information are sent to the detection network for feature fusion detection to obtain the final vehicle detection result, so as to inform the driver of the collision risk in a timely manner.

2. The intelligent collision avoidance detection method for heavy vehicles according to claim 1, characterized in that, The specific process of step S2 is as follows: S21. Crop the point cloud area and set the length, width, and height dimensions of the voxel units [l voxel w voxel h voxel The points are divided along the X, Y, and Z axes of the point cloud coordinate system to obtain several voxels. S22. Feed the original point cloud into a simple multilayer perceptron for feature encoding to obtain the original point cloud features.

3. The intelligent collision avoidance detection method for heavy vehicles according to claim 2, characterized in that, The specific process of step S3 is as follows: S31. For voxels containing a different number of original points, perform average calculation to complete the feature encoding of the voxels. S32. Use 3D sparse convolution to extract features from the voxels obtained after feature encoding. Then, perform downsampling processing of 1×, 2×, 4×, and 8× on the four convolution blocks contained in the voxel features in sequence to obtain the three-dimensional feature map after downsampling by 8 times. S33. Compress and superimpose the 3D feature map after 3D sparse convolution in the height direction to obtain the pseudo-image features under the bird's-eye view.

4. The intelligent collision avoidance detection method for heavy vehicles according to claim 3, characterized in that, The specific features of step S4 are as follows: The network used for feature extraction of images is the lightweight DenseNet, which includes an input processing layer, two DenseBlock modules, and a Transition layer. in, The input processing layer includes 2D convolutional kernel pooling operations; DenseBlock is a densely connected module that includes two batch normalizations, two activation transformations, and two convolution processes. The transition layer is a transition layer connecting two DenseBlocks, reducing the size of the feature map.

5. The intelligent collision avoidance detection method for heavy vehicles according to claim 4, characterized in that, The specific process of step S5 is as follows: S51. Establish the original point cloud and bird's-eye view feature map F bev Camera view feature map F cv The correspondence between them is represented by the transformation matrix M. bev and M cv To represent the mapping relationship between the original point cloud of the LiDAR and the feature maps from the bird's-eye view and the camera's view, let's assume a certain point in the point cloud... The position of a point in the point cloud within the corresponding feature map can be obtained using the following formula. : ， S52. The required point features are obtained using bilinear interpolation, and the calculation formula is as follows: ， Among them, Q 11 Q 12 Q 21 Q 22 It is the coordinates of a point on a known feature map and its corresponding feature value.

6. The intelligent collision avoidance detection method for heavy vehicles according to claim 5, characterized in that, The specific process of step S7 is as follows: The spatial attention module distinguishes the features of foreground and background points, and the calculation formula is as follows: ， Among them, M s The output feature map is for spatial attention. MaxPool represents the global max pooling operation, AvgPool represents the global average pooling operation, Linear represents the linear transformation process, σ represents the sigmoid activation function, and ⊕ represents the feature concatenation operation.

7. The intelligent collision avoidance detection method for heavy vehicles according to claim 6, characterized in that: It also includes step S10, inputting the obtained vehicle detection results into the early warning processing module and alerting the driver through the early warning device.

8. A heavy vehicle intelligent collision avoidance detection system, wherein a computer program executable by a processor is stored therein, the computer program being able to perform the heavy vehicle intelligent collision avoidance detection method as described in any one of claims 1 to 7.

Images