Camera and lidar based multi-class object detection method

Abstract

The application discloses a multi-class target detection method based on a camera and a laser radar, and mainly solves the problem of low detection accuracy of an existing target detection method.The implementation scheme is as follows: acquiring a road surface image and point cloud data;performing voxelization preprocessing on the point cloud data;strengthening the spatial information of the preprocessed point cloud;performing color information fusion on the spatial information strengthened point cloud;obtaining a double-strengthened pseudo image through convolution on the fused point cloud;obtaining a to-be-detected feature map according to the double-strengthened pseudo image;and sending the to-be-detected feature map into an SSD detector to generate a detection result of a front target in the process of automobile driving.The application strengthens the spatial information of the point cloud by establishing a modal internal mapping matrix and sampling, and fuses the color information of an RGB image with the point cloud through size adjustment and a pseudo view transformation matrix, so that the accuracy of target detection is improved, and the application can be used for automatic driving of an unmanned automobile.

Description

Technical Field

[0001] This invention belongs to the field of physical technology, and further relates to a multi-target detection method that can be used for autonomous driving of unmanned vehicles. Background Technology

[0002] Cameras, as the most commonly used sensors in autonomous vehicle driving systems, capture images containing color information of the external environment, playing a crucial role in target detection. However, their drawbacks include a lack of depth information and susceptibility to environmental conditions. LiDAR, on the other hand, offers advantages such as low dependence on lighting conditions, resistance to adverse weather conditions, and high detection accuracy. It generates point cloud data, which, by analyzing and processing, provides more detailed information on target shape and location, helping the vehicle better understand its environment and offering more stable and accurate perception results. Therefore, the fusion of data from cameras and LiDAR to detect targets in front of the vehicle, such as other cars, pedestrians, and cyclists, has become a growing trend in autonomous vehicles.

[0003] Currently, data fusion algorithms for cameras and LiDAR sensors are categorized into three types: front-end fusion, deep fusion, and back-end fusion. Front-end fusion primarily targets the data level, fusing data from different modalities into a single feature vector before input for subsequent operations. This data can be either raw sensor data or pre-processed data. Deep fusion focuses on the feature level, extracting features from the raw data of different modalities and transforming it into a high-dimensional feature representation before performing interactive fusion operations at different feature layers. Back-end fusion targets the decision level, processing the raw data of different modalities using their respective networks, outputting classification scores, and then fusing the scores.

[0004] Zhongcheng Hualong Computer Technology Co., Ltd. disclosed a "Multi-sensor Data Fusion-Based Autonomous Driving Decision-Making Method and SoC Chip" in its patent application (CN202211082826). This method inputs point cloud data into a trained point cloud target detection neural network model for obstacle detection. Although this method uses a camera, it only utilizes point cloud data collected by LiDAR for obstacle detection, failing to fully leverage sensor data. The obtained images are used only for lane detection, ignoring the importance of color information for obstacles, thus hindering efficient obstacle detection.

[0005] Srun Tianlang (Beijing) Technology Co., Ltd. disclosed a "V2X Multi-Sensor Fusion Method and Device Based on Scene Awareness" in patent application number CN202211314591. This method first performs time-synchronized preprocessing on the acquired sensor data, then compares the two types of preprocessed data to obtain the correlation coefficient between the sensors and the confidence level of each sensor, thereby determining the scene. Finally, based on the determined scene and the types of input sensors, it determines the fusion weights of each sensor in a preset neural network to obtain the fusion result. However, because this method fuses different sensor data at the decision-making level, it ignores the relationships between different sensor data, thus failing to complement the characteristics of different modalities and ultimately reducing the accuracy of target detection. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a multi-target detection method based on cameras and lidar. This method aims to solve the drawbacks of single lidar target detection algorithms that lack color information and the shortcomings of decision-level fusion that cannot fully consider the characteristic relationships of different modal data, thereby improving the accuracy of target detection.

[0007] The technical approach of this invention is as follows: by setting up an image auxiliary processing branch for RGB images, and setting up a main point cloud processing branch and a point cloud auxiliary processing branch for point clouds, the problem of lack of color information in single-laser radar target detection is solved; by using single-modal self-fusion, the spatial information of the point cloud auxiliary processing branch can be fused into the point cloud main processing branch, thereby solving the problem that decision-level fusion cannot complement the characteristics between different modal data.

[0008] Based on the above approach, the implementation steps of this invention include the following:

[0009] (1) Data is obtained through sensors respectively, the camera obtains RGB image F1, and the lidar obtains point cloud R1(x,y,z,r);

[0010] (2) Perform voxelization preprocessing on the point cloud R1(x,y,z,r) to obtain the original pseudo-image P:

[0011] (3) The spatial information of the pseudo-image P is enhanced by using a single-modal self-fusion method to generate a spatially enhanced pseudo-image Ps. i :

[0012] (3a) Point cloud features R i Register with pseudo-image features P to generate an intramodal mapping matrix M. RP :

[0013] M RP =R i / P i ;

[0014] (3b) Based on the sampling position p' and the intramodal mapping matrix M RP To generate a feature representation V RP :

[0015]

[0016] Where K represents the bilinear interpolation function, This represents the features of the adjacent pixels at sampling position p';

[0017] (3c) Using the SetAbstraction sampling operation to sample point cloud features R i Feature extraction is performed to generate the point cloud features R to be fused. i ';

[0018] (3d) Represent the features V RP and the point cloud features R to be fused i 'Point-by-point fusion to generate fused point cloud features R' i ":

[0019] R i =σ(Wtanh(UV) RP +VR i ))

[0020] Where W, U, and V are three learnable weight matrices with different values, σ represents the sigmoid activation function, and tanh represents the hyperbolic tangent function;

[0021] (3e) For the fused point cloud features R i "A spatially enhanced pseudo-image Ps is obtained using two fully connected layers FC()." i :

[0022] Ps i =P i ×FC(FC(R i ”))

[0023] (4) Convert the RGB image F i Color information is incorporated into spatially enhanced pseudo-images (Ps) i Generate color-enhanced pseudo-images on PC i :

[0024] (4a) For RGB image F i The size was adjusted to enhance the size and spatial composition of the pseudo-image (Ps). i They are the same size;

[0025] (4b) Resize the RGB image F i Spatial Enhancement Ps iPerform BatchNorm and ReLU operations respectively, and then connect them along the channel dimension to obtain the graph PF to be transformed;

[0026] (4c) Generate the spatial factor matrix M from the graph to be transformed, PF, using the spatial attention formula and the channel attention formula respectively. s and channel factor matrix M c And obtain the pseudo-view transformation matrix M cs :

[0027] M cs =0.6*M c +0.4*M s ;

[0028] (4d) Transform the pseudoview matrix M cs With RGB image F i Multiplication yields the viewpoint transformation pseudo-image Pv i And put the pseudo image Pv i Spatially enhanced pseudo-image Ps i The images are stitched together along the channel dimension to generate a double-enhanced pseudo-image Pc. i ;

[0029] (5) Based on the double-enhanced pseudo-image Pc i Obtain a double-enhanced pseudo-image:

[0030] (5a) For the double-enhanced pseudo-image Pc i Convolution is performed to obtain a downsampled, double-enhanced pseudo-image Pc. i+1 Return to step (3);

[0031] (5b) Repeat step (5a) to finally obtain the double-enhanced pseudo-image Pc3;

[0032] (6) Perform two transposed convolutions on the double-enhanced pseudo-image Pc3 to obtain two transposed feature maps Pt1 and Pt2. Concatenate the double-enhanced pseudo-image Pc3 with the two transposed feature maps Pt1 and Pt2 to obtain the feature map F to be detected. U ;

[0033] (7) The feature map to be detected F U The data is fed into the SSD detector to generate detection results of targets ahead as the car moves.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] First, this invention utilizes a single-modal self-fusion method to enhance the spatial information of pseudo-images by establishing an intramodal mapping matrix and SetAbstraction sampling. This enhances the spatial representation ability of the target by the pseudo-images, overcomes the drawback of existing technologies in losing useful spatial information during voxelization, and improves the accuracy of target detection.

[0036] Secondly, this invention utilizes a dual-modal cross-fusion method to fuse the color information of RGB images with the spatial information of point clouds by adjusting the size and generating a pseudo-view transformation matrix, thereby further improving the accuracy of target detection. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 This is a schematic diagram of single-modal self-fusion in this invention;

[0039] Figure 3 This is a schematic diagram of the dual-modal cross-fusion in this invention. Detailed Implementation

[0040] The embodiments and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0041] Reference Figure 1 The implementation steps for this example are as follows.

[0042] Step 1: Obtain road surface images and point cloud data.

[0043] In driving scenarios, vehicle detection of multiple targets requires the acquisition of road surface images and point cloud data via sensors. This information can be obtained using a camera to acquire RGB images (F1) and a LiDAR to obtain point clouds (R1(x,y,z,r)). In this embodiment, the internationally published KITTI dataset is used as input to simulate data acquisition by sensors in a real-world scenario. The KITTI dataset is one of the most authoritative computer vision evaluation datasets internationally, containing real-world images and point cloud data collected from urban, rural, and highway scenes.

[0044] Step 2: Perform voxelization preprocessing on the point cloud data.

[0045] The point cloud R1 contains approximately 20,000 data points, including a large number of noisy and redundant points. To reduce the computational load on the system, the point cloud R1(x,y,z,r) first needs to be preprocessed by voxelization to obtain a raw pseudo-image P that can express the important information of the point cloud. The specific implementation is as follows:

[0046] 2.1) In the z direction, from z = 0 m to z = 4 m, the point cloud space is divided into four height spaces of equal height, namely height spaces S1 (z1 = [0,1) m), S2 (z2 = [1,2) m), S3 (z3 = [2,3) m), and S4 (z4 = [3,4] m);

[0047] 2.2) In the xy direction, the point cloud in each height space is divided into regular columnar sub-point clouds with a base size of 0.16m × 0.16m. In this embodiment, height spaces S1, S2, S3, and S4 are all divided into 496 × 432 regular columnar sub-point clouds.

[0048] 2.3) Set a sampling threshold D. Randomly sample the columnar sub-point clouds with more points than the threshold, and fill the columnar sub-point clouds with fewer points than the threshold with zeros. In this embodiment, D = 32. If the number of points in the 496×432 regular columnar sub-point clouds in each height space is greater than 32, then 32 points are randomly sampled; if it is less than 32, the missing points are filled with 0s.

[0049] 2.4) Calculate the arithmetic mean (xi) of all points within each columnar sub-point cloud. c ,y c ,z c ) and offset (x) p ,y p The enhanced point cloud (x,y,z,r,x) is obtained. c ,y c ,z c ,x p ,y p );

[0050] 2.5) Augmenting point clouds (x,y,z,r,x) using max pooling and a simplified PointNet network. c ,y c ,z c ,x p ,y p Encode the data to generate a sub-pseudo-image p corresponding to each height space. i In this embodiment, the sub-pseudo-images corresponding to the height spaces S1, S2, S3, and S4 are p1, p2, p3, and p4, respectively. The size of each sub-pseudo-image is 496×432.

[0051] 2.6) Obtain height dimension weights and channel dimension weights using four fully connected layers:

[0052] 2.6.1) Compress the pseudo-image p using the first fully connected layer W1. i The height dimension is then extracted using the second fully connected layer W2, and its height dimension weight S is further extracted. i :

[0053] S i =W2δ(W1p i )

[0054] Where δ() represents the activation function;

[0055] 2.6.2) Compressing the pseudo-image p using the third fully connected layer W3. i The channel dimension is then used to extract its channel dimension weight T using the fourth fully connected layer W4. i :

[0056] T i =W4δ(W3p i )

[0057] Where δ() represents the activation function;

[0058] 2.7) Transfer the four sub-pseudo-images p i Its corresponding height dimension weight S i and channel dimension weight T i Multiply the images, then concatenate them along the channel dimension and perform max pooling to obtain the final original pseudo-image P.

[0059] In this embodiment, the shape of the four sub-pseudo-images after being stitched together in the channel dimension is [4,496,432]. By performing max pooling in the channel dimension, the final shape of the original pseudo-image P is [496,432].

[0060] Step 3: Enhance the spatial information of the preprocessed point cloud.

[0061] During voxelization, point clouds lose some spatial information, which reduces the accuracy of target detection. Therefore, it is necessary to enhance the spatial information of the pseudo-image P. This example enhances the spatial information of the pseudo-image P through registration, sampling, and fusion, ultimately generating a spatially enhanced pseudo-image Ps. i .

[0062] Reference Figure 2 The specific implementation of this step is as follows:

[0063] 3.1) Transfer the point cloud features R i Register with pseudo-image features P to generate an intramodal mapping matrix M. RP :

[0064] M RP =R i / P i ;

[0065] 3.2) Based on the intramodal mapping matrix M RP And sampling position p', generate a feature representation V RP :

[0066]

[0067] Where K represents the bilinear interpolation function, This represents the features of the adjacent pixels at sampling position p';

[0068] 3.3) Using the SetAbstraction sampling operation to sample point cloud features R i Feature extraction is performed to generate the point cloud features R to be fused. i Its specific implementation is as follows:

[0069] 3.3.1) Use the farthest point sampling method on the point cloud R1 to obtain key points;

[0070] 3.3.2) Using the nearest k-neighbor method, multiple points around a key point are grouped to form a local point region;

[0071] 3.3.3) A simplified PointNet network is used to encode the local point regions to obtain the features of the point cloud R1;

[0072] In this embodiment, if i = 1, R1 extracts 4096 points from 20000 points; if i = 2, R2 extracts 1024 points from 4096 points; if i = 3, R3 extracts 256 points from 1024 points.

[0073] 3.4) Represent the features V RP With the point cloud features R to be fused i Perform point-by-point fusion to generate fused point cloud features R. i ":

[0074] R i =σ(Wtanh(UV) RP +VR i ))

[0075] Where W, U, and V are three learnable weight matrices with different values, σ represents the sigmoid activation function, and tanh represents the hyperbolic tangent function;

[0076] 3.5) For the fused point cloud features R i "Using two fully connected layers (FC()) to sequentially compress features in the batch and channel dimensions to obtain a spatially enhanced pseudo-image Ps." i .

[0077] Ps i =P i ×FC(FC(R i ”))

[0078] In this embodiment, the spatially enhanced pseudo-image Ps i The size of the feature P is the same as that of the original pseudo-image, which is [496, 432].

[0079] Step 4: Perform color information fusion on the point cloud after spatial information enhancement.

[0080] One of the biggest drawbacks of single point clouds is the lack of color information, which makes it difficult to improve the accuracy of point cloud object detection. This step involves processing the RGB image F... i Size adjustment and pseudo-image transformation are performed to integrate color information into the spatial representation of the pseudo-image Ps. i Enhancement is performed to generate a color-enhanced pseudo-image Pc i .

[0081] Reference Figure 3 The specific implementation of this step is as follows:

[0082] 4.1) Adjust the F of the RGB image i Size, making its size and space enhance the pseudo-image Ps i The sizes are the same; in this embodiment, the RGB image F... i The original size was [1300, 400], and it was adjusted to [496, 432].

[0083] 4.2) For the resized RGB image F i Spatial Enhancement Ps i Both BatchNorm and ReLU operations are performed sequentially, and the two images are stitched together along the channel dimension to obtain the image to be transformed, PF; in this embodiment, the resized RGB image F i The shape is [3,496,432], and the spatially enhanced pseudo-image Ps i The shape is [64,496,432], and the shape of the stitched image to be transformed, PF, is [67,496,432].

[0084] 4.3) Generate the spatial factor matrix M of the graph to be transformed, PF, using the spatial attention formula and the channel attention formula respectively. s and channel factor matrix M c :

[0085] M s =σ(Conv(AvgPool(PF),MaxPool(PF)))

[0086] M c =σ(W5ReLu(W6(PF)))

[0087] Where σ() represents the activation function, Conv() represents the convolution operation, AvgPool() represents the average pooling operation, MaxPool() represents the max pooling operation, W5 and W6 represent two different fully connected layers, and ReLU() represents the ReLU operation.

[0088] 4.4) Based on the spatial factor matrix M s and channel factor matrix M c The pseudo-view transformation matrix M is calculated. cs :

[0089] M cs =0.6*M c +0.4*M s , where * denotes multiplication;

[0090] 4.5) Transform the pseudo-view matrix M cs With RGB image F i Multiplication yields the viewpoint transformation pseudo-image Pv i And put the pseudo image Pv i Spatially enhanced pseudo-image Ps i After concatenating along the channel dimension, a convolution operation is performed to generate a double-enhanced pseudo-image Pc. i In this embodiment, the dual-enhanced pseudo-image Pc i The shape is [64,496,432].

[0091] Step 5: Convolution to obtain the double-enhanced pseudo-image Pc3

[0092] 5.1) For the double-enhanced pseudo-image Pc i Convolution is performed to obtain a downsampled, double-enhanced pseudo-image Pc. i+1 Return to step (3);

[0093] 5.2) Repeat step (5.1) to finally obtain the double-enhanced pseudo-image Pc3;

[0094] In this embodiment, the number of convolution kernels corresponding to the convolution operation are [6, 128, 256], and the stride is [1, 2, 4].

[0095] Step 6: Obtain the feature map F to be detected based on the double-enhanced pseudo-image Pc3. U

[0096] 6.1) Perform two transposed convolutions on the double-enhanced pseudo-image Pc3 to obtain two transposed feature maps Pt1 and Pt2 respectively;

[0097] 6.2) Concatenate the double-enhanced pseudo-image Pc3 with the two transposed feature maps Pt1 and Pt2 sequentially to obtain the feature map F to be detected. UIn this embodiment, the transposed convolution kernels of both transposed convolutions are 128.

[0098] Step 7, transfer the feature map to be detected F U The data is fed into the SSD detector to generate detection results of targets ahead as the car moves.

[0099] The SSD detector is one of the best existing object detectors, capable of directly generating object detection results from the input feature map. Its specific implementation is as follows:

[0100] 7.1) The SSD detector detects the input feature map F. U Perform convolution to generate classification prediction scores and regression prediction scores;

[0101] 7.2) Generate a default bounding box based on the classification prediction score and the regression prediction score;

[0102] 7.3) Based on the default bounding boxes, obtain the classification loss and regression loss;

[0103] 7.4) Based on the classification loss and regression loss, the default boxes are decoded and non-maximum suppression is applied to obtain the final detection results.

[0104] The effects of this invention can be further illustrated by the following simulation experiments.

[0105] I. Simulation Experiment Dataset

[0106] The dataset uses KITTI, one of the most authoritative computer vision evaluation datasets in the world, which contains real images and point cloud data collected from urban, rural and highway scenes.

[0107] II. Simulation Content

[0108] The present invention method was used in conjunction with five existing single-LiDAR methods: MV3D, VoxelNet, SECOND, PointPillars, and PointRCNN, and five camera-based LiDAR methods: H. 2 3DRCNN, Point-GNN, IPOD, F-ConvNet, and PointPainting were used to perform object detection on the KITTI dataset. Their detection accuracy was compared, and the results are shown in Table 1.

[0109] Table 1 Comparison results of the method of the present invention with other existing methods

[0110]

[0111]

[0112] The MV3D method described is from "Multi-view 3D object detection network for autonomous driving".

[0113] The VoxelNet method described is from "VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection".

[0114] The SECOND method described is from "Sensors|FreeFull-Text|SECOND:SparselyEmbeddedConvolutionalDetection".

[0115] The PointPillars method is from "PointPillars: FastEncoders for Object Detection from PointClouds".

[0116] The Point R-CNN method described is from "Point R-CNN: 3D Object Proposal Generation and Detection from Point Cloud".

[0117] The H 2 The 3DRCNN method comes from "FromMulti-ViewtoHollow-3D:HallucinatedHollow-3DR-CNN for3DObjectDetection"

[0118] The Point-GNN method described is from the book "Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud".

[0119] The IPOD method described is from "IPOD: Intensive Point-based Object Detector for Point Cloud".

[0120] The F-ConvNet method described is from "FrustumConvNet: Sliding Frustums to AggregateLocalPoint-Wise Features for Amodal3DObjectDetection".

[0121] The PointPainting method described is from "PointPainting: Sequential Fusion for 3D Object Detection".

[0122] As shown in Table 1, our proposed method achieves excellent results in all accuracy metrics, especially for bicycles and cyclists. For example, compared to the classic object detection algorithm PointPillars, the accuracy is improved by an average of 2.66%. Specifically, for small object detection, the accuracy is improved by an average of 2.97%, with the largest improvement (5.27%) observed for simple cyclists. Compared to the classic object detection method PointRCNN, the accuracy is improved by an average of 11.50%, while the accuracy for small object detection is improved by an average of 11.3%. Specifically, the accuracy for simple bicycle detection is improved by 13.7%.

Claims

1. A multi-target detection method based on camera and lidar, characterized in that, Includes the following steps: (1) Data is obtained through sensors, RGB images are obtained by the camera, and point clouds are obtained by the lidar. ; (2) Point cloud Voxelization preprocessing is performed to obtain the original pseudo-image. : (3) Set the execution count variable i, with an initial value of i=1: when i=1, the pseudo image Original pseudo-image When i>1, pseudo-image The downsampled, double-enhanced pseudo-image obtained from the previous step (5a); the pseudo-image Spatial information enhancement, generating spatially enhanced pseudo-images : (3a) Point cloud features With pseudo-images Perform registration to generate an intramodal mapping matrix. : ； (3b) Based on sampling location and modal mapping matrix Generate a feature representation : ； in, Represents the bilinear interpolation function. Indicates sampling position Features of adjacent pixels; (3c) Using the Set Abstraction operation to analyze point cloud features Feature extraction is performed to generate point cloud features to be fused. ; (3d) Representing features Features of point clouds to be fused Point-by-point fusion to generate fused point cloud features : ； in, , and These are three learnable weight matrices with different values. This represents the sigmoid activation function. Represents the hyperbolic tangent function; (3e) Features of the merged point cloud Using two fully connected layers Obtain spatially enhanced pseudo-images : ； (4) Integrate the color information of the RGB image into the spatially enhanced pseudo image Generate color-enhanced pseudo-images : (4a) Adjust the size of the RGB image to make its size and spatial enhancement pseudo-image With the same dimensions, obtain the resized RGB image. ; (4b) Resize the RGB image Spatial Enhancement Pseudoimages Perform BatchNorm and ReLU operations respectively, and then concatenate them along the channel dimension to obtain the graph to be transformed. ; (4c) The diagram to be transformed Spatial factor matrices are generated using the spatial attention formula and the channel attention formula, respectively. and channel factor matrix ; (4d) Based on the spatial factor matrix and channel factor matrix The pseudo-view transformation matrix is ​​calculated. : ； (4e) Transform the pseudoview matrix Compared with the resized RGB image Multiplication yields a viewpoint transformation pseudo-image and the fake image Spatial Enhancement Pseudoimages Stitching is performed along the channel dimension to generate a double-enhanced pseudo-image. ; (5) Based on the double-enhanced pseudo-image Obtain a double-enhanced pseudo-image: (5a) For double-enhanced pseudo-images Perform convolution to obtain a downsampled, double-enhanced pseudo-image. ; (5b) Determine the value of variable i: If i < 3, then let i = i + 1 and return to step (3); if i = 3, then end the loop and finally obtain the double-enhanced pseudo-image. ; (6) For double-enhanced pseudo-images Perform two transposed convolutions to obtain two transposed feature maps. and ; Enhance the pseudo-image with double enhancement With the two transposed feature maps , By concatenating the data sequentially, the feature map to be detected is obtained. ; (7) The feature map to be detected The data is fed into the SSD detector to generate detection results of targets ahead as the car moves.

2. The method according to claim 1, characterized in that, In step (2), the point cloud Voxelization preprocessing is performed to achieve the following: (2a) in In terms of direction, the point cloud space is divided into four equal-height spaces; (2b) In In the direction, the point cloud within each height space is divided into areas with a base size of Regular columnar point clouds; (2c) Set the sampling threshold Random sampling is performed on columnar sub-point clouds with a number of points greater than a threshold, and zero-filling is performed on columnar sub-point clouds with a number of points less than a threshold. (2d) Calculate the arithmetic mean of all points within each columnar sub-point cloud. and offset ; (2e) Using max pooling and a simplified PointNet network to process the computed point cloud Encode the image to generate a sub-pseudo-image corresponding to each height space. ; And obtain each sub-pseudo-image Corresponding height dimension weight and channel dimension weight : ； ； in, , , , These represent four different fully connected layers. This represents the sigmoid activation function; (2f) Transfer the four sub-pseudo-images Its corresponding height dimension weight and channel dimension weight Multiply the images, then concatenate them along the channel dimension and perform max pooling to obtain the original pseudo-image. .

3. The method according to claim 1, characterized in that, In step (3c), the Set Abstraction operation is used to analyze the point cloud features. Feature extraction is performed as follows: (3c1) Point cloud features Key points are obtained using the farthest point sampling method; (3c2) Using the nearest k-neighbor method, multiple points around the key point are grouped to form a local point region; (3c3) A simplified PointNet network is used to encode local point regions to obtain point cloud features. Its characteristics.

4. The method according to claim 1, characterized in that, Step (4c) The graph to be transformed Spatial factor matrices are generated using the spatial attention formula and the channel attention formula, respectively. and channel factor matrix The formula is as follows: ； ； in, This represents the sigmoid activation function. This represents the convolution operation. This indicates the average pooling operation. This indicates a max pooling operation. and These represent two different fully connected layers. This indicates the ReLU operation.

5. The method according to claim 1, characterized in that, In step (7), the feature map to be detected is... The data is fed into the SSD detector to generate detection results of targets ahead during the car's movement, as follows: (7a) SSD detector on the feature map to be detected Perform convolution to generate classification prediction scores and regression prediction scores; (7b) Generate a default box based on the classification prediction score and the regression prediction score; (7c) Based on the default box, obtain the classification loss and regression loss; (7d) Based on the classification loss and regression loss, the default box is decoded and non-maximum suppression is applied to obtain the final prediction result.

Images