A traffic target detection method and system of cross-modal cross attention mechanism

Abstract

The application discloses a traffic target detection method and system of a cross-modal cross attention mechanism, at least comprising a 3D target detection network model of multi-modal feature deep fusion, the model comprising a cross-modal data enhancement module, a point cloud branch backbone network module, an image branch backbone network module, a point cloud-image cross attention fusion module and a detection task processing module, the method and system respectively perform fine-grained feature extraction on the point cloud branch backbone network and the image branch backbone network, in the point cloud branch, dynamic voxelization and sparse convolution are adopted to further improve the detection speed, in the image branch, through cross-layer connection of multi-level features, the image features have rich multi-scale information, the network improves the detection performance, and meanwhile, stable target detection can be realized for a shielding scene and a small-scale target scene, the inaccuracy of single sensor sensing targets is effectively reduced, and more accurate and robust detection performance is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and automatic driving intelligent perception, and relates to a radar and vision fusion multi-traffic target detection method, and mainly relates to a traffic target detection method and system of a cross-modal cross-attention mechanism. BACKGROUND

[0002] In the urban traffic scene, multi-traffic target perception for detecting the environment around the automatic driving car is an important prerequisite for car automatic driving. By identifying targets such as traffic signal signs, other cars, bicycles and pedestrians, and perceiving the distance and speed of surrounding objects, timely judgment and reaction can be made, which can provide data support for subsequent intelligent navigation, path planning and other function modules. Laser radar and camera are two most commonly used sensors in the field of automatic driving target perception, which respectively provide accurate 3D spatial information and rich color texture features. Due to the limitation of the sensors themselves, the laser radar point cloud data itself lacks specific semantic features and is relatively sparse, and is almost ineffective in long-distance or congestion conditions. The RGB image obtained by the camera sensor does not have depth information of three-dimensional space, and the imaging function is extremely sensitive to light, which also causes the sensitivity of the perception accuracy.

[0003] Relying on a single modal method is inefficient and unreliable under the conditions of occlusion and multi-scale change. In order to obtain more accurate environmental information in a larger range, it is possible to provide high-precision perception of multi-traffic targets in high-density urban environments by fully combining the advantages of both. However, since the RGB image and the point cloud belong to heterogeneous data, the difference in fusion method directly affects the performance of the detector, and how to effectively combine these two kinds of data still has challenges.

[0004] In recent years, perception methods based on deep learning provide new solutions for cross-modal data fusion. Data fusion methods include three levels: data level, feature level and decision level. The most widely used radar and vision fusion method in the industry is decision-level fusion, which has the advantage of strong robustness, but its essence is still single-sensor detection. Data-level fusion creates new data representations that depend on multiple modalities in the initial stage. Although a reasonable fusion method can slightly improve the perception accuracy, it requires strict alignment between modalities, and the operation based on the original data volume also increases the perception inference time, which is not suitable for vehicle-mounted bumpy or other easily shaken scenes. Feature-level fusion is between data-level and decision-level, which realizes cross-layer feature interaction between modalities. Most existing methods directly use feature splicing, ignoring the interference of noise interference on the quality of fused features, and lack of comprehensive consideration of feature fusion granularity. In addition, deep learning itself requires a sufficient number of rich samples to extract strongly related features of the target. The existing data enhancement method is only for single-modal data, and how to ensure the continuity of multi-modal data and the correlation of data enhancement method is particularly important to the generalization performance of the model. SUMMARY

[0005] The present application is aimed at the problem of how to align and fuse heterogeneous data in current multi-source sensor fusion target detection research, and provides a traffic target detection method and system based on cross-modal cross-attention mechanism, which at least includes a 3D target detection network model of multi-modal feature deep fusion. The model includes a cross-modal data enhancement module, a point cloud branch backbone network module, an image branch backbone network module, a point cloud-image cross-attention fusion module and a detection task processing module. The present method and system perform fine-grained feature extraction on the point cloud branch backbone network and the image branch backbone network respectively. In the point cloud branch, dynamic voxelization and sparse convolution are used to further improve the detection speed. In the image branch, multi-level features are connected across layers to make the image features have rich multi-scale information. The network improves the detection performance, and can also realize stable target detection for occlusion scenes and small-scale target scenes, effectively reducing the inaccuracy of single-sensor target perception, and achieving more accurate and robust detection performance.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is: a traffic target detection method based on cross-modal cross-attention mechanism, which at least includes a 3D target detection network model of multi-modal feature deep fusion. The model includes a cross-modal data enhancement module, a point cloud branch backbone network module, an image branch backbone network module, a point cloud-image cross-attention fusion module and a detection task processing module, wherein,

[0007] The cross-modal data enhancement module: after sequentially passing through point cloud data enhancement, image data enhancement and image-point cloud data association, enhanced pixel coordinate points are obtained.

[0008] The point cloud branch backbone network module: after sequentially passing through the grouping, sampling and sparse convolution feature extraction steps, the point cloud is divided into independent voxels by the voxel division method, and point cloud feature extraction is performed;

[0009] The image branch backbone network module: comprising an encoder and a decoder, and adopting a combination of ResNet and a feature pyramid network to extract image features;

[0010] The point cloud-image cross attention fusion module: used for fusing the aligned extracted point cloud features and image features;

[0011] The detection task processing module: a center point-based detection head is used for target regression, and the center point position and target frame geometric parameters are solved respectively to complete the target detection task.

[0012] In order to achieve the above purpose, the technical scheme adopted by the present application is: a traffic target detection method based on a cross-modal cross attention mechanism, comprising the following steps:

[0013] S1, data set construction: the data set comprises point cloud data and image data, a multi-traffic target perception data set, synchronous data of a vehicle-mounted camera and a laser radar is collected for labeling and calibration to form a point cloud-image multi-traffic target perception data set;

[0014] S2, cross-modal data enhancement: including point cloud data enhancement, image data enhancement and image-point cloud data association,

[0015] The point cloud data enhancement method specifically comprises: after randomly rotating, scaling and twice randomly rotating the point cloud and the points in the 3D bounding box thereof, the ID of the frame of point cloud and the random processing parameters are recorded;

[0016] The image data enhancement method specifically comprises: using the methods of random rotation, random flipping and multi-scale transformation, the image is subjected to data enhancement operation;

[0017] The image-point cloud data association specifically comprises: based on the data enhancement parameters saved after the point cloud data enhancement, all enhanced data is reversed according to the transformation parameters, the original coordinates of the 3D key points are obtained, the corresponding original pixel coordinates are found in the camera space according to the external projection parameters, the corresponding pixel coordinate points are processed according to the image data enhancement method, and the enhanced pixel coordinate points are obtained;

[0018] S3, point cloud feature extraction: containing three steps of grouping, sampling and sparse convolution feature extraction, the point cloud is divided into independent voxels by the voxel division method, and point cloud feature extraction is performed;

[0019] S4, image feature extraction: ResNet combined with feature pyramid network is used to extract image features;

[0020] S5, point cloud-image cross attention fusion: based on the remapping of graph features and cross attention mechanism, the extracted point cloud features and image features are fused;

[0021] S6, detection task processing: the center point-based detection head is used for target regression, and the center point position and target frame geometric parameters are solved respectively;

[0022] S7, loss function calculation: the loss function includes heat map loss, center point position offset loss, ground height loss, target size loss and yaw angle cosine and sine loss, and all losses are combined to obtain a combined loss;

[0023] S8, training model and output: the target detection network model is trained by using the PyTorch deep learning framework, and the target detection is completed.

[0024] As an improvement of the application, the point cloud data enhancement in step S2 specifically includes:

[0025] S21: set the point cloud data P i (x i ,y i ,z i )∈R 3 , a total of N points, wherein the real 3D bounding box B i of the kth target is parameterized as wherein is the center position, is the length, width and height, is the yaw rotation around the z axis;

[0026] S22: randomly rotate the real 3D bounding box and the points in the box, and the rotation factor is a uniformly distributed random variable Δθ ∈ [-π / 10, π / 10];

[0027] S23: apply scale transformation to the global point cloud, multiply all points P i (x i ,y i ,z i ) in the frame by a uniformly distributed scale transformation factor λ ∈ [0.95, 1.05], and globally scale down and up all point clouds;

[0028] S24: apply random rotation to the global point cloud, apply global rotation to all 3D bounding boxes B i and the entire point cloud, and the global rotation offset is a uniformly distributed random variable Δγ ∈ [-π / 4, π / 4];

[0029] S25: Record the ID of the frame point cloud and the random processing parameters, rotate the point cloud x.pcd clockwise by angel °, denoted as {x, angel °}.

[0030] As an improvement of the application, the step S3 specifically comprises:

[0031] S31: Group the point cloud, given the point cloud P = {p1, p2, …, p N N points are assigned to a buffer with a size of K × T × F, where K is the maximum voxel number, T is the maximum point number in the voxel, and F represents the feature dimension;

[0032] S32: Perform voxel sampling and introduce dynamic voxelization, define F V (p i ) as the mapping of each point p i to a voxel v j where the point is located, and define F P (v j ) as the mapping of collecting points within a voxel v j , which is expressed as follows:

[0033]

[0034]

[0035] S33: Design sparse convolution to extract point cloud features, use submanifold sparse convolution SubMConv3d, and only perform convolution operation when the center point position of the receptive field is active. The step size of the middle three convolution blocks is set to 2, the input voxel feature size is 16 × 10 × 1600 × 1408, and the submanifold sparse convolution 1 size is 16 × 10 × 1600 × 1408.

[0036] As another improvement of the application, the step S5 specifically comprises:

[0037] S51: Let the feature map extracted in the image backbone be Z ∈ R h×w×c , where h, w, and c are the height, width, and channel of the global feature map, respectively;

[0038] S52: Add a 1 × 1 convolution to reduce the feature dimension, and create a new feature mapping F ∈ R h×w×d ;

[0039] S53: Flatten the spatial dimension of F to one dimension to obtain a feature vector with height and width;

[0040] S54: Design an attention mechanism to extract features, the key Q and the value V are from the image feature F i = {f1, f2, …, fk} generated, the query is generated by the voxel feature P = {p1, p2,..., pi,..., pn} i} generated:

[0041] Q i = p i W Q ,K j = f j W K ,V j = f j W V

[0042] Wherein and is a linear projection, for the i-th query Q i , the attention weight is calculated according to the dot product similarity between cross-modal query and key:

[0043]

[0044] Wherein, Is a proportional factor, the output of cross attention mechanism is defined as the weighted sum of all values according to attention weight:

[0045]

[0046] The normalized attention weight s i , j represents the similarity between different spatial pixels f j And the voxel p i ;

[0047] S55: design a feedforward network to generate the final two-dimensional image feature, as shown in the formula:

[0048]

[0049] Wherein, FFN(·) is a single-layer fully connected layer neural network, the aggregated image feature is processed by the fully connected layer, and then connected with the original laser radar point cloud feature to generate the backbone feature.

[0050] As another improvement of the application, the step S6 specifically comprises:

[0051] S61: adopt a regression method based on heat map, the value is Y [0, 1], and the center point position of each target is predicted according to the feature map;

[0052] S62: design a convolution head to output the center point position offset o R 2 , height h R, size s R 3 , and cosine of yaw angle γ R 2 , combined with the center point position, form a complete 3D detection frame.

[0053] As a further improvement of the present application, in the step S7,

[0054] The focal loss function is used to calculate the heatmap loss of the whole network, wherein and are hyperparameters, and are set to 2 and 4, and N is the number of targets in the picture;

[0055]

[0056] The SmoothL1 loss function is used to calculate the center point offset regression loss value, wherein o p represents the predicted center point offset, o g represents the center point offset to be learned;

[0057]

[0058] The loss function SmoothL1 is used to calculate the ground height loss, wherein h p represents the predicted center point distance from the ground height, h g represents the true value of the distance from the ground;

[0059]

[0060] The SmoothL1 function is used to calculate the target size loss, wherein s p is the predicted target length, width and height set, s g is the target true value length, width and height set:

[0061]

[0062] The loss function SmoothL1 is used to calculate the yaw angle cosine loss, wherein γ g , γ p ∈R 2 contains the sine and cosine values of the yaw angle, representing the predicted value and the true value respectively:

[0063]

[0064] Compared with the prior art, the present application has the beneficial effects of:

[0065] (1) The present application proposes a radar and vision fusion multi-traffic target detection method based on cross-modal data enhancement and cross-attention mechanism, which realizes multi-scale detection of traffic targets by means of the complementary perception advantages of laser radar and camera, overcomes the problem of easy missed detection in occlusion scene and small scale target scene, and effectively improves the robustness of the detector;

[0066] (2) The present application is aimed at the problem that the features are difficult to align in the multi-source sensor data enhancement process, and the image-point cloud data association step is designed by using the calibration information between the laser radar and the camera, so as to ensure the continuity of the data in the data enhancement mode, and further increase the diversity of the samples, guide the network to learn the strong correlation task of the traffic target, compared with the existing fusion method such as PointAugmenting which can only process the data before enhancement, the data association module proposed can not only align different types of key points (for example, voxel center), but also can process the case that both modalities are enhanced;

[0067] (3) The present application is aimed at the problem of heterogeneous data expression difficulty in multi-source sensor fusion, starting from the essence of point cloud and image, dynamically paying attention to image pixel level features and point cloud voxel level features, establishing a mapping relationship to delimit the feature neighborhood of both, aggregating and connecting the features based on the relationship, obtaining rich multi-scale information, enhancing the expression of heterogeneous data, and improving the operation speed through sub-manifold sparse convolution, and greatly improving the perception accuracy of traffic targets, compared with the detection method of single sensor or feature real stacking fusion expression, the method proposed in the present application obtains better real-time performance and more accurate detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is the whole flow chart of the step execution operation of the method of the present application;

[0069] Figure 2 is the 3D target detection network structure schematic diagram of the multi-modal feature deep fusion of the present application;

[0070] Figure 3 is the schematic diagram of the image-point cloud data association process in the method of the present application;

[0071] Figure 4 is the sub-manifold sparse convolution structure schematic diagram of the point cloud feature extraction key module in the method of the present application;

[0072] Figure 5 is the feature pyramid network structure schematic diagram of the image feature extraction key module of the present application;

[0073] Figure 6 is the point cloud-image cross attention fusion module structure schematic diagram of the present application. DETAILED DESCRIPTION

[0074] The present application will be further illustrated in combination with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0075] Example 1

[0076] A radar and vision fusion multi-traffic target detection method based on cross-modal data enhancement and cross-attention mechanism, as shown in Figure 1 , specifically comprising the following steps:

[0077] Step S1: Construct a point cloud-image multi-traffic target perception dataset, collect synchronous data of vehicle-mounted cameras and laser radars for labeling and calibration, and combine existing data samples to jointly build original data and labels.

[0078] Design a 3D target detection network model with deep fusion of multi-modal features, as shown in Figure 2 , which mainly includes a cross-modal data enhancement module, a point cloud branch backbone network module, an image branch backbone network module, a cross-attention fusion module, and a detection task processing module, which realizes deep feature alignment and effective fusion of multi-modal data features, and specifically includes the following steps:

[0079] Step S2: Construct a cross-modal data enhancement module, which contains three contents, namely point cloud data enhancement processing, image data enhancement processing, and image-point cloud data association:

[0080] Design a point cloud data enhancement method, which randomly rotates, scales, and randomly rotates the point cloud and the points within its 3D bounding box. Assume that the point cloud data P i (x i ,y i ,z i )∈R 3 is obtained in each frame, with N points, and the true 3D bounding box B i of the kth target is parameterized as where is the center position, is the length, width, and height, is the yaw rotation around the z-axis, and let the set of all laser radar points within the bounding box B i be defined as:

[0081] S k ={p|p∈R N×3}

[0082] where p(x k ,y k ,z k ) represents the kth point in the current frame of point cloud, with a range of

[0083] First, randomly rotate the true 3D bounding box and the points within it. According to the 3D bounding box B i , the laser radar point set S k can be obtained, and through a uniformly distributed random variable △θ ∈ [-π / 10, π / 10], Sk All points p(x) inside k ,y k ,z k Rotate around the z-axis. Then move towards S. k Each point p(x) in k ,y k ,z k Add a Gaussian-distributed translation variable ΔT = (Δx, Δy, Δz) with a mean of zero and a standard deviation of 1, where Δx, Δy, and Δz are random translation variables of point p. To avoid interference between the transformed 3D bounding box and the points within the box and other objects, if interference is detected, the transformation will be canceled and the next transformation will be performed.

[0084] Secondly, a scale transformation is applied to the global point cloud to scale all points P within the frame. i (x i ,y i ,z i Multiplying the uniform distribution scale transformation factor λ∈[0.95,1.05] by the global scaling factor, the entire point cloud is globally scaled down and scaled up. The introduction of global scale enhancement improves the robustness of the network used to detect targets of different sizes and distances.

[0085] Then, a random rotation is applied to the global point cloud, and this global rotation is applied to all 3D bounding boxes B. i The global rotation rotates the entire point cloud along the z-axis by adding a global rotation offset, which is a uniformly distributed random variable Δγ∈[-π / 4,π / 4].

[0086] Finally, record the ID and random processing parameters of the point cloud in that frame. For example, rotate the point cloud 000001.pcd clockwise by 5° and denote it as {000001, 5°}. It should be noted that all the above data augmentation operations are not performed simultaneously. Design a probability function to select whether to perform augmentation processing and to select the augmentation processing method.

[0087] The design of image data augmentation methods references geometric transformation data augmentation methods for point cloud data. It employs random rotation, random flipping, and multi-scale transformation to perform data augmentation operations on images, increasing sample diversity and preventing the network from learning tasks unrelated to traffic target detection.

[0088] Designing an image-point cloud data association module is crucial for multimodal data. Ensuring data continuity and establishing the correspondence between LiDAR point clouds and camera images directly impacts final detection accuracy. Different data augmentation strategies are used for the two modes of data; for example, rotating the 3D point cloud along the z-axis combined with randomly flipping the 2D image. Aligning the features of both is paramount in the data association module design. Figure 3As shown, the process is as follows:

[0089] When the data related to geometry is applied to data enhancement, the data enhancement parameters saved in the point cloud data enhancement method are used to reverse all enhanced data according to the transformation parameters, so as to obtain the original coordinates of the 3D key points, and then find the corresponding original pixel coordinates in the camera space according to the external projection parameters, and then process the corresponding pixel coordinate points according to the steps of image data enhancement to obtain the enhanced pixel coordinate points, at this time, the 3D space points and 2D pixel points can be associated. Compared with the existing fusion method such as PointAugmenting which can only process data before enhancement, the data association module DA proposed can not only align different types of key points (for example, voxel centers), but also process the case where both modalities are enhanced;

[0090] Step S3: design a point cloud branch backbone network, the point cloud is a set of sparse and unordered points in three-dimensional space, and a voxel representation based method is used to extract point cloud features, the point cloud is divided into independent voxels by voxel division method, which solves the problem of unorderedness of point cloud, mainly including grouping, sampling and sparse convolution feature extraction three sub-steps:

[0091] S31: point cloud grouping, given a point cloud P={p1,p2,…,p N}, N points are allocated to a buffer with a size of K×T×F, where K is the maximum number of voxels, T is the maximum number of points in a voxel, and F represents the feature dimension. In the grouping stage, each point p i is allocated to a voxel v j according to its spatial coordinates.

[0092] S32: voxel sampling, the conventional sampling method has limitations, when the points and voxels exceed the fixed capacity, the points that may be useful for detection are directly discarded, which leads to uncertain voxel embedding, resulting in unstable or jittering detection results, and the cost of filled voxels and unnecessary calculation hinder the performance at runtime. The dynamic voxelization method is introduced to overcome these shortcomings, and F V (p i ) is defined as the mapping of each point p i to a voxel v j where the point is located, and F P (v j ) is defined as the mapping of collecting points in a voxel v j , and the formula is as follows:

[0093]

[0094]

[0095] The number of voxels and the number of points per voxel will depend on the specific mapping function and will no longer be fixed. This eliminates the need for a fixed size buffer and eliminates the discarding of random points and voxels;

[0096] S33: Sparse convolution extracts point cloud features. The sparsity of the point cloud causes many empty voxels in the voxel grid. The large amount of calculation of conventional 3D convolution operations and the existence of empty voxel features will greatly increase the calculation cost in the back propagation process. The method uses a submanifold sparse convolution SubMConv3d. Convolution operation is only performed when the center point position of the receptive field is in an active state, avoiding the generation of too many active positions by spatial sparse convolution. The designed submanifold sparse convolution is as shown in Figure 4 The step size of the middle three convolution blocks is set to 2, thereby quickly extracting abstract voxel features. By combining submanifold sparse convolution and spatial sparse convolution, the receptive field is increased while the sparsity is maintained as much as possible.

[0097] Step S4: Design the image branch backbone network. For a deep convolutional network, the convolution kernel traverses the entire picture to perform convolution from one feature layer to another. The pixels occupied by large-scale targets are more than those occupied by small-scale targets, so more information of large-scale targets is often retained during convolution. As the network depth increases, the features of small-scale targets become less and less. In order to alleviate the network degradation and marginalization of small-scale targets, a combination of ResNet and feature pyramid network is used to extract image features. The image feature extraction branch is as shown in Figure 5 The image feature extraction branch is divided into an encoder and a decoder. The encoder part includes 5 different convolution blocks, each of which corresponds to a module of ResNet-50.

[0098] During image feature encoding, each module of the backbone network extracts features of different dimensions from the image. After passing through each module, the feature map size becomes half of the original, and the feature channel dimension increases by one. The output of each layer is (C1, C2, C3, C4, C5), where C1 represents shallow features, and C5 represents deep features.

[0099] In the decoding module, there are 3 deconvolution blocks. First, the deep feature information is upsampled by bilinear interpolation to restore the size to the same size as the upper layer feature. Then, the deep feature is combined with the feature of the upper layer by splicing operation, and continues to be transmitted to the shallow layer. In each splicing process, the features (C1, C2, C3, C4, C5) are reduced to a dimension of 256 by 1x1 convolution. After splicing, the fused features are processed by 3x3 convolution to obtain the final output.

[0100] Step S5: design a point cloud-image cross-attention fusion module CAF, which dynamically focuses on the pixel-level features of the image, and each voxel feature defines the feature neighborhood of the image plane in a universal mapping relationship ensured by the deterministic projection matrix, then CAF aggregates the image features based on the feature alignment relationship and connects them with the original three-dimensional features, and the fusion module is as shown in Figure 6 ;

[0101] S51: let the feature map extracted in the image backbone be Z∈R h×w×c , where h, w, c are the height, width and channel of the global feature map, respectively;

[0102] S52: add a 1x1 convolution to reduce the feature dimension, and create a new feature map F∈R h×w×d ;

[0103] S53: change the spatial dimension of F to one dimension to obtain a feature vector with height and width;

[0104] S54: let the image feature F i ={f1,f2,...,f k}(f j represents the jth image feature contained in the ith voxel space) and the voxel feature P extracted by the point cloud branch backbone network P={p1,p2,...,p i}(p i represents each non-empty voxel feature), the key and value are generated from the image feature F i , and the query is generated from the voxel feature P,

[0105] Q i =p i W Q ,K j =f j W K ,V j =f j W V

[0106] where and are linear projections, and for the ith query Q i , the attention weight is calculated according to the dot product similarity between the cross-modal query and the key:

[0107]

[0108] where is a scale factor, and the output of the cross-attention mechanism is defined as the weighted sum of all values according to the attention weight:

[0109]

[0110] Normalized attention weight s i j represents the similarity between different spatial pixels f j and voxels p i The weighted sum of these values can aggregate fine-grained spatial pixels to update p i , enrich voxel features with two-dimensional information in a global view;

[0111] S55: Use the feedforward network to generate the final two-dimensional image features, as shown in the formula:

[0112]

[0113] Where FFN(·) is a single-layer fully connected layer neural network, and the aggregated image features are processed by the fully connected layer and connected with the original laser radar point cloud features to generate the backbone features.

[0114] Step S6: Design a detection task processing module. Three-dimensional target detection objects are usually represented by 3D bounding boxes in point clouds. The bounding box has no specific direction, so it is difficult for an anchor-based detector to enumerate all directions or fit an axis-aligned detection box for a rotating object. Therefore, this method designs a center point-based detection head for target regression.

[0115] S61: Predict the center point position of each target according to the feature map. A heat map-based regression method is used: for any feature map with size WxHx3, a convolution layer is used to generate a heat map with size , where K is the number of detected classes, R is the step size, and the value of the element in the heat map is Y∈[0,1]. If the point in the heat map is 1, it is the center of a detection box in the image. If it is 0, the point is in the background of the image.

[0116] S62: In order to completely represent the 3D bounding box, other attributes need to be predicted: center point position offset o∈R 2 , height h∈R, size s∈R 3 , and yaw angle cosine γ∈R 2 Since the center target point is mapped to the voxel grid and is not accurate, a deviation value o needs to be predicted for correction. The ground height h helps to locate the object in three-dimensional space and complete the missing height information deleted by projection. The size s represents the actual length, width, and height of the target in three-dimensional space. Combined with the above information, a 3D bounding box can be obtained.

[0117] Step S7: training the designed multi-modal feature deep fusion 3D target detection network, the iterative optimization of network parameters mainly uses a random gradient descent algorithm, wherein the loss function mainly includes a heat map loss, a center point position offset loss, a ground height loss, a target size loss and a yaw angle cosine and sine loss, and the definition and calculation of the loss function are divided into the following 5 steps:

[0118] (1): calculating the heat map loss, the heat map is used to predict the target center, determines the presence and classification of the target, and in an actual scene, due to the sparsity of point clouds and the special nature of the view of the map, only a small amount of foreground vehicle points are often included in the detection area, and most of the points are background points; this makes the loss value of the negative sample in the detection area greater than that of the positive sample, and the learning of the classification task is easily dominated by the negative sample. Considering the above problems, a focal loss function is used to calculate the heat map loss of the entire network:

[0119]

[0120] Wherein and α and β are hyperparameters, set to 2 and 4, and N is the number of targets in the picture;

[0121] (2): calculating the center point position offset loss, each center point has a certain error after being predicted, so the center point offset loss is used for compensation, and then the actual position coordinates are calculated. In the training process, the SmoothL1 loss function can be used to calculate the regression loss value:

[0122]

[0123] Wherein o p represents the predicted center point offset, and o g represents the learned center point offset.

[0124] (3): calculating the ground height loss, the center point position offset is only the position offset in the BEV view, and the actual position of the target in the three-dimensional space is also missing the height information, so on the basis of calculating the center point position offset, the loss of the ground height also needs to be calculated. The loss function uses SmoothL1 to calculate:

[0125]

[0126] Wherein, h p represents the predicted center point distance from the ground height, and h g represents the true value of the distance from the ground height.

[0127] (4):Calculate the target size loss. After obtaining the center point position of the target in the three-dimensional space, the length, width and height of the target need to be predicted. The SmoothL1 function is used to calculate the size of the target:

[0128]

[0129] wherein s p is a set of predicted length, width and height of the target, s g is a set of true value of length, width and height of the target

[0130] (5):Calculate the yaw angle cosine loss. For the yaw angle loss of the 3D bounding box of the target, if the yaw angle is directly predicted and the loss is calculated according to the yaw angle true value and the predicted value, the loss will be larger when the yaw angle is 0 and π, which will slow down the convergence speed of the network. Compared with the position, height and size loss, the loss calculation of the yaw angle often converges slowly. Considering the above problems, the sine and cosine values of the yaw angle are used to predict the size of the angle and whether the direction is the same as the true value. The SmoothL1 function is used to calculate the yaw angle cosine loss of the target:

[0131]

[0132] wherein γ g , γ p ∈R 2 contains the sine and cosine values of the yaw angle, representing the predicted value and the true value respectively;

[0133] Combining all the losses of the network, the combined loss L can be obtained:

[0134] L=ω1L heatmap +ω2(L off +L height +L size +L angle )

[0135] wherein ω1 and ω2 represent the weight coefficients of different part losses;

[0136] Step S8: During training, the designed network architecture is run using the PyTorch deep learning framework. The maximum number of iterations is set to I, the learning rate is set to lr, and the weight decay is initially set to λ. The Loss is calculated and the gradient is backpropagated, and the model parameters are updated.

[0137] In conclusion, the application discloses a radar and vision fusion multi-traffic target detection method and system based on cross-modal data enhancement and cross attention mechanism, discloses a cross-modal data enhancement module (CDA, Cross-modal Data Augment Module) and a cross attention fusion module (CAF, Cross Attention Fusion Module) to realize deep feature alignment, inputs point cloud data and image data into the cross-modal data enhancement module, and synchronously ensures the consistency of heterogeneous data through data correlation while performing data enhancement operation; and based on a feature fusion mode, proposes a 3D target detection network for multi-modal feature deep fusion, which respectively performs fine design on a point cloud branch backbone network and an image branch backbone network, further improves the detection speed in the point cloud branch by adopting a dynamic voxelization and sparse convolution mode, and makes the image features have rich multi-scale information by cross-layer connection of multi-level features in the image branch, so that the network improves the detection performance, and can realize stable target detection for a shielding scene and a small-scale target scene, and effectively overcomes the problem that a single sensor feature is not complete and it is difficult to accurately recall traffic targets.

[0138] It should be noted that the above content only illustrates the technical idea of the application, and cannot limit the protection scope of the application. For ordinary skilled persons in the art, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements fall within the protection scope of the claims of the application.

Claims

1. A traffic target detection system based on a cross-modal cross-attention mechanism, characterized in that: The model includes at least a 3D object detection network model with deep fusion of multimodal features. This model comprises a cross-modal data augmentation module, a point cloud branch backbone network module, an image branch backbone network module, a point cloud-image cross-attention fusion module, and a detection task processing module. The cross-modal data augmentation module: sequentially performs point cloud data augmentation, image data augmentation, and image-point cloud data association to obtain the augmented pixel coordinates. The point cloud branch backbone network module: After sequentially going through the steps of grouping, sampling and sparse convolution feature extraction, the point cloud is divided into independent voxels by the voxel partitioning method for point cloud feature extraction. The image branch backbone network module includes an encoder and a decoder, and uses a combination of ResNet and feature pyramid network to extract image features. The point cloud-image cross-attention fusion module is used to fuse the extracted point cloud features and image features. The detection task processing module uses a center-point-based detection head for target regression, solving for the center point position and the geometric parameters of the target box to complete the target detection task.

2. A traffic target detection method using a cross-modal cross-attention mechanism of the system as described in claim 1, characterized in that, Includes the following steps: S1, Dataset Construction: The dataset includes point cloud data and image data, a multi-traffic target perception dataset, which is formed by collecting synchronous data from vehicle-mounted cameras and lidar, labeling and calibrating them. S2, Cross-modal data augmentation: including point cloud data augmentation, image data augmentation, and image-point cloud data association. The point cloud data augmentation method specifically involves: performing random rotation, scale transformation, and secondary random rotation on the point cloud and the points within its 3D bounding box, and then recording the ID and random processing parameters of the frame point cloud. The image data enhancement method specifically involves using random rotation, random flipping, and multi-scale transformation to perform data enhancement operations on the image. The image-point cloud data association specifically involves: based on the data enhancement parameters saved after point cloud data enhancement, inverting all enhanced data according to the transformation parameters to obtain the original coordinates of 3D key points, finding the corresponding original pixel coordinates in the camera space according to the external projection parameters, and processing the corresponding pixel coordinates according to the image data enhancement method to obtain the enhanced pixel coordinates. S3, Point Cloud Feature Extraction: This includes three steps: grouping, sampling, and sparse convolution feature extraction. The point cloud is divided into independent voxels by the voxel partitioning method, and point cloud features are extracted. S4, Image Feature Extraction: Image features are extracted using a combination of ResNet and Feature Pyramid Network; S5, Point Cloud-Image Cross-Attention Fusion: Based on graph feature remapping and cross-attention mechanism, the point cloud features and image features extracted after alignment are fused together; S6, Detection Task Processing: The center-point-based detection head is used for target regression to solve for the center point position and the geometric parameters of the target box. S7, Loss Function Calculation: The loss function includes heatmap loss, center point position offset loss, ground height loss, target size loss, and yaw angle sine and cosine loss. Combining all losses yields the combined Loss. S8, Train the model and output: Train the object detection network model using the PyTorch deep learning framework to complete object detection.

3. The traffic target detection method based on a cross-modal cross-attention mechanism as described in claim 2, characterized in that: The point cloud data enhancement in step S2 specifically includes: S21: Assume the point cloud data acquired in each frame ∈R 3 There are N points in total, where the true 3D bounding box B of the k-th target is... i Parameterization ,in It is the central location. It refers to length, width, and height. It is a yaw rotation about the z-axis; S22: Randomly rotate the real 3D bounding box and its points, with the rotation factor being a uniformly distributed random variable. ; S23: Apply scale transformation to the global point cloud, and scale all points within the frame. With uniform distribution scaling factor Multiplication performs a global scaling down and scaling up of all point clouds; S24: Apply random rotation to the global point cloud, applying the global rotation to all 3D bounding boxes B. i The global rotation offset, along with the entire point cloud, is a uniformly distributed random variable. ; S25: Record the ID and random processing parameters of the point cloud in this frame, and rotate the point cloud x.pcd clockwise by angel°, denoted as {x, angel°}.

4. The traffic target detection method based on a cross-modal cross-attention mechanism as described in claim 2, characterized in that: Step S3 specifically includes: S31: Group the point cloud, given the point cloud N points are allocated to a buffer of size K×T×F, where K is the maximum number of voxels, T is the maximum number of points in a voxel, and F represents the feature dimension. S32: Perform voxel sampling, introduce dynamic voxelization, and define... To make each point Assigned to a voxel containing that point The mapping, and Defined in a voxel The mapping of internal collection points is expressed by the following formula: ； ； S33: Design sparse convolution to extract point cloud features. Use SubMConv3d sparse convolution. Convolution operation is only performed when the center point of the receptive field is active. The stride of the three middle convolution blocks is set to 2. The input voxel feature size is 16×10×1600×1408. The size of SubMConv3d sparse convolution 1 is 16×10×1600×1408.

5. The traffic target detection method based on a cross-modal cross-attention mechanism as described in claim 2, characterized in that: Step S5 specifically includes: S51: Let the feature map extracted from the image backbone be... , where h, w, and c are the height, width, and channels of the global feature map, respectively; S52: Add a 1×1 convolution to reduce the feature dimension and create a new feature map. ; S53: Flatten the spatial dimension of F to one dimension to obtain a feature vector with high and wide dimensions; S54: Design an attention mechanism to extract features, with key Q and value V from image features. Generated in the middle, query based on voxel features generate: ； in , and It is a linear projection, for the i-th query Attention weights are calculated based on the dot product similarity between cross-modal queries and keys: ； in, It is a scaling factor, and the output of the cross-attention mechanism is defined as a weighted sum of all values ​​based on the attention weights: ； Normalized attention weights s i j represents different spatial pixels and voxels Similarity between them; S55: Design a feedforward network to generate the final 2D image features, as shown in the formula: ； Among them, FFN(·) is a single-layer fully connected neural network. The aggregated image features are processed by the fully connected layer and then connected with the original LiDAR point cloud features to generate the backbone features.

6. The traffic target detection method based on a cross-modal cross-attention mechanism as described in claim 2, characterized in that: Step S6 specifically includes: S61: Using a heatmap-based regression method, the value is... The center point of each target is predicted based on the feature map; S62: Design the offset of the output center point of the convolution head ,high ,size yaw angle sine and cosine Combined with the position of the center point, a complete 3D detection frame is formed.

7. The traffic target detection method based on a cross-modal cross-attention mechanism as described in claim 2, characterized in that: In step S7 The focus loss function is used to calculate the heatmap loss of the entire network, where and and These are hyperparameters, set to 2 and 4, where N is the number of objects in the image; ； The center point offset regression loss value is calculated using the SmoothL1 loss function, where Indicates the predicted center point offset. This indicates the desired offset of the center point; ； The SmoothL1 loss function is used to calculate the ground height loss, where This indicates the height of the predicted center point above the ground. This represents the actual value of the height above the ground. ； The target size loss is calculated using the SmoothL1 function, where s p For predicting the length, width, and height set of the target, s g The set of length, width, and height for the objective truth value: ； The yaw angle sine and cosine loss is calculated using the loss function SmoothL1, where Includes the sine and cosine values ​​of the yaw angle, representing the predicted and actual values, respectively: 。

Images