Double-branch road traffic target detection method and system based on multi-attention mechanism fusion

Abstract

The invention provides a double-branch road traffic target detection method based on multiple attention mechanism fusion, and relates to the technical field of image detection. Firstly, a road traffic data set image is obtained and preprocessed; then, a double-branch road traffic target detection model based on multi-attention mechanism fusion is constructed and trained, and the double-branch road traffic target detection model comprises a double-branch parallel feature extraction Backbone network provided with auxiliary network branches, a Neck neck network and a Head detection head network; inputting a road traffic image to be detected into the trained double-branch road traffic target detection model; in the Backbone network, multi-scale feature details are captured based on an attention mechanism, features are extracted and fused, and auxiliary network branches are utilized to supplement semantic information extraction; and in the Neck neck network, an attention mechanism is introduced to fuse the multi-scale features, a candidate frame of the road traffic target is predicted by using a Head detection head network, and a final road traffic target detection result is generated. According to the invention, the accuracy and efficiency of traffic target detection are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image detection, and more specifically relates to a dual-branch road traffic target detection method and system based on the fusion of multiple attention mechanisms. Background Art

[0002] With the rapid development of autonomous driving technology and smart urban transportation, the importance of traffic object detection technology has become increasingly prominent. Object detection methods can accurately identify key road targets such as vehicles, pedestrians, and traffic signs in real time, providing timely warnings of potential hazards and reducing the occurrence of traffic accidents. The development of object detection and tracking technology also provides the necessary perception information for autonomous driving systems, improving traffic safety and enabling autonomous vehicles to better adapt to complex and changing road environments, providing a key basis for decision-making and control of autonomous driving systems.

[0003] In recent years, with the rapid development of deep learning, object detection and recognition technology has achieved significant breakthroughs. Using deep learning for object detection enables real-time analysis of video or image data, quickly identifying and locating objects in complex scenes. Deep learning-based object detection algorithms are currently categorized into one-stage and two-stage algorithms. Two-stage algorithms, when used for object recognition, first generate candidate boxes from an image and then perform object detection within the candidate box regions. These algorithms include regional convolutional neural networks, fast regional convolutional neural networks, and masked regional convolutional neural networks. Two-stage algorithms achieve high accuracy in object detection tasks, but are generally slower. Compared to two-stage algorithms, one-stage algorithms not only meet high-precision requirements but also significantly increase image processing speed, ensuring the effectiveness of the results. These include single-step multi-box detectors, the YOLO series of models, and real-time Transformer detectors.

[0004] The YOLO series of models has been widely used in traffic object detection due to its single-stage nature and higher real-time performance. However, existing YOLO models suffer from reduced accuracy and efficiency when detecting road traffic objects due to dynamic changes in object scale and mutual occlusion between objects, leading to false detections and missed detections. Summary of the Invention

[0005] In order to solve the problem of reduced accuracy and efficiency of existing traffic target detection methods, the present invention proposes a dual-branch road traffic target detection method and system based on the fusion of multiple attention mechanisms to improve the accuracy and efficiency of the road traffic target detection method.

[0006] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0007] In a first aspect, the present invention provides a dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms, comprising the following steps:

[0008] Acquire road traffic dataset images and preprocess the road traffic dataset images;

[0009] A dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms is constructed. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction backbone network with auxiliary network branches, a neck network, and a head detection network;

[0010] The preprocessed road traffic dataset images are used to train the dual-branch road traffic target detection model to obtain a trained dual-branch road traffic target detection model;

[0011] Input the road traffic image to be detected into the trained dual-branch road traffic target detection model;

[0012] In the dual-branch parallel feature extraction Backbone network, the attention mechanism is used to capture the multi-scale feature details of the road traffic image to be detected, extract and fuse the features, and use the auxiliary network branch to supplement the semantic information extraction; in the Neck network, the attention mechanism is introduced to fuse multi-scale features, and the Head detection network is used to predict the candidate boxes of road traffic targets to generate the final road traffic target detection results.

[0013] Furthermore, the dual-branch parallel feature extraction Backbone network includes: a first Conv module, a first EMA multi-scale attention combination module, a second EMA multi-scale attention combination module, a third EMA multi-scale attention combination module and a fourth EMA multi-scale attention combination module; any EMA multi-scale attention module among the first EMA multi-scale attention combination module, the second EMA multi-scale attention combination module, the third EMA multi-scale attention combination module and the fourth EMA multi-scale attention combination module includes a second Conv layer, a first C2f layer and a first EMA multi-scale attention layer; the first EMA multi-scale attention combination module extracts first-scale detail features through the EMA multi-scale attention layer and inputs them into the second EMA multi-scale attention combination module, the second EMA multi-scale attention combination module extracts second-scale detail features through the EMA multi-scale attention layer and inputs them into the third EMA multi-scale attention combination module, the third EMA multi-scale attention combination module extracts third-scale detail features through the EMA multi-scale attention layer and inputs them into the fourth EMA multi-scale attention combination module, and the fourth EMA multi-scale attention combination module extracts fourth-scale detail features through the EMA multi-scale attention layer;

[0014] Also includes: a first CBLinear module, a second CBLinear module, a third CBLinear module, a fourth CBLinear module, a fifth CBLinear module and an auxiliary network branch;

[0015] The auxiliary network branch includes: a first Conv and CBFuse combination module, a first multi-scale extraction module, a second multi-scale extraction module, a third multi-scale extraction module, and a fourth multi-scale extraction module; any multi-scale extraction module among the first multi-scale extraction module, the second multi-scale extraction module, the third multi-scale extraction module, and the fourth multi-scale extraction module includes: a third Conv layer, a first CBFuse layer, a second C2f layer, and a second EMA multi-scale attention layer; the primary features output by the first Conv module are connected to the first Conv and CBFuse combination module through the first CBLinear module; the initial features output by the first Conv and CBFuse combination module and the first scale detail features output by the first EMA multi-scale attention combination module The input end of the first multi-scale extraction module is connected through the second CBLinear module; the detail semantic information features output by the first multi-scale extraction module and the second-scale detail features output by the second EMA multi-scale attention combination module are connected to the input end of the second multi-scale extraction module through the third CBLinear module; the detail semantic information features output by the second multi-scale extraction module and the third-scale detail features output by the third EMA multi-scale attention combination module are connected to the input end of the third multi-scale extraction module through the fourth CBLinear module; the detail semantic information features output by the third multi-scale extraction module and the fourth-scale detail features output by the fourth EMA multi-scale attention combination module are connected to the input end of the fourth multi-scale extraction module through the fifth CBLinear module.

[0016] Based on the above technical means, an auxiliary network is added to the dual-branch parallel feature extraction backbone network to help the backbone network better extract global and local features of traffic target detection images, improving the efficiency and accuracy of feature extraction. At the same time, the introduction of the EMA multi-scale attention layer enhances feature representation capabilities, thereby improving the accuracy and efficiency of target detection.

[0017] Furthermore, the Neck network includes: a first SPPF module, a first upsampling fusion module, a second upsampling fusion module, a first C35TR module, a first multi-scale attention feature fusion module, and a second multi-scale attention feature fusion module connected in sequence;

[0018] Any upsampling fusion module in the first upsampling fusion module and the second upsampling fusion module includes: a first Upsample layer and a first Concat-C2f layer;

[0019] Any multi-scale attention feature fusion module in the first multi-scale attention feature fusion module and the second multi-scale attention feature fusion module includes: a fourth Conv layer, a second Concat-C2f layer and a second C35TR layer; the output end of the first upsampling fusion module is connected to the input end of the second multi-scale attention feature fusion module; the output end of the second upsampling fusion module is connected to the input end of the first multi-scale attention feature fusion module; the first multi-scale attention feature fusion module and the second multi-scale attention feature fusion module use the C35TR layer to fuse multi-scale features.

[0020] Furthermore, any EMA multi-scale attention layer among the first EMA multi-scale attention layer, the second EMA multi-scale attention layer, the third EMA multi-scale attention layer, the fourth EMA multi-scale attention layer, the fifth EMA multi-scale attention layer, the sixth EMA multi-scale attention layer, the seventh EMA multi-scale attention layer and the eighth EMA multi-scale attention layer includes: a first Re-weight operation, a first GroupNorm operation, a first AvgPool operation, a first Softmax operation, a first Matmul operation, a third Sigmoid operation and a second Re-weight operation connected in sequence; a fifth Conv operation, a second AvgPool operation, a second Softmax operation and a second Matmul operation connected in sequence; the output end of the first Groups operation is respectively connected to the input end of the first X AvgPool operation, the first Y AvgPool operation, the fifth Conv operation, the first Re-weight operation and the second Re-weight operation; the first X AvgPool operation and the first Y The output ends of the AvgPool operation are all connected to the input ends of the first Concat_Conv operation; the output ends of the first Concat_Conv operation are respectively connected to the input ends of the first Sigmoid operation and the second Sigmoid operation; the output ends of the first Sigmoid operation and the second Sigmoid operation are both connected to the input end of the first Re-weight operation; the output end of the first Re-weight operation is connected to the input end of the second Matmul operation; the output end of the sixth Conv operation is connected to the input end of the first Matmul operation; the output end of the second Matmul operation and the output end of the third Sigmoid operation are spliced and connected to the input end of the third Sigmoid operation.

[0021] Furthermore, the first C35TR module and the second C35TR layer each include: a first C3 structure and a first SwinTransformer structure;

[0022] The first Swin Transformer structure includes: a first PartitionPatch operation, a first LinearBmbedding operation, a first SwinTransformerBlock combination operation, a first patch Merging operation, a second SwinTransformerBlock combination operation, a second patch Merging operation, a third SwinTransformerBlock combination operation, a third patchMerging operation and a fourth SwinTransformerBlock combination operation, which are connected in sequence.

[0023] Based on these technical approaches, the Swin Transformer structure is introduced into the Neck network to efficiently extract multi-scale features and enhance their expressiveness. The Patch Merging operation further reduces the dimensionality of the feature map while retaining important feature information, thus achieving multi-scale feature fusion. Multiple Swin Transformer Block operations can more comprehensively capture the detailed features of the detected target and the relationship between the target and the background, thereby improving detection accuracy.

[0024] Furthermore, in the EMA multi-scale attention layer: Let X∈R C×H×W Represents the input feature map, which is divided into G sub-feature maps according to the channel dimension. The calculation expression for the division is:

[0025] X=[X0,X1,...,X G-1 ],X i ∈R C / / G×H×W ,G<<C

[0026] In the formula, G represents the number of sub-features, X i represents the i-th sub-feature;

[0027] Three parallel branches are used for feature encoding. The feature encoding process is as follows:

[0028] In the first 1×1 feature encoding branch, one-dimensional global average pooling is performed on the channels in the horizontal direction. In the second 1×1 feature encoding branch, one-dimensional global average pooling is performed on the channels in the vertical direction. The calculation expression of global average pooling is:

[0029]

[0030] Where, represents the output of the Cth channel with height H, Represents the output of the Cth channel with width w;

[0031] The feature encoding after the global average pooling operation is spliced along the height direction, decomposed into two vectors using convolution operation, and linearly fitted with the nonlinear Sigmoid function. Finally, the linearly fitted features are aggregated using multiplication.

[0032] The aggregated features are subjected to a two-dimensional global average pooling operation and a Softmax function linear transformation. The calculation expression of the two-dimensional global average pooling operation is:

[0033]

[0034] Where Z C Represents the output of the Cth channel;

[0035] In the 3×3 feature encoding branch, local features of the input feature map are obtained through convolution, and the local features are subjected to a two-dimensional global average pooling operation and a Softmax linear transformation;

[0036] The output features of the 1×1 feature encoding branch and the 3×3 feature encoding branch are multiplied by matrix dot product operation to extract and fuse feature details from dual spatial scales and different channel dimensions.

[0037] Furthermore, in the Swin Transformer structure: the input size is H×W×3 feature map, the feature map is split into feature maps with a dimension of H / 4×W / 4×48 through the PartitionPatch operation, and then the split feature map is straightened and linearly transformed through the LinearBmbedding operation. The SwinTransformerBlock combination operation extracts the local and global feature relationship of the feature map, and the convolution-free downsampling operation is performed through the patch Merging operation. The SwinTransformerBlock combination operation and patch Merging operation are repeated, and finally the feature map with a dimension of H / 32×W / 32×8C is output.

[0038] Furthermore, during the training process, the Slide Loss classification loss function is used as the classification loss function of the dual-branch road traffic target detection model. The expression of the Slide Loss classification loss function is:

[0039]

[0040] Where x represents the IoU value and μ represents the threshold parameter.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the dual-branch road traffic target detection program based on the fusion of multiple attention mechanisms is executed by the processor, the steps of the dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms are implemented.

[0042] The present invention also provides a dual-branch road traffic target detection system based on the fusion of multiple attention mechanisms, comprising:

[0043] A data acquisition module is used to obtain road traffic data set images and preprocess the road traffic data set images;

[0044] A model building module is used to build a dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction backbone network with auxiliary network branches, a neck network, and a head detection network;

[0045] A training module is used to train the two-branch road traffic target detection model using the preprocessed road traffic dataset images to obtain a trained two-branch road traffic target detection model;

[0046] The detection module is used to input the road traffic image to be detected into the trained dual-branch road traffic target detection model; in the dual-branch parallel feature extraction Backbone network, the attention mechanism is used to capture the multi-scale feature details of the road traffic image to be detected, extract and fuse the features, and use the auxiliary network branch to supplement the semantic information extraction; in the neck network, the attention mechanism is introduced to fuse multi-scale features, and the head detection network is used to predict the candidate box of the road traffic target to generate the final road traffic target detection result.

[0047] Compared with the prior art, the beneficial effects of this method are:

[0048] The present invention provides a dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms. In the method proposed by the present invention, a dual-branch road traffic target detection model is constructed and trained. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction Backbone network with auxiliary network branches, a Neck network, and a Head detection network. The road traffic image to be detected is input into the trained dual-branch road traffic target detection model. In the dual-branch parallel feature extraction Backbone network, the multi-scale feature details of the road traffic image to be detected are captured based on the attention mechanism, the features are extracted and fused, and the auxiliary network branches are used to supplement the semantic information extraction; in the Neck network, the attention mechanism is introduced to fuse the multi-scale features, and the Head detection network is used to predict the candidate box of the road traffic target to generate the final road traffic target detection result. The present invention as a whole uses a dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms to improve the accuracy and efficiency of road traffic target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart showing a dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms proposed in an embodiment of the present invention;

[0050] Figure 2 A structural diagram showing a dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms proposed in an embodiment of the present invention;

[0051] Figure 3 Schematic diagram showing the structure of the EMA multi-scale attention layer proposed in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram showing the Swin Transformer structure proposed in an embodiment of the present invention;

[0053] Figure 5 An example diagram showing an image of a road traffic dataset proposed in an embodiment of the present invention;

[0054] Figure 6 A diagram showing the detection results of a road traffic target proposed in an embodiment of the present invention;

[0055] Figure 7 A schematic diagram showing an electronic device according to an embodiment of the present invention;

[0056] Figure 8 A structural diagram of a dual-branch road traffic target detection system based on the fusion of multiple attention mechanisms proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0058] In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size;

[0059] It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0060] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0061] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent;

[0062] Example 1

[0063] In this embodiment, a dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms is proposed. Figure 1 The flowchart shown includes the following steps:

[0064] S1: Acquire road traffic dataset images and preprocess the road traffic dataset images;

[0065] S2: Construct a dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction backbone network with auxiliary network branches, a neck network, and a head detection network;

[0066] S3: Using the preprocessed road traffic dataset images to train the dual-branch road traffic target detection model to obtain a trained dual-branch road traffic target detection model;

[0067] S4: Input the road traffic image to be detected into the trained dual-branch road traffic target detection model;

[0068] S5: In the dual-branch parallel feature extraction Backbone network, the attention mechanism is used to capture the multi-scale feature details of the road traffic image to be detected, extract and fuse the features, and use the auxiliary network branch to supplement the semantic information extraction; in the Neck network, the attention mechanism is introduced to fuse multi-scale features, and the Head detection network is used to predict the candidate box of the road traffic target to generate the final road traffic target detection result.

[0069] In this embodiment, if Figure 2The structural diagram of the dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms is shown. The dual-branch parallel feature extraction Backbone backbone network includes: a first Conv module, a first EMA multi-scale attention combination module, a second EMA multi-scale attention combination module, a third EMA multi-scale attention combination module and a fourth EMA multi-scale attention combination module; any EMA multi-scale attention module among the first EMA multi-scale attention combination module, the second EMA multi-scale attention combination module, the third EMA multi-scale attention combination module and the fourth EMA multi-scale attention combination module includes a second Conv layer, a first C2f layer and a first EMA multi-scale attention layer; the first EMA multi-scale attention combination module extracts the first-scale detail feature through the EMA multi-scale attention layer and inputs it into the second EMA multi-scale attention combination module, the second EMA multi-scale attention combination module extracts the second-scale detail feature through the EMA multi-scale attention layer and inputs it into the third EMA multi-scale attention combination module, the third EMA multi-scale attention combination module extracts the third-scale detail feature through the EMA multi-scale attention layer and inputs it into the fourth EMA multi-scale attention combination module, and the fourth EMA multi-scale attention combination module extracts the fourth-scale detail feature through the EMA multi-scale attention layer;

[0070] Also includes: a first CBLinear module, a second CBLinear module, a third CBLinear module, a fourth CBLinear module, a fifth CBLinear module and an auxiliary network branch;

[0071] The auxiliary network branch includes: a first Conv and CBFuse combination module, a first multi-scale extraction module, a second multi-scale extraction module, a third multi-scale extraction module, and a fourth multi-scale extraction module; any multi-scale extraction module among the first multi-scale extraction module, the second multi-scale extraction module, the third multi-scale extraction module, and the fourth multi-scale extraction module includes: a third Conv layer, a first CBFuse layer, a second C2f layer, and a second EMA multi-scale attention layer; the primary features output by the first Conv module are connected to the first Conv and CBFuse combination module through the first CBLinear module; the initial features output by the first Conv and CBFuse combination module and the first scale detail features output by the first EMA multi-scale attention combination module The input end of the first multi-scale extraction module is connected through the second CBLinear module; the detail semantic information features output by the first multi-scale extraction module and the second-scale detail features output by the second EMA multi-scale attention combination module are connected to the input end of the second multi-scale extraction module through the third CBLinear module; the detail semantic information features output by the second multi-scale extraction module and the third-scale detail features output by the third EMA multi-scale attention combination module are connected to the input end of the third multi-scale extraction module through the fourth CBLinear module; the detail semantic information features output by the third multi-scale extraction module and the fourth-scale detail features output by the fourth EMA multi-scale attention combination module are connected to the input end of the fourth multi-scale extraction module through the fifth CBLinear module.

[0072] The Neck network includes: a first SPPF module, a first upsampling fusion module, a second upsampling fusion module, a first C35TR module, a first multi-scale attention feature fusion module, and a second multi-scale attention feature fusion module connected in sequence;

[0073] Any upsampling fusion module in the first upsampling fusion module and the second upsampling fusion module includes: a first Upsample layer and a first Concat-C2f layer;

[0074] Any multi-scale attention feature fusion module in the first multi-scale attention feature fusion module and the second multi-scale attention feature fusion module includes: a fourth Conv layer, a second Concat-C2f layer and a second C35TR layer; the output end of the first upsampling fusion module is connected to the input end of the second multi-scale attention feature fusion module; the output end of the second upsampling fusion module is connected to the input end of the first multi-scale attention feature fusion module; the first multi-scale attention feature fusion module and the second multi-scale attention feature fusion module use the C35TR layer to fuse multi-scale features.

[0075] In this embodiment, any Conv module consists of a Conv convolution layer, a BN (batchnorm2d) layer, and a silu activation function layer. The Conv convolution layer extracts the features of the feature map, the BN (batchnorm2d) layer normalizes the output of the convolution layer, and the silu activation function layer performs a nonlinear transformation on the input feature map to increase the nonlinearity of the features.

[0076] The head detection network includes: a first head module, a second head module and a third head module.

[0077] In this embodiment, if Figure 3 The structural diagram of the EMA multi-scale attention layer shown in the figure, any EMA multi-scale attention layer among the first EMA multi-scale attention layer, the second EMA multi-scale attention layer, the third EMA multi-scale attention layer, the fourth EMA multi-scale attention layer, the fifth EMA multi-scale attention layer, the sixth EMA multi-scale attention layer, the seventh EMA multi-scale attention layer and the eighth EMA multi-scale attention layer includes: a first Re-weight operation, a first GroupNorm operation, a first AvgPool operation, a first Softmax operation, a first Matmul operation, a third Sigmoid operation and a second Re-weight operation connected in sequence; a fifth Conv operation, a second AvgPool operation, a second Softmax operation and a second Matmul operation connected in sequence; the output end of the first Groups operation is respectively connected to the input end of the first XAvgPool operation, the first Y AvgPool operation, the fifth Conv operation, the first Re-weight operation and the second Re-weight operation; the first X AvgPool operation and the first Y The output ends of the AvgPool operation are all connected to the input ends of the first Concat_Conv operation; the output ends of the first Concat_Conv operation are respectively connected to the input ends of the first Sigmoid operation and the second Sigmoid operation; the output ends of the first Sigmoid operation and the second Sigmoid operation are both connected to the input end of the first Re-weight operation; the output end of the first Re-weight operation is connected to the input end of the second Matmul operation; the output end of the sixth Conv operation is connected to the input end of the first Matmul operation; the output end of the second Matmul operation and the output end of the third Sigmoid operation are spliced and connected to the input end of the third Sigmoid operation.

[0078] In the EMA multi-scale attention layer: Let X∈R C×H×W Represents the input feature map, which is divided into G sub-feature maps according to the channel dimension. The calculation expression for the division is:

[0079] X=[X0,X1,...,X G-1 ],X i ∈R C / / G×H×W ,G<<C

[0080] In the formula, G represents the number of sub-features, X i represents the i-th sub-feature;

[0081] Three parallel branches are used for feature encoding. The feature encoding process is as follows:

[0082] In the first 1×1 feature encoding branch, one-dimensional global average pooling is performed on the channels in the horizontal direction. In the second 1×1 feature encoding branch, one-dimensional global average pooling is performed on the channels in the vertical direction. The calculation expression of global average pooling is:

[0083]

[0084] Where, represents the output of the Cth channel with height H, Represents the output of the Cth channel with width w;

[0085] The feature encoding after the global average pooling operation is spliced along the height direction, and then decomposed into two vectors through convolution operation. The two vectors are linearly fitted using the nonlinear Sigmoid function, and finally the linearly fitted features are aggregated using multiplication;

[0086] The aggregated features are subjected to a two-dimensional global average pooling operation and a Softmax function linear transformation. The calculation expression of the two-dimensional global average pooling operation is:

[0087]

[0088] Where Z C Represents the output of the Cth channel.

[0089] In the 3×3 feature encoding branch, local features of the input feature map are obtained through convolution, and the local features are subjected to a two-dimensional global average pooling operation and a Softmax linear transformation;

[0090] The output features of the 1×1 feature encoding branch and the 3×3 feature encoding branch are multiplied by matrix dot product operation, and the output feature map containing dual spatial attention weights is output.

[0091] In this embodiment, in the EMA multi-scale attention layer, the first Groups operation divides the input feature map into G groups. An average pooling operation is performed horizontally through the first X AvgPool operation to reduce the width of the feature map, and an average pooling operation is performed vertically through the first Y AvgPool operation to reduce the height of the feature map. The horizontally and vertically pooled feature maps are concatenated through the first Concat_Conv operation, and then feature transformation is performed through a 1×1 convolution to obtain a feature map with a shape of C / G×H×W. The transformed feature map is passed through the sigmoid function in the first and second Sigmoid operations to obtain an attention weight map with the same shape as the input feature map. The weight values range from 0 to 1, indicating the importance of different positions. Simultaneously, the sixth Conv operation uses a 3×3 convolution kernel to convolve the original feature map to extract local features, resulting in a feature map with a shape of C / G×H×W. The first and second AvgPool operations perform global average pooling on the input feature map, resulting in a feature vector of shape C / G × 1 × 1, which is used to capture global channel information. The Softmax function in the first and second Softmax operations is used to obtain channel attention weights, which are used to adjust the importance of different channels. The first and second Matmul operations perform matrix multiplication on the channel attention weights and the input feature map, reweighting the channels of the feature map. The output features of the two branches are combined to form a feature map with dual spatial attention weights, which helps the model capture pixel-level relationships in the image and enhances feature representation capabilities.

[0092] The feature maps extracted by the Backbone network are fed into the auxiliary subnetwork via the CBLinear module, which enhances network information flow through reversible connections. The auxiliary network first passes through a Conv module and a CBFuse module. During the training phase, the CBFuse module upsamples feature maps from different layers to the same size and then fuses them to enhance the model's object recognition capabilities. Finally, the extracted image feature information is fed into the Neck network via the auxiliary network.

[0093] In this embodiment, if Figure 4 Schematic diagram of the Swin Transformer structure shown, the first C35TR module and the second C35TR layer both include: a first C3 structure and a first Swin Transformer structure;

[0094] The first Swin Transformer structure includes: a first PartitionPatch operation, a first LinearBmbedding operation, a first SwinTransformerBlock combination operation, a first patch Merging operation, a second SwinTransformerBlock combination operation, a second patch Merging operation, a third SwinTransformerBlock combination operation, a third patchMerging operation and a fourth SwinTransformerBlock combination operation, which are connected in sequence.

[0095] In the Swin Transformer structure: the input H×W×3 feature map is split into feature maps of dimension H / 4×W / 4×48 through the first PartitionPatch operation, and then the segmented feature map is straightened and linearly transformed through the Linear Bmbedding operation. The SwinTransformerBlock operation extracts the local and global feature relationships of the feature map, and the convolution-free downsampling operation is performed through the patch Merging operation. The SwinTransformerBlock operation and patch Merging operation are repeated, and finally the feature map of dimension H / 32×W / 32×8C is output.

[0096] In this embodiment, the PartitionPatch operation divides the input feature map into non-overlapping local blocks, reducing the complexity of subsequent calculations. The Linear Bmbedding operation converts the original pixels into semantic embeddings, providing feature representation for subsequent self-attention calculations. The SwinTransformerBlock operation includes window self-attention W-MSA and shifted window partitioning SW-MSA. W-MSA windowizes the segmented feature map so that the segmented feature map calculates attention in a fixed window. SW-MSA moves the window to the lower right corner to fuse information across windows with adjacent windows, while maintaining the computational efficiency of non-overlapping windows, improving the performance of the network, and avoiding limiting attention to each window. Patch Merging reduces the resolution through convolution-free downsampling, expands the receptive field, and increases the number of feature channels. Features of the feature map are extracted through multiple SwinTransformerBlock operations.

[0097] Finally, the output of the Neck network is input into the Head part, and the input passes through the RepVGG structure. RepVGG adopts a residual structure and obtains feature information of receptive fields of different ranges by applying multiple different branches and convolution kernels to the model, thereby achieving high performance benefits in model training and inference.

[0098] Example 2

[0099] In this embodiment, the original image data is obtained by shooting with a camera placed at a traffic intersection. The non-motorized vehicles and pedestrians in the image are annotated using the Vott annotation tool. The annotated categories are electrocar, bike, person, and car. A traffic target detection image dataset is constructed based on the original image data obtained by shooting and the annotated information. There are 5829 images in this dataset, such as Figure 5 The following figure shows an example of a road traffic dataset image. The dataset is randomly divided into training, test, and validation sets in a ratio of 7:2:1.

[0100] This embodiment uses a training set to train the constructed dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms, and during the training process, uses a validation set to verify the effectiveness of the dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms.

[0101] The road traffic dataset images are input into the dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms for detection to obtain category information and detection boxes; the detection results and annotation information are input into the Slide Loss classification loss function to calculate the loss value.

[0102] In this embodiment, during the training process, the Slide Loss classification loss function is used as the classification loss function of the dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms. The expression of the Slide Loss classification loss function is:

[0103]

[0104] Where x represents the IoU value and μ represents the threshold parameter.

[0105] The Slide Loss classification loss function defines the average of all bounding box IoU values as a threshold, treating samples with less than an IoU as negative samples and samples with greater than an IoU as positive samples. A weighting function is then used to emphasize samples at the boundary. The IoU between the predicted bounding box and the target bounding box is first calculated and passed as a weight to the loss function. The weight is dynamically adjusted based on the IoU value: a smaller weight is assigned for lower IoU values, while an exponentially decreasing weight is assigned for higher IoU values. This loss function more flexibly responds to predictions of varying confidence levels, fine-tunes the loss, and incorporates bounding box regression information, thereby improving the accuracy and robustness of object detection.

[0106] Build the Pytorch framework for training, set the learning rate to 0.01, the weight decay coefficient to 0.0005, the image size to 640×640, the batch size to 8, and the training cycle to 300. Figure 6 As shown in the detection result diagram of road traffic targets, the road image is input into the dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms for target detection. The required road targets in the image are marked with rectangular boxes. According to the detection results, the dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms can accurately grasp the global information and local information, accurately detect road traffic targets, and improve the accuracy and efficiency of target detection.

[0107] Example 3

[0108] The embodiment of the present invention further provides an electronic device, such as Figure 7 The schematic diagram of the electronic device shown includes a memory 101, a processor 102, and a computer program stored in the memory 101 and running on the processor 102, wherein when the processor 102 executes the computer program, the steps of the dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms proposed in this embodiment are implemented.

[0109] Specifically, in the present embodiment, the processor 102 may include a central processing unit (CPU) or a specific integrated circuit, or be configured to implement one or more integrated circuits of the present embodiment, and the memory 101 may include a large-capacity memory for data or instructions. It may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 101 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 101 may be inside or outside the integrated gateway disaster recovery device.

[0110] Example 4

[0111] In this embodiment, a dual-branch road traffic target detection system based on the fusion of multiple attention mechanisms is provided. Figure 8 The structural diagram of the system shown includes:

[0112] A data acquisition module is used to obtain road traffic data set images and preprocess the road traffic data set images;

[0113] A model building module is used to build a dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction backbone network with auxiliary network branches, a neck network, and a head detection network;

[0114] A training module is used to train the two-branch road traffic target detection model using the preprocessed road traffic dataset images to obtain a trained two-branch road traffic target detection model;

[0115] The detection module is used to input the road traffic image to be detected into the trained dual-branch road traffic target detection model; in the dual-branch parallel feature extraction Backbone network, the attention mechanism is used to capture multi-scale feature details, extract and fuse features, and use auxiliary network branches to supplement semantic information extraction; in the neck network, the attention mechanism is introduced to fuse multi-scale features, and the head detection network is used to predict the road traffic target candidate box to generate the final road traffic target detection result.

[0116] The embodiments are provided merely to illustrate the present invention and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications may be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the claims.

Claims

1. A dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms, characterized in that: The following steps are involved: Acquire road traffic dataset images and preprocess the road traffic dataset images; A dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms is constructed. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction backbone network with auxiliary network branches, a neck network, and a head detection network; The preprocessed road traffic dataset images are used to train the dual-branch road traffic target detection model to obtain a trained dual-branch road traffic target detection model; Input the road traffic image to be detected into the trained dual-branch road traffic target detection model; In the dual-branch parallel feature extraction Backbone network, the attention mechanism is used to capture the multi-scale feature details of the road traffic image to be detected, extract and fuse the features, and use the auxiliary network branch to supplement the semantic information extraction; in the Neck network, the attention mechanism is introduced to fuse multi-scale features, and the Head detection network is used to predict the candidate boxes of road traffic targets to generate the final road traffic target detection results.

2. According to the dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms in claim 1, the dual-branch parallel feature extraction backbone network comprises: a first Conv module, a first EMA multi-scale attention combination module, a second EMA multi-scale attention combination module, a third EMA multi-scale attention combination module and a fourth EMA multi-scale attention combination module; any EMA multi-scale attention module among the first EMA multi-scale attention combination module, the second EMA multi-scale attention combination module, the third EMA multi-scale attention combination module and the fourth EMA multi-scale attention combination module includes a second Conv layer, a first C2f layer and a first EMA multi-scale attention layer; the first EMA multi-scale attention combination module extracts first-scale detail features through the EMA multi-scale attention layer and inputs them into the second EMA multi-scale attention combination module, the second EMA multi-scale attention combination module extracts second-scale detail features through the EMA multi-scale attention layer and inputs them into the third EMA multi-scale attention combination module, the third EMA multi-scale attention combination module extracts third-scale detail features through the EMA multi-scale attention layer and inputs them into the fourth EMA multi-scale attention combination module, and the fourth EMA multi-scale attention combination module extracts fourth-scale detail features through the EMA multi-scale attention layer; Also includes: a first CBLinear module, a second CBLinear module, a third CBLinear module, a fourth CBLinear module, a fifth CBLinear module and an auxiliary network branch; The auxiliary network branch includes: a first Conv and CBFuse combination module, a first multi-scale extraction module, a second multi-scale extraction module, a third multi-scale extraction module, and a fourth multi-scale extraction module; any multi-scale extraction module among the first multi-scale extraction module, the second multi-scale extraction module, the third multi-scale extraction module, and the fourth multi-scale extraction module includes: a third Conv layer, a first CBFuse layer, a second C2f layer, and a second EMA multi-scale attention layer; the primary features output by the first Conv module are connected to the first Conv and CBFuse combination module through the first CBLinear module; the initial features output by the first Conv and CBFuse combination module and the first scale detail features output by the first EMA multi-scale attention combination module The input end of the first multi-scale extraction module is connected through the second CBLinear module; the detail semantic information features output by the first multi-scale extraction module and the second-scale detail features output by the second EMA multi-scale attention combination module are connected to the input end of the second multi-scale extraction module through the third CBLinear module; the detail semantic information features output by the second multi-scale extraction module and the third-scale detail features output by the third EMA multi-scale attention combination module are connected to the input end of the third multi-scale extraction module through the fourth CBLinear module; the detail semantic information features output by the third multi-scale extraction module and the fourth-scale detail features output by the fourth EMA multi-scale attention combination module are connected to the input end of the fourth multi-scale extraction module through the fifth CBLinear module.

3. The dual-branch road traffic target detection method based on multiple attention mechanism fusion according to claim 1 is characterized in that: The Neck network includes: a first SPPF module, a first upsampling fusion module, a second upsampling fusion module, a first C35TR module, a first multi-scale attention feature fusion module, and a second multi-scale attention feature fusion module connected in sequence; Any upsampling fusion module in the first upsampling fusion module and the second upsampling fusion module includes: a first Upsample layer and a first Concat-C2f layer; Any multi-scale attention feature fusion module in the first multi-scale attention feature fusion module and the second multi-scale attention feature fusion module includes: a fourth Conv layer, a second Concat-C2f layer and a second C35TR layer; the output end of the first upsampling fusion module is connected to the input end of the second multi-scale attention feature fusion module; the output end of the second upsampling fusion module is connected to the input end of the first multi-scale attention feature fusion module; the first multi-scale attention feature fusion module and the second multi-scale attention feature fusion module use the C35TR layer to fuse multi-scale features.

4. The dual-branch road traffic target detection method based on the fusion of multiple attention mechanisms according to claim 2 is characterized in that: Any of the first EMA multi-scale attention layer, the second EMA multi-scale attention layer, the third EMA multi-scale attention layer, the fourth EMA multi-scale attention layer, the fifth EMA multi-scale attention layer, the sixth EMA multi-scale attention layer, the seventh EMA multi-scale attention layer and the eighth EMA multi-scale attention layer includes: a first Re-weight operation, a first GroupNorm operation, a first AvgPool operation, a first Softmax operation, a first Matmul operation, a third Sigmoid operation and a second Re-weight operation connected in sequence; a fifth Conv operation, a second AvgPool operation, a second Softmax operation and a second Matmul operation connected in sequence; the output end of the first Groups operation is connected to the input end of the first X AvgPool operation, the first Y AvgPool operation, the fifth Conv operation, the first Re-weight operation and the second Re-weight operation respectively; the first XAvgPool operation and the first Y The output ends of the AvgPool operation are all connected to the input ends of the first Concat_Conv operation; the output ends of the first Concat_Conv operation are respectively connected to the input ends of the first Sigmoid operation and the second Sigmoid operation; the output ends of the first Sigmoid operation and the second Sigmoid operation are both connected to the input end of the first Re-weight operation; the output end of the first Re-weight operation is connected to the input end of the second Matmul operation; the output end of the sixth Conv operation is connected to the input end of the first Matmul operation; the output end of the second Matmul operation and the output end of the third Sigmoid operation are spliced and connected to the input end of the third Sigmoid operation.

5. The dual-branch road traffic target detection method based on multiple attention mechanism fusion according to claim 2 is characterized in that: The first C35TR module and the second C35TR layer each include: a first C3 structure and a first SwinTransformer structure; The first Swin Transformer structure includes: a first PartitionPatch operation, a first LinearBmbedding operation, a first SwinTransformerBlock combination operation, a first patch Merging operation, a second SwinTransformerBlock combination operation, a second patch Merging operation, a third SwinTransformerBlock combination operation, a third patchMerging operation and a fourth SwinTransformerBlock combination operation, which are connected in sequence.

6. The dual-branch road traffic target detection method based on multiple attention mechanism fusion according to claim 4 is characterized in that: In the EMA multi-scale attention layer: Let X∈R C×H×W Represents the input feature map, which is divided into G sub-feature maps according to the channel dimension. The calculation expression for the division is: X=[X0,X1,...,X G-1 ],X i ∈R C / / G×H×W ,G<<C In the formula, G represents the number of sub-features, X i represents the i-th sub-feature; Three parallel branches are used for feature encoding. The feature encoding process is as follows: In the first 1×1 feature encoding branch, one-dimensional global average pooling is performed on the channels in the horizontal direction. In the second 1×1 feature encoding branch, one-dimensional global average pooling is performed on the channels in the vertical direction. The calculation expression of global average pooling is: Where, represents the output of the Cth channel with height H, Represents the output of the Cth channel with width w; The feature encoding after the global average pooling operation is spliced along the height direction, decomposed into two vectors using convolution operation, and linearly fitted with the nonlinear Sigmoid function. Finally, the linearly fitted features are aggregated using multiplication. The aggregated features are subjected to a two-dimensional global average pooling operation and a Softmax function linear transformation. The calculation expression of the two-dimensional global average pooling operation is: Where Z C Represents the output of the Cth channel; In the 3×3 feature encoding branch, local features of the input feature map are obtained through convolution, and the local features are subjected to a two-dimensional global average pooling operation and a Softmax linear transformation; The output features of the 1×1 feature encoding branch and the 3×3 feature encoding branch are multiplied by matrix dot product operation to extract and fuse feature details from dual spatial scales and different channel dimensions.

7. The dual-branch road traffic target detection method based on multiple attention mechanism fusion according to claim 5 is characterized in that: In the Swin Transformer structure: the input size is H×W×3 feature map, the feature map is split into feature maps of dimension H / 4×W / 4×48 through the PartitionPatch operation, and then the segmented feature map is straightened and linearly transformed through the LinearBmbedding operation. The SwinTransformerBlock combination operation extracts the local and global feature relationship of the feature map, and the convolution-free downsampling operation is performed through the patch Merging operation. The SwinTransformerBlock combination operation and patch Merging operation are repeated, and the final output is a feature map of dimension H / 32×W / 32×8C.

8. The dual-branch road traffic target detection method based on multiple attention mechanism fusion according to claim 1 is characterized in that: During the training process, the Slide Loss classification loss function is used as the classification loss function of the dual-branch road traffic target detection model. The expression of the Slide Loss classification loss function is: Where x represents the IoU value and μ represents the threshold parameter.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and running on the processor. When the dual-branch road traffic target detection program based on the fusion of multiple attention mechanisms is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A dual-branch road traffic target detection system based on multiple attention mechanism fusion, used to implement the dual-branch road traffic target detection method based on multiple attention mechanism fusion according to any one of claims 1 to 8, characterized in that: include: A data acquisition module is used to obtain road traffic data set images and preprocess the road traffic data set images; A model building module is used to build a dual-branch road traffic target detection model based on the fusion of multiple attention mechanisms. The dual-branch road traffic target detection model includes: a dual-branch parallel feature extraction backbone network with auxiliary network branches, a neck network, and a head detection network; A training module is used to train the two-branch road traffic target detection model using the preprocessed road traffic dataset images to obtain a trained two-branch road traffic target detection model; The detection module is used to input the road traffic image to be detected into the trained dual-branch road traffic target detection model; in the dual-branch parallel feature extraction Backbone network, the attention mechanism is used to capture the multi-scale feature details of the road traffic image to be detected, extract and fuse the features, and use the auxiliary network branch to supplement the semantic information extraction; in the neck network, the attention mechanism is introduced to fuse multi-scale features, and the head detection network is used to predict the candidate box of the road traffic target to generate the final road traffic target detection result.

Images