Multi-spectral feature fusion multi-scale target detection method based on universal attention mechanism

The multi-spectral feature fusion method addresses inefficiencies in drone target detection by employing a pan-attention mechanism with GFT networks and optimized modules, improving detection accuracy and robustness for small targets.

CN119091122AActive Publication Date: 2024-12-06ZHEJIANG UNIV

Patent Information

Application Number
CN202411219092.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-12-06
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

Existing target detection methods using multispectral fusion of visible light and infrared images for drones suffer from inefficiencies in feature fusion, computational overhead, and poor performance in detecting small targets, limiting their applicability in practical scenarios.

Method used

A multi-spectral feature fusion method based on a pan-attention mechanism, utilizing a GFT feature fusion network with proxy and skip attention, BiFPN, VoVGSCSP, and GSConv modules, and a RepConv module, along with the Wise IoU v3 loss function, to enhance feature extraction and detection accuracy for small targets.

Benefits of technology

The method improves the detection accuracy and robustness of small targets by reducing computational overhead, enhancing feature fusion, and optimizing the detection process for drone applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091122A_ABST
    Figure CN119091122A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-spectral feature fusion multi-scale target detection method based on a universal attention mechanism, and the method comprises the steps: enabling a backbone network to read a visible light image and an infrared image, and carrying out the parallel double-current network feature extraction; constructing a GFT feature fusion network which is based on Transform and introduces agency attention, an attention skipping structure and cross attention, and fusing features extracted by the double-flow network in different stages; a BiFPN module, a VoVGSCSP module and a GSConv module are introduced into the neck network, fusion features in different stages are used as input of the neck network and are enhanced, and diversified features of the enhanced features in different stages are captured through a RepConv module; and finally, obtaining a final target detection result through a detection head and an output layer. The network structure provided by the invention has stronger environmental adaptability in target detection capability, and improves the robustness and detection precision of target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image target recognition, and in particular relates to a multi-spectral feature fusion multi-scale target detection method based on a pan-attention mechanism. Background Art

[0002] With the development of drone technology, target detection and recognition technology applicable to drones has gradually attracted attention. Traditional target detection data sources are mostly visible light images, but images taken by drones in the air are easily affected by tree cover, smoke, etc., and visible light images of night reconnaissance will be greatly affected by insufficient light, thus affecting the detection results.

[0003] With the evolution of machine learning algorithms, the fusion of multispectral information of visible light and infrared images has become one of the hot technologies for target detection, because the two modalities have different characteristics and complementary information. Infrared light has strong penetration and can penetrate some obstructions, such as mist and smoke, so it can still capture the contour and position information of the target in harsh environments, thus making up for the shortcomings of visible light in harsh environments such as occlusion, smoke, and poor visibility. On the other hand, visible light provides richer texture features of the target, making up for the lack of infrared light feature expression ability, and providing an important basis for the fine recognition and classification of the target. Therefore, the organic fusion of the image information of these two modalities can make full use of their respective advantages and improve the accuracy and robustness of target detection.

[0004] Considering the relatively small size of the target under the perspective of the drone, the simple method of adding feature information or dot product cannot well fuse the feature information of the two modalities. Zhang H et al. proposed a new cyclic fusion and refinement module in the paper Multispectral fusion for object detection with cyclic fuse-and-refine blocks published by ICIP to improve multispectral feature fusion and take into account the complementarity and consistency balance of features. However, this method failed to effectively fuse the information of infrared images and visible light images, and it was difficult to fully tap the complementary advantages of the two modalities. Fang Q et al. proposed using Transformer to fuse the intra-modal features of infrared images and visible light images in the paper Cross-modality fusion transformer for multispectral object detection published by arxiv, which effectively improved the detection accuracy of pedestrian targets under low illumination conditions. However, this method did not learn the features between modalities, resulting in room for improvement in the fusion effect. On the other hand, the multi-head attention layer in Transformer brings a large computational overhead, which limits its application in actual detection tasks. In addition, the network has weak detection capabilities for small targets and poor applicability for detecting small targets under the perspective of drones, which is difficult to meet the needs of practical applications.

[0005] Therefore, future research needs to explore the fusion mechanism of multimodal data in more depth and develop more efficient, accurate and robust fusion algorithms. This requires not only that the algorithms be able to fully explore and utilize the complementary information of the two modalities, but also that the algorithm's computational efficiency and the needs of actual application scenarios be considered, thereby further promoting the development of small target detection technology in scenarios such as drones. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism. A new GFT feature fusion network that integrates proxy attention, skip attention and cross attention is designed in the backbone network. BiFPN, VoVGSCSP and GSConv are used in the neck network for improvement to improve the detection ability of small targets. A RepConv module is added before the detection head, and the re-parameterization technology is used to improve the computational efficiency and accuracy of the convolution layer. The Wise IoU v3 loss function is used for model training to improve the prediction accuracy, which is suitable for target detection from the perspective of drones.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] The embodiment of the present invention provides a multi-spectral feature fusion multi-scale target detection method based on a pan-attention mechanism, characterized in that it includes the following steps:

[0009] Based on the YOLOv5 model framework, a multi-spectral dual-stream detection network consisting of a backbone network, a neck network, a detection head, and an output layer was built;

[0010] Among them, the backbone network reads the visible light image and the infrared image and performs parallel dual-stream network feature extraction. A GFT feature fusion network based on Transformer and introducing proxy attention, skip attention structure and cross attention is constructed to further extract the original features extracted by the dual-stream network at different stages and fuse them with the original features through the residual structure to obtain the fusion features at different stages.

[0011] The neck network introduces BiFPN module, VoVGSCSP module and GSConv module to replace the Concat cascade structure, C3 module and Conv module of the original YOLOv5 neck network, and uses the fusion features of different stages as the input of the neck network and performs feature enhancement. The enhanced features of different stages are then passed through the RepConv module to capture diversified features.

[0012] The detection head receives the output of the RepConv module corresponding to the features of different stages and processes it to generate detection results;

[0013] The output layer fuses the detection results corresponding to the features at different stages to obtain the final target detection result.

[0014] Specifically, in the backbone network, it includes:

[0015] The visible light image F R Input the first Focus module to extract the feature map F R1 , F R1 Input the first Conv+C3 module to extract the feature map F R2 , F R2 After the feature map is extracted by the second Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of visible light features and the output of the second Conv+C3 module is fused with the visible light feature output of the first GFT feature fusion network through the residual structure to obtain the feature map F. R3 , F R3 After the feature map is extracted by the third Conv+C3 module, it is input into the second GFT feature fusion network to cross-fuse the visible light features with the infrared features extracted at the same stage, and the output of the third Conv+C3 module and the visible light feature output of the second GFT feature fusion network are fused through the residual structure to obtain the feature map F. R4 , F R4After the feature map is extracted by the first Conv+SPP+C3 module, it is input into the third GFT feature fusion network for internal feature fusion of visible light features and the output of the first Conv+SPP+C3 module is fused with the visible light feature output of the third GFT feature fusion network through the residual structure to obtain the feature map F. R5 ;

[0016] The infrared image F T Input the second Focus module to extract the feature map F T1 , F T1 Input the fourth Conv+C3 module to extract the feature map F T2 , F T2 After the feature map is extracted by the fifth Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of infrared features and the output of the fifth Conv+C3 module is fused with the infrared feature output of the first GFT feature fusion network through the residual structure to obtain the feature map F T3 , F T3 After the feature map is extracted by the sixth Conv+C3 module, it is input into the second GFT feature fusion network to cross-fuse the infrared features with the visible light features extracted at the same stage, and the output of the sixth Conv+C3 module and the infrared feature output of the second GFT feature fusion network are fused through the residual structure to obtain the feature map F. T4 , F T4 The second Conv+SPP+C3 module is input to extract the feature map, and then input to the third GFT feature fusion network for internal feature fusion of infrared features. The output of the second Conv+SPP+C3 module is fused with the infrared feature output of the third GFT feature fusion network through the residual structure to obtain the feature map F. T5 .

[0017] Specifically, in the first GFT feature fusion network or the third GFT feature fusion network, including:

[0018] Use proxy attention modules to replace the multi-head self-attention modules in each layer of Transformer, and on this basis, introduce skip attention structure in the proxy attention module;

[0019] For the skip attention structure, the output of the l-1th layer proxy attention module in the Transformer is expressed as The Transformer at layer l is represented as a residual structure:

[0020]

[0021] Z l ′←MLP(Z l )+Z l

[0022] Among them, SkipAt(·) is the skip attention structure, MLP(·) is the multi-layer perceptron, and Z l is the attention output of the lth layer, Z l ′ is the Transformer output of the lth layer, and the skip attention acts on Make The features of are provided by the proxy attention module of the previous layer instead of being calculated by the proxy attention module of the current layer. The specific formula of the SkipAt(·) action layer is as follows:

[0023]

[0024] Among them, the first fully connected layer FC1(·) is used to expand the channel dimension, and then the depth-wise separable convolution module DwC(·) is used to obtain the relationship between tokens. The result will be flattened into a vector and input into the second fully connected layer FC2(·) to restore the channel dimension. Finally, the output of the current layer skipping attention is calculated by the ECA module ECA(·)

[0025] Specifically, the ECA module is implemented through ECA-Net, which aggregates features using global average pooling according to the channel dimension, then performs an adaptive 1×1 convolution kernel according to the channel dimension, and inputs the final GELU activation function.

[0026] Specifically, in the second GFT feature fusion network, it includes:

[0027] A Transformer-based cross-attention mechanism is adopted. The query, key and value of the visible light feature are defined as Q1, K1 and V1 respectively, and the query, key and value of the infrared feature are defined as Q2, K2 and V2 respectively. For the visible light feature part, Q1 is queried for K2 and combined with V2 for attention calculation to obtain the attention output of the visible light part. For the infrared feature part, Q2 is queried for K1 and combined with V1 for attention calculation to obtain the attention output of the infrared part.

[0028] Specifically, in the neck network, including:

[0029] F R3 and F T3 The fused features P3 and F R4 and F T4 The fused features P4 and F R5 and F T5The fused feature P5 is divided into three paths and input into the neck network. P3 is sequentially extracted by the first BiFPN module, the first VoVGSCSP module and the first RepConv module. P4 is sequentially extracted by the second BiFPN module, the second VoVGSCSP module, the third BiFPN module, the third VoVGSCSP module and the second RepConv module. P5 is sequentially extracted by the second GSConv module, the fourth BiFPN module, the fourth VoVGSCSP module and the third RepConv module.

[0030] Among them, the first BiFPN module is also combined with the output of the second VoVGSCSP module after passing through the first GSConv and the first upsampling module, the second BiFPN module is also combined with the output of the second GSConv after passing through the second upsampling module, the third BiFPN module is also combined with P4 and the output of the first VoVGSCSP module after passing through the first Conv module, and the fourth BiFPN module is also combined with the output of the third VoVGSCSP module after passing through the second Conv module.

[0031] Specifically, the detection head includes:

[0032] The first detection module, the second detection module and the third detection module receive outputs from the first RepConv module, the second RepConv module and the third RepConv module respectively.

[0033] Specifically, in the output layer, it includes:

[0034] The final target detection result is obtained by performing weighted average calculation on the detection results of the first detection module, the second detection module and the third detection module.

[0035] Specifically, during the training phase, each RepConv module uses multiple parallel convolutional layers to capture diverse features:

[0036] Y=X*W s +X*W p +X*W d

[0037] Among them, * represents the convolution operation, X represents the input feature map, and W represents s Standard convolution kernel, W p represents the point-by-point convolution kernel, W d Represents convolution kernels with different padding;

[0038] During the inference phase, the convolutional layers in each RepConv module are reparameterized into a single convolutional layer to accelerate the extraction of diverse features:

[0039] W rep =W s +W p +W d

[0040] Y=X*W rep

[0041] Among them, W rep represents a single convolutional layer after reparameterization.

[0042] Specifically, the Wise IoU v3 loss function is used to train the multi-spectral dual-stream detection network.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] In the first GFT feature fusion network and the third GFT feature fusion network in the GFT feature fusion network, the present invention utilizes the similarity between the multi-head attention layers to realize the skip attention structure to reduce the overhead of the feature fusion network, and uses the proxy attention module to replace the multi-head self-attention module to accelerate the calculation of the attention module in each layer of the Transformer; the multi-head self-attention layer of the second GFT feature fusion network in the GFT feature fusion network uses the cross attention structure, which increases the information interaction between the modalities and improves the feature fusion ability and detection accuracy of the dual-stream network; the present invention uses the BiFPN module, the VoVGSCSP module and the GSConv module in the neck network to improve the ability to detect small targets and balance the parameter quantity and detection accuracy; RepConv is used before the detection head to improve the calculation efficiency and accuracy of the convolution module; and the Wise IoU v3 loss function is used to improve the accuracy of the prediction frame. Therefore, the new multi-spectral dual-stream detection network proposed by the present invention is suitable for the detection of small targets of drones in dual-stream images, and improves the robustness and detection accuracy of target detection of drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a flowchart of a multi-scale target detection method based on a pan-attention mechanism and fusion of multi-spectral features provided by an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of the structure of a backbone network provided by an embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the structure of a first GFT feature fusion network and a third GFT feature fusion network provided in an embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of the structure of a second GFT feature fusion network provided by an embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of the structure of a neck network and a detection head provided by an embodiment of the present invention;

[0051] Figure 6 It is a comparison result diagram of the overall accuracy of the feature fusion network using the GFT under the LLVIP data set provided by the embodiment of the present invention and the feature fusion network using the CFT module in the prior art;

[0052] Figure 7 It is a comparison result diagram of the recall rate of the feature fusion network using the GFT under the LLVIP data set provided by the embodiment of the present invention and the feature fusion network using the CFT module in the prior art;

[0053] Figure 8 This is a comparison result diagram of the average accuracy of the GFT feature fusion network and the CFT module feature fusion network in the prior art at mAP0.5 under the LLVIP data set provided by the embodiment of the present invention;

[0054] Fig. 9 This is a comparison result of the average accuracy of the GFT feature fusion network and the CFT module in the prior art at mAP0.75 under the LLVIP dataset provided by the embodiment of the present invention;

[0055] Fig.10 This is a comparison result diagram of the average accuracy of the feature fusion network using the GFT feature fusion network and the feature fusion network using the CFT module in the prior art at mAP0.95 under the LLVIP dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0057] The inventive concept of the present invention is: in view of the fact that the small target detection method based on multispectral information fusion of visible light and infrared images in the prior art has insufficient accuracy and other problems, the embodiment of the present invention provides a multispectral feature fusion multi-scale target detection method based on a pan-attention mechanism, based on the YOLOv5 model framework, a GFT (Generalized Fusion Transformer) feature fusion network is constructed in the neck network to fuse the dual-stream image features, and the proxy attention, skip attention structure and cross attention are used to reduce the feature fusion network overhead, accelerate the attention calculation of each layer of Transformer and improve the feature fusion ability and detection accuracy of the dual-stream network. The BiFPN module, VoVGSCSP module and GSConv module are used in the neck network to improve the ability to detect small targets and balance the parameter quantity and detection accuracy. RepConv is used before the detection head to improve the computational efficiency and accuracy of the convolution module. And the Wise IoU v3 loss function is used to improve the accuracy of the prediction box, so that the multispectral dual-stream detection network including the backbone network, the neck network, the detection head and the output layer after training is more suitable for the small target detection of unmanned aerial vehicles with dual-stream images, and the robustness and detection accuracy of the target detection of unmanned aerial vehicles are improved.

[0058] Figure 1 Schematic diagram of the process of multi-spectral feature fusion multi-scale target detection method based on pan-attention mechanism provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a multi-spectral feature fusion multi-scale target detection method based on a pan-attention mechanism, comprising the following steps:

[0059] S1, build the backbone network of the multi-spectral dual-stream detection network. After the backbone network reads the visible light image and the infrared image, it performs parallel dual-stream network feature extraction. It builds a GFT feature fusion network based on Transformer and introduces proxy attention, skip attention structure and cross attention. At different stages, the original features extracted by the dual-stream network are further extracted and fused with the original features through the residual structure to obtain the fusion features of different stages.

[0060] In the embodiment, Figure 2 As shown, in the visible light image feature extraction part of the backbone network, the visible light image F R Input the first Focus module to extract pixels from the spatial dimension to the channel dimension at intervals and obtain the feature map F R1 , F R1 Input the first Conv+C3 module to extract the feature map F R2 , and then multi-scale feature extraction is performed through the adjacent Conv module C3 module to convert F R2After the feature map is extracted by the second Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of visible light features and the output of the second Conv+C3 module is fused with the visible light feature output of the first GFT feature fusion network through the residual structure to obtain the feature map F. R3 , F R3 After the feature map is extracted by the third Conv+C3 module, it is input into the second GFT feature fusion network to cross-fuse the visible light features with the infrared features extracted at the same stage, and the output of the third Conv+C3 module and the visible light feature output of the second GFT feature fusion network are fused through the residual structure to obtain the feature map F. R4 , F R4 After the feature map is extracted by the first Conv+SPP+C3 module, it is input into the third GFT feature fusion network for internal feature fusion of visible light features and the output of the first Conv+SPP+C3 module is fused with the visible light feature output of the third GFT feature fusion network through the residual structure to obtain the feature map F. R5 .

[0061] like Figure 2 As shown, in the infrared image feature extraction part of the backbone network, the infrared image F T Input the second Focus module to extract the feature map F T1 , F T1 Input the fourth Conv+C3 module to extract the feature map F T2 , F T2 After the feature map is extracted by the fifth Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of infrared features and the output of the fifth Conv+C3 module is fused with the infrared feature output of the first GFT feature fusion network through the residual structure to obtain the feature map F T3 , F T3 After the feature map is extracted by the sixth Conv+C3 module, it is input into the second GFT feature fusion network to cross-fuse the infrared features with the visible light features extracted at the same stage, and the output of the sixth Conv+C3 module and the infrared feature output of the second GFT feature fusion network are fused through the residual structure to obtain the feature map F. T4 , F T4 The second Conv+SPP+C3 module is input to extract the feature map, and then the third GFT feature fusion network is input to perform internal feature fusion of infrared features. The output of the second Conv+SPP+C3 module is fused with the infrared feature output of the third GFT feature fusion network through the residual structure to obtain the feature map F. T5 .

[0062] Finally, F R3 and F T3 The fused feature P3 is F R4and F T4 The fusion obtains feature P4 and transforms F R5 and F T5 The fusion results in feature P5.

[0063] In the embodiment, the structural design of the GFT feature fusion network follows the mid-fusion characteristic. The first GFT feature fusion network (GFT-1) and the third GFT feature fusion network (GFT-3) of the first and last layers perform feature fusion within the dual-stream network, and the second GFT feature fusion network (GFT-2) of the middle layer performs feature fusion between the dual-stream networks. The feature fusion steps include:

[0064] (1) Figure 3 As shown in Figure 1, in GFT-1 and GFT-3, the Agent Attention module is used to replace the Multi-Head Self-Attention module in each layer of the Transformer, and on this basis, the Skip Attention structure is introduced in the Agent Attention module.

[0065] For the skip attention structure, the output of the l-1th layer proxy attention module in the Transformer is expressed as The Transformer at layer l is represented as a residual structure:

[0066]

[0067] Z′ l ←MLP(Z l )+Z l

[0068] Among them, SkipAt(·) is the skip attention structure, MLP(·) is the multi-layer perceptron, and Z l is the attention output of the lth layer, Z′ l is the Transformer output of the lth layer, and the skip attention acts on it through the SkipAt(·) structure Make The features are provided by the proxy attention module of the previous layer instead of being calculated by the proxy attention module of the current layer. In the embodiment, the Transformer includes 8 layers in total. In GFT-1 and GFT-3, the 4th and 7th layers of the 8-layer proxy attention are selected as the action layers of SkipAt(·).

[0069] The specific formula of SkipAt(·) action layer is as follows:

[0070]

[0071] Among them, the first fully connected layer FC1(·) is used: Expand the channel dimension, n represents the number of samples of the input or output feature, d represents the feature dimension, and then use the depthwise separable convolution module DwC(·): The relationship between tokens is obtained, and the result is flattened into a vector and input into the second fully connected layer FC2(·): To restore the channel dimension, the output of the current layer skipping attention is finally calculated through the ECA (Efficient Channel Attention) module ECA(·) The ECA module is implemented through ECA-Net, which aggregates features using global average pooling according to the channel dimension, then performs an adaptive 1×1 convolution kernel according to the channel dimension and inputs the final GELU activation function.

[0072] (2) Figure 4 As shown in Figure 2, in GFT-2, a Transformer-based cross-attention mechanism is adopted. The module realizes the interaction of feature information between modules, so that the two-stream network learns the feature information of each other. In the Transformer, the query (Query, Q), key (Key, K) and value (Value, V) need to be passed into the multi-head self-attention layer, and different linear projection transformations are performed. After the projection transformation, Q, K and V are obtained to obtain D of different dimensions. k and D v , where Q, K and V are the input sequence The vector matrix obtained by linear transformation, i and d are used to represent the sequence length and feature size respectively. The specific formula is as follows:

[0073]

[0074] Among them, D k Denotes the matrix dimensions of Q and K, D v represents the matrix dimension of V, W Q , W K and W V are the parameter matrices after linear mapping.

[0075] The attention layer multiplies the matrix Q and the transposed matrix of the matrix K to obtain a coefficient, then multiplies it by the scale factor, normalizes it through the Softmax function, and then multiplies it by the matrix V to obtain the weighted value. The attention function formula through the Q, K and V matrix operations is:

[0076]

[0077] The superscript T indicates the matrix transpose.

[0078] In GFT-2, the query, key, and value of the visible light feature are defined as Q1, K1, and V1 respectively, and the query, key, and value of the infrared feature are defined as Q2, K2, and V2 respectively. For the visible light feature part, Q1 is used to query K2 and is combined with V2 for attention calculation to obtain the attention output of the visible light part. For the infrared feature part, Q2 is used to query K1 and is combined with V1 for attention calculation to obtain the attention output of the infrared part. The specific formula is as follows:

[0079] Z RGB ←GFT-2(Q1,K2,V2)

[0080] Z IR ←GFT-2(Q2,K1,V1)

[0081] Where GFT-2(·) represents the GFT-2 module, Z RGB represents the attention output of the visible light part, Z IR Represents the attention output of the IR part.

[0082] S2, build the neck network of the multi-spectral dual-stream detection network. The neck network introduces the BiFPN module, VoVGSCSP module and GSConv module to replace the Concat cascade structure, C3 module and Conv module of the original YOLOv5 neck network. The fused features of different stages are used as the input of the neck network and feature enhancement is performed. The enhanced features of different stages are then passed through the RepConv module to capture diversified features.

[0083] In the embodiment, Figure 5 As shown in the figure, the BiFPN (Bi-directional Feature Pyramid Network) module is used to replace the Concat cascade structure of the original YOLOv5 neck network to achieve top-down and bottom-up feature interaction, and learn input features at different levels. The final feature map output can combine the features of the current layer and the upper and lower layers by weighted fusion of input features at different levels. The specific formula is as follows:

[0084]

[0085] Among them, I i represents the i-th input feature, O represents the fusion feature output, and w i represents the learnable weight, j represents the index of the learnable weight, and ε represents a small parameter and is used to prevent the denominator from being too small to produce unstable output.

[0086] At the same time, the Slim-Neck structure is used, that is, the C3 module of the original YOLOv5 neck network is replaced by the VoVGSCSP module, and the Conv convolution module in P3 is replaced by the GSConv module, to improve the ability to detect small targets and balance the number of parameters and detection accuracy.

[0087] like Figure 5 As shown, in the neck network, P3 is sequentially subjected to feature extraction by the first BiFPN module, the first VoVGSCSP module and the first RepConv module, P4 is sequentially subjected to feature extraction by the second BiFPN module, the second VoVGSCSP module, the third BiFPN module, the third VoVGSCSP module and the second RepConv module, and P5 is sequentially subjected to feature extraction by the second GSConv module, the fourth BiFPN module, the fourth VoVGSCSP module and the third RepConv module. Among them, the first BiFPN module also combines the output of the second VoVGSCSP module after the first GSConv and the first upsampling module, the second BiFPN module also combines the output of the second GSConv after the second upsampling module, the third BiFPN module also combines P4 and the output of the first VoVGSCSP module after the first Conv module, and the fourth BiFPN module also combines the output of the third VoVGSCSP module after the second Conv module.

[0088] (3) For the first RepConv (Re-parameterization Convolutional) module, the second RepConv module and the third RepConv module designed as the last layer of the neck network before detecting the head, the re-parameterization technology is used to optimize the computational efficiency and accuracy of the convolutional layer. The core idea is to improve the model performance by using different convolution structures in the training stage and the inference stage.

[0089] During the training phase, each RepConv module uses multiple parallel convolutional layers to capture diverse features:

[0090] Y=X*W s +X*W p +X*W d

[0091] Among them, * represents the convolution operation, X represents the input feature map, and W represents s Standard convolution kernel, W p represents the point-by-point convolution kernel, W d Represents convolution kernels with different padding.

[0092] During the inference phase, the convolutional layers in each RepConv module are reparameterized into a single convolutional layer, which reduces the amount of computation and speeds up the inference process:

[0093] W rep =W s +W p +W d

[0094] Y=X*W rep

[0095] Among them, W rep represents a single convolutional layer after reparameterization.

[0096] Through this reparameterization technology, the RepConv module only needs to perform one convolution operation during the inference phase, which greatly improves the inference speed while maintaining the model's complex feature extraction capabilities during the training phase.

[0097] S3, build the detection head of the multi-spectral dual-stream detection network. The detection head receives the output of the RepConv module corresponding to different stages and processes it to generate detection results.

[0098] In the embodiment, the first detection module, the second detection module and the third detection module receive outputs from the first RepConv module, the second RepConv module and the third RepConv module respectively.

[0099] S4, builds the output layer of the multi-spectral dual-stream detection network. The output layer fuses the detection results corresponding to different stages to obtain the final target detection result.

[0100] In the embodiment, the final target detection result is obtained by performing weighted average calculation on the detection results of the first detection module, the second detection module and the third detection module.

[0101] S5, uses Wise IoU v3 loss function to train the multi-spectral dual-stream detection network.

[0102] In the embodiment, IoU is an indicator used to measure the overlap between the anchor box and the target box. It avoids the influence of the target box size in the form of ratio. Represents the Bounding Box Regression loss function:

[0103]

[0104] Among them, S u =(wh+w gt h gt -W i H i) represents the overlapped area between the anchor box and the target box, W i and H i Indicates the length and width of the overlapped area, the anchor box x and y represent the horizontal and vertical coordinates of the anchor box, w and h represent the length and width of the anchor box, and the target box x gt and gt Indicates the horizontal and vertical coordinates of the anchor box, w gt and h gt Indicates the length and width of the anchor box.

[0105] The Wise IoU v3 loss function is improved from the Wise IoU v1 loss function. The Wise IoU v1 loss function is expressed as The specific formula is as follows:

[0106]

[0107]

[0108] Among them, W g and H g is the minimum bounding box length and width, W g and H g Part of it is separated from the computational graph to avoid introducing the gradient generated by the aspect ratio and affecting the calculation of the covariance. The superscript * indicates this operation. Based on the distance attention mechanism, the anchor box of normal quality is significantly enlarged. When the anchor box and the target box are completely consistent, the dependence ratio of the high-quality anchor box can be significantly reduced, and tend to depend on the distance between the center points.

[0109] Based on this, the Wise IoU v3 loss function is used The specific formula is as follows:

[0110]

[0111] Among them, β is the coefficient of Dynamic non-monotonic Focusing Mechanism, which is used to describe the outlier ratio of the anchor box. is a dynamic parameter related to the average momentum, represents the loss function affected by the * operation, α and δ are two hyperparameters and are taken as 1.7 and 2.7 respectively in the embodiment.

[0112] For the trained multi-spectral dual-stream detection network, on the LLVIP dataset, with YOLOv5s as the benchmark weight, the multi-spectral dual-stream detection network including the GFT network and the detection model including the CFT network in the prior art are trained and tested respectively. The overall accuracy comparison results are shown in the figure below. Figure 6 As shown in Figure 7 As shown, the recall rate indicates the correct proportion of predictions in positive samples, and mAP0.5 is the average accuracy when the IoU value is 50%. Figure 8 As shown, mAP0.75 is the average accuracy when the IoU value is 75%. Fig. 9 As shown, mAP0.95 is the average accuracy when the IoU value is 95%. Fig.10 As shown, the higher these indicators are, the better. It can be seen that the method provided by the embodiment of the present invention has a basic improvement of about 2% in accuracy and a 2% improvement in recall, which can prove that the method has better performance and expression.

[0113] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-spectral feature fusion multi-scale target detection method based on pan-attention mechanism, characterized in that: The following steps are involved: Based on the YOLOv5 model framework, a multi-spectral dual-stream detection network consisting of a backbone network, a neck network, a detection head, and an output layer was built; Among them, the backbone network reads the visible light image and the infrared image and performs parallel dual-stream network feature extraction. A GFT feature fusion network based on Transformer and introducing proxy attention, skip attention structure and cross attention is constructed to further extract the original features extracted by the dual-stream network at different stages and fuse them with the original features through the residual structure to obtain the fusion features at different stages. The neck network introduces BiFPN module, VoVGSCSP module and GSConv module to replace the Concat cascade structure, C3 module and Conv module of the original YOLOv5 neck network, and uses the fusion features of different stages as the input of the neck network and performs feature enhancement. The enhanced features of different stages are then passed through the RepConv module to capture diversified features. The detection head receives the output of the RepConv module corresponding to the features of different stages and processes it to generate detection results; The output layer fuses the detection results corresponding to the features at different stages to obtain the final target detection result.

2. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 1 is characterized in that: In the backbone network, it includes: The visible light image F R Input the first Focus module to extract the feature map F R1 , F R1 Input the first Conv+C3 module to extract the feature map F R2 , F R2 After the feature map is extracted by the second Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of visible light features and the output of the second Conv+C3 module is fused with the visible light feature output of the first GFT feature fusion network through the residual structure to obtain the feature map F. R3 , F R3 After the feature map is extracted by the third Conv+C3 module, it is input into the second GFT feature fusion network to cross-fuse the visible light features with the infrared features extracted at the same stage, and the output of the third Conv+C3 module and the visible light feature output of the second GFT feature fusion network are fused through the residual structure to obtain the feature map F. R4 , F R4 After the feature map is extracted by the first Conv+SPP+C3 module, it is input into the third GFT feature fusion network for internal feature fusion of visible light features and the output of the first Conv+SPP+C3 module is fused with the visible light feature output of the third GFT feature fusion network through the residual structure to obtain the feature map F. R5 ; The infrared image F T Input the second Focus module to extract the feature map F T1 , F T1 Input the fourth Conv+C3 module to extract the feature map F T2 , F T2 After the feature map is extracted by the fifth Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of infrared features and the output of the fifth Conv+C3 module is fused with the infrared feature output of the first GFT feature fusion network through the residual structure to obtain the feature map F T3 , F T3 After the feature map is extracted by the sixth Conv+C3 module, it is input into the second GFT feature fusion network to cross-fuse the infrared features with the visible light features extracted at the same stage, and the output of the sixth Conv+C3 module and the infrared feature output of the second GFT feature fusion network are fused through the residual structure to obtain the feature map F. T4 , F T4 The second Conv+SPP+C3 module is input to extract the feature map, and then input to the third GFT feature fusion network for internal feature fusion of infrared features. The output of the second Conv+SPP+C3 module is fused with the infrared feature output of the third GFT feature fusion network through the residual structure to obtain the feature map F. T5 .

3. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 2 is characterized in that: In the first GFT feature fusion network or the third GFT feature fusion network, including: Use proxy attention modules to replace the multi-head self-attention modules in each layer of Transformer, and on this basis, introduce skip attention structure in the proxy attention module; For the skip attention structure, the output of the l-1th layer proxy attention module in the Transformer is expressed as The Transformer at layer l is represented as a residual structure: WITH' l ←MLP(Z l )+Z l Among them, SkipAt(·) is the skip attention structure, MLP(·) is the multi-layer perceptron, and Z l is the attention output of the lth layer, Z′ l is the Transformer output of the lth layer, and the skip attention acts on it through the SkipAt(·) structure Make The features of are provided by the proxy attention module of the previous layer instead of being calculated by the proxy attention module of the current layer. The specific formula of the SkipAt(·) action layer is as follows: Among them, the first fully connected layer FC1(·) is used to expand the channel dimension, and then the depth-wise separable convolution module DwC(·) is used to obtain the relationship between tokens. The result will be flattened into a vector and input into the second fully connected layer FC2(·) to restore the channel dimension. Finally, the output of the current layer skipping attention is calculated by the ECA module ECA(·) 4. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 3 is characterized in that: The ECA module is implemented through ECA-Net, which aggregates features using global average pooling according to the channel dimension, then performs an adaptive 1×1 convolution kernel according to the channel dimension, and inputs the final GELU activation function.

5. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 2 is characterized in that: In the second GFT feature fusion network, it includes: A Transformer-based cross-attention mechanism is adopted. The query, key and value of the visible light feature are defined as Q1, K1 and V1 respectively, and the query, key and value of the infrared feature are defined as Q2, K2 and V2 respectively. For the visible light feature part, Q1 is queried for K2 and combined with V2 for attention calculation to obtain the attention output of the visible light part. For the infrared feature part, Q2 is queried for K1 and combined with V1 for attention calculation to obtain the attention output of the infrared part.

6. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 2 is characterized in that: In the neck network, these include: F R3 and F T3 The fused features P3 and F R4 and F T4 The fused features P4 and F R5 and F T5 The fused feature P5 is divided into three paths and input into the neck network. P3 is sequentially extracted by the first BiFPN module, the first VoVGSCSP module and the first RepConv module. P4 is sequentially extracted by the second BiFPN module, the second VoVGSCSP module, the third BiFPN module, the third VoVGSCSP module and the second RepConv module. P5 is sequentially extracted by the second GSConv module, the fourth BiFPN module, the fourth VoVGSCSP module and the third RepConv module. Among them, the first BiFPN module is also combined with the output of the second VoVGSCSP module after passing through the first GSConv and the first upsampling module, the second BiFPN module is also combined with the output of the second GSConv after passing through the second upsampling module, the third BiFPN module is also combined with P4 and the output of the first VoVGSCSP module after passing through the first Conv module, and the fourth BiFPN module is also combined with the output of the third VoVGSCSP module after passing through the second Conv module.

7. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 6 is characterized in that: The detection head includes: The first detection module, the second detection module and the third detection module receive outputs from the first RepConv module, the second RepConv module and the third RepConv module respectively.

8. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 7 is characterized in that: In the output layer, we include: The final target detection result is obtained by performing weighted average calculation on the detection results of the first detection module, the second detection module and the third detection module.

9. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to claim 6 is characterized in that: During the training phase, each RepConv module uses multiple parallel convolutional layers to capture diverse features: Y=X*W s +X*W p +X*W d Among them, * represents the convolution operation, X represents the input feature map, and W represents s Standard convolution kernel, W p represents the point-by-point convolution kernel, W d Represents convolution kernels with different padding; During the inference phase, the convolutional layers in each RepConv module are reparameterized into a single convolutional layer to accelerate the extraction of diverse features: IN rep =In s +W p +W d Y=X*W rep Among them, W rep represents a single convolutional layer after reparameterization.

10. The multi-spectral feature fusion multi-scale target detection method based on the pan-attention mechanism according to any one of claims 1 to 9, characterized in that: The Wise IoU v3 loss function is used to train the multi-spectral dual-stream detection network.

Citation Information

Patent Citations

  • Visual task processing method and system based on agency attention

    CN117671371A

  • Multispectral target detection model training method, target detection method and system

    CN117911710A

  • Pedestrian detection method based on multispectral iteration cross feature fusion in severe environment

    CN118522035A

Cited By

  • Multispectral image change detection method and system

    CN119295445A

  • Visible light target recognition method and device based on deep learning, electronic equipment and storage medium

    CN119649006A

  • Lightweight aviation multispectral target detection method

    CN120032196A

  • Method and device for detecting power transmission line fault through line fault detection model

    CN120876400A