Multi-spectral feature fusion multi-scale target detection method based on general attention mechanism

Abstract

The application discloses a multispectral feature fusion multi-scale target detection method based on a general attention mechanism, and comprises the following steps: after a main network reads a visible light image and an infrared image, parallel double-flow network feature extraction is performed; a GFT feature fusion network based on a Transformer and introducing proxy attention, skip attention structure and cross attention is constructed to fuse the features extracted by the double-flow network at different stages; a neck network introduces a BiFPN module, a VoVGSCSP module and a GSConv module, takes the fused features at different stages as the input of the neck network and performs feature enhancement, and the enhanced features at different stages are captured by a RepConv module to capture diversified features; and finally, a detection head and an output layer are used to obtain the final target detection result. The network structure provided by the application has stronger environmental adaptability in target detection capability, and improves the robustness and detection precision of target detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image target recognition, and particularly relates to a multispectral feature fusion multi-scale target detection method based on a general attention mechanism. BACKGROUND

[0002] With the development of unmanned aerial vehicle technology, target detection and recognition technology suitable for unmanned aerial vehicle field has gradually attracted attention. Traditional target detection data sources are mostly visible light images, but images taken by unmanned aerial vehicles in the air are easily affected by forest shelter, smoke cover and the like, and night reconnaissance visible light images are greatly affected due to insufficient light, thereby affecting the detection result.

[0003] With the evolution of machine learning algorithms, multispectral information fusion of visible light and infrared images has become one of the hot technologies of target detection because the two modalities have different features and complementary information. Infrared light has strong penetration and can penetrate some sheltering objects such as thin fog, smoke and the like, so it can still capture the contour and position information of the target in a harsh environment, thereby making up for the deficiency of visible light in a harsh environment such as sheltering, smoke, poor visibility and the like. On the other hand, visible light provides more abundant texture features of the target, making up for the deficiency of infrared light in feature expression ability, and providing an important basis for fine identification and classification of the target. Therefore, organic fusion of image information of the two modalities can fully utilize their respective advantages, and improve the accuracy and robustness of target detection.

[0004] Considering the relatively small size of the target under the perspective of the UAV, the simple feature information addition or dot product method cannot well fuse the feature information of the two modalities. Zhang H et al. in the ICIP published Multispectral fusion for object detection with cyclic fuse-and-refine blocks proposed a new cyclic fusion and refinement module to improve multispectral feature fusion and consider the balance of feature complementarity and consistency, but this method fails to effectively fuse the information of infrared images and visible light images, making it difficult to fully exploit the complementary advantages of the two modalities. Fang Q et al. in arxiv published Cross-modality fusion transformer for multispectral object detection proposed to use the Transformer to fuse the intra-modal features of infrared images and visible light images, effectively improving the detection accuracy of pedestrian targets under low light conditions, but this method does not learn the inter-modal features, resulting in room for improvement in fusion effect. On the other hand, the multi-head attention layer in the Transformer brings a large computational overhead, limiting its application in real-time detection tasks. In addition, the network has weak detection ability for small targets and poor applicability for small targets under the perspective of the UAV, making it difficult to meet the needs of practical applications.

[0005] Therefore, future research needs to further explore the fusion mechanism of multi-modal data and develop more efficient, accurate and robust fusion algorithms. Not only does the algorithm need to fully exploit and utilize the complementary information of the two modalities, but also needs to consider the computational efficiency of the algorithm and the needs of the actual application scenario, so as to further promote the development of small target detection technology in UAV and other scenarios. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a multispectral feature fusion multi-scale target detection method based on a general attention mechanism. A new GFT feature fusion network is designed in the backbone network, which integrates proxy attention, skip attention and cross attention. BiFPN, VoVGSCSP and GSConv are used in the neck network to improve the detection ability of small targets. RepConv modules are added before the detection head to improve the computational efficiency and accuracy of the convolution layer using the reparameterization technology. Wise IoU v3 loss function is used for model training to improve the prediction accuracy, which is suitable for target detection under the perspective of the UAV.

[0007] To achieve the above invention purpose, the technical solutions provided by the present application are as follows:

[0008] The embodiment of the application provides a multispectral feature fusion multi-scale target detection method based on a general attention mechanism, and has the characteristics that the method comprises the following steps:

[0009] A multispectral double-flow detection network comprising a backbone network, a neck network, a detection head and an output layer is built based on a YOLOv5 model framework.

[0010] The backbone network performs parallel double-flow network feature extraction after reading visible light images and infrared images, and a GFT feature fusion network based on a Transformer and introducing proxy attention, skip attention structure and cross attention is constructed to further extract features of original features extracted by the double-flow network at different stages and fuse the original features through a residual structure to obtain fusion features at different stages.

[0011] The neck network replaces the Concat cascade structure, a C3 module and a Conv module of the original YOLOv5 neck network by introducing a BiFPN module, a VoVGSCSP module and a GSConv module, takes the fusion features at different stages as the input of the neck network and performs feature enhancement, and the enhanced features at different stages are further captured by a RepConv module to capture diversified features.

[0012] The detection head receives the output of the RepConv module corresponding to the features at different stages and processes to generate a detection result.

[0013] The output layer fuses the detection results corresponding to the features at different stages to obtain a final target detection result.

[0014] Specifically, in the backbone network, the following steps are included:

[0015] The visible light image F R is input into a first Focus module to extract a feature map F R1 . The F R1 is input into a first Conv+C3 module to extract a feature map F R2 . The F R2 is input into a second Conv+C3 module to extract a feature map, and then the feature map is input into a first GFT feature fusion network to perform internal feature fusion of visible light features and fuse the output of the second Conv+C3 module and the visible light feature output of the first GFT feature fusion network through a residual structure to obtain a feature map F R3 . The F R3 is input into a third Conv+C3 module to extract a feature map, and then the feature map is input into a second GFT feature fusion network to perform cross fusion of the visible light features and infrared features extracted at the same stage, and fuse the output of the third Conv+C3 module and the visible light feature output of the second GFT feature fusion network through a residual structure to obtain a feature map F R4 . The F R4The first Conv+SPP+C3 module extracts a feature map, and the third GFT feature fusion network is inputted to perform internal feature fusion of the visible light feature, and the output of the first Conv+SPP+C3 module and the visible light feature output of the third GFT feature fusion network are fused through a residual structure to obtain a feature map F R5 ;

[0016] The infrared image F T The second Focus module is inputted to extract a feature map F T1 , and F T1 The fourth Conv+C3 module is inputted to extract a feature map F T2 , and F T2 The fifth Conv+C3 module is inputted to extract a feature map, and the first GFT feature fusion network is inputted to perform internal feature fusion of the infrared feature, and the output of the fifth Conv+C3 module and the infrared feature output of the first GFT feature fusion network are fused through a residual structure to obtain a feature map F T3 , and F T3 The sixth Conv+C3 module is inputted to extract a feature map, and the second GFT feature fusion network is inputted to perform cross fusion of the infrared feature and the visible light feature extracted at the same stage, and the output of the sixth Conv+C3 module and the infrared feature output of the second GFT feature fusion network are fused through a residual structure to obtain a feature map F T4 , and F T4 The second Conv+SPP+C3 module is inputted to extract a feature map, and the third GFT feature fusion network is inputted to perform internal feature fusion of the infrared feature, and the output of the second Conv+SPP+C3 module and the infrared feature output of the third GFT feature fusion network are fused through a residual structure to obtain a feature map F T5 .

[0017] Specifically, in the first GFT feature fusion network or the third GFT feature fusion network, the following is included:

[0018] The proxy attention module is used to replace the multi-head self-attention module in each layer of the Transformer, and on this basis, the skip attention structure is introduced into the proxy attention module;

[0019] For the skip attention structure, the output of the proxy attention module in the l-1 layer of the Transformer is represented as The l-th layer of the Transformer is represented as a residual structure:

[0020]

[0021] Z l ←MLP(Z l )+Z l

[0022] where SkipAt(·) is the skip attention structure, MLP(·) is the multi-layer perceptron, Z l is the attention output of the l-th layer, Z l is the output of the l-th layer of the Transformer, the skip attention acts on Z such that the features of Z

[0023]

[0024] where the first fully connected layer FC1(·) is used to expand the channel dimension, and then the deep separable convolution module DwC(·) is used to obtain the relationship between the tokens, the result is flattened into a vector and input into the second fully connected layer FC2(·) to restore the channel dimension, and finally the output of the current layer of the skip attention is calculated through the ECA module ECA(·)

[0025] Specifically, the ECA module is implemented through ECA-Net, which aggregates features according to the channel dimension size using global average pooling, then performs adaptive 1x1 convolution kernel according to the channel dimension, and inputs the final GELU activation function.

[0026] Specifically, in the second GFT feature fusion network, it includes:

[0027] The cross attention mechanism based on the Transformer is adopted, the query, key and value of the visible light feature are defined as Q1, K1 and V1 respectively, the query, key and value of the infrared feature are defined as Q2, K2 and V2, the attention output of the visible light part is calculated by querying K2 through Q1 and combining with V2, and the attention output of the infrared part is calculated by querying K1 through Q2 and combining with V1.

[0028] Specifically, in the neck network, it includes:

[0029] The features P3, F R3 and F T3 are fused to obtain the feature P4, and the features P3, F R4 and F T4 are fused to obtain the feature P4, and the features P3, F R5 and F T5The obtained feature P5 is input into a neck network in three paths, P3 is sequentially subjected to feature extraction by a first BiFPN module, a first VoVGSCSP module and a first RepConv module, P4 is sequentially subjected to feature extraction by a second BiFPN module, a second VoVGSCSP module, a third BiFPN module, a third VoVGSCSP module and a second RepConv module, and P5 is sequentially subjected to feature extraction by a second GSConv module, a fourth BiFPN module, a fourth VoVGSCSP module and a third RepConv module;

[0030] The first BiFPN module is further combined with an output of the second VoVGSCSP module after being subjected to the first GSConv and the first up-sampling module, the second BiFPN module is further combined with an output of the second GSConv after being subjected to the second up-sampling module, the third BiFPN module is further combined with P4 and an output of the first VoVGSCSP module after being subjected to the first Conv module, and the fourth BiFPN module is further combined with an output of the third VoVGSCSP module after being subjected to the second Conv module.

[0031] Specifically, in the detection head, the detection head comprises:

[0032] The first detection module, the second detection module and the third detection module respectively receive outputs from the first RepConv module, the second RepConv module and the third RepConv module.

[0033] Specifically, in the output layer, the output layer comprises:

[0034] The final target detection result is obtained by weighted average calculation on detection results of the first detection module, the second detection module and the third detection module.

[0035] Specifically, in the training stage, each RepConv module uses a plurality of parallel convolution layers to capture diversified features:

[0036] Y = X * W s + X * W p + X * W d

[0037] Wherein, * represents a convolution operation, X represents an input feature map, W s represents a standard convolution kernel, W p represents a point-wise convolution kernel, W d represents a convolution kernel with different padding;

[0038] In the inference stage, the convolution layers in each RepConv module are re-parameterized into a single convolution layer to accelerate the extraction of diversified features:

[0039] W rep =W s +W p +W d

[0040] Y=X*W rep

[0041] wherein, W rep represents a single volume layer obtained by reparameterization.

[0042] Specifically, the multispectral dual-flow detection network is trained by using a Wise IoU v3 loss function.

[0043] Compared with the prior art, the present application has at least the following beneficial effects:

[0044] In the first GFT feature fusion network and the third GFT feature fusion network part of the GFT feature fusion network, the similarity between the multi-head attention layers is used to realize a skip attention structure to reduce the overhead of the feature fusion network, and a proxy attention module is used to replace the multi-head self-attention module to speed up the calculation of the attention module in each layer of the Transformer; in the multi-head self-attention layer of the second GFT feature fusion network part of the GFT feature fusion network, the structure of cross attention is used to increase the information interaction between modalities and improve the feature fusion capability and detection accuracy of the dual-flow network; in the neck network, BiFPN modules, VoVGSCSP modules and GSConv modules are used to improve the ability of small target detection and balance the parameter quantity and detection accuracy; RepConv is used before the detection head to improve the calculation efficiency and accuracy of the convolution module; and the Wise IoU v3 loss function is used to improve the accuracy of the prediction box. Therefore, the new multispectral dual-flow detection network provided by the present application is suitable for unmanned aerial vehicle small target detection of dual-flow images, and improves the robustness and detection accuracy of unmanned aerial vehicle target detection. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 is a flowchart of the multispectral feature fusion multi-scale target detection method based on the general attention mechanism provided by the embodiments of the present application;

[0047] Figure 2 is a structural diagram of the backbone network provided by the embodiments of the present application;

[0048] Figure 3 is a structural schematic diagram of the first GFT feature fusion network and the third GFT feature fusion network provided by the embodiment of the application;

[0049] Figure 4 is a structural schematic diagram of the second GFT feature fusion network provided by the embodiment of the application;

[0050] Figure 5 is a structural schematic diagram of the neck network and the detection head provided by the embodiment of the application;

[0051] Figure 6 is a comparison result diagram of the overall accuracy of the GFT feature fusion network and the feature fusion network using the CFT module of the prior art under the LLVIP data set provided by the embodiment of the application;

[0052] Figure 7 is a comparison result diagram of the recall rate of the GFT feature fusion network and the feature fusion network using the CFT module of the prior art under the LLVIP data set provided by the embodiment of the application;

[0053] Figure 8 is a comparison result diagram of the average accuracy of the GFT feature fusion network and the feature fusion network using the CFT module of the prior art at mAP0.5 under the LLVIP data set provided by the embodiment of the application;

[0054] Figure 9 is a comparison result diagram of the average accuracy of the GFT feature fusion network and the feature fusion network using the CFT module of the prior art at mAP0.75 under the LLVIP data set provided by the embodiment of the application;

[0055] Figure 10 is a comparison result diagram of the average accuracy of the GFT feature fusion network and the feature fusion network using the CFT module of the prior art at mAP0.95 under the LLVIP data set provided by the embodiment of the application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0057] The inventive concept of the present application is that, in view of the problems of insufficient precision and the like of the prior art multispectral information fusion-based small target detection method based on visible light and infrared images, the embodiment of the present application provides a multispectral feature fusion multi-scale target detection method based on a generalized attention mechanism, which is based on a YOLOv5 model framework, constructs a GFT (Generalized Fusion Transformer) feature fusion network in a neck network to fuse double-flow image features, uses proxy attention, skip attention structure and cross attention to reduce the overhead of the feature fusion network, accelerate the calculation of each layer of the Transformer attention and improve the feature fusion capability and detection precision of the double-flow network. The BiFPN module, VoVGSCSP module and GSConv module are used in the neck network to improve the ability of small target detection and balance the parameter quantity and detection precision. The RepConv is used before the detection head to improve the calculation efficiency and accuracy of the convolution module. The Wise IoU v3 loss function is used to improve the precision of the prediction box, so that the multispectral double-flow detection network comprising the backbone network, the neck network, the detection head and the output layer after training is more suitable for the unmanned aerial vehicle small target detection of double-flow images, and the robustness and detection precision of the unmanned aerial vehicle target detection are improved.

[0058] Figure 1 is a flowchart of the multispectral feature fusion multi-scale target detection method based on the generalized attention mechanism provided by the embodiment of the present application. As shown in Figure 1 , the embodiment provides a multispectral feature fusion multi-scale target detection method based on a generalized attention mechanism, comprising the following steps:

[0059] S1, a backbone network of a multispectral double-flow detection network is built, the backbone network performs parallel double-flow network feature extraction after reading visible light images and infrared images, a GFT feature fusion network based on a Transformer and introducing proxy attention, skip attention structure and cross attention is constructed to further feature extraction of original features extracted by the double-flow network at different stages and fusion of the original features through a residual structure to obtain fusion features at different stages.

[0060] In the embodiment, as shown in Figure 2 , in the visible light image feature extraction part of the backbone network, the visible light image F R is input into the first Focus module to extract the features F R1 from the spatial dimension to the channel dimension at intervals and obtain the feature map F R1 , and then the F R2 is input into the first Conv+C3 module to extract the feature map F R2After the feature map is extracted by the second Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of visible light features. The output of the second Conv+C3 module is then fused with the visible light feature output of the first GFT feature fusion network through a residual structure to obtain the feature map F. R3 , will F R3 After the feature map is extracted by the third Conv+C3 module, it is input into the second GFT feature fusion network for cross-fusion of visible light features and infrared features extracted in the same stage. The feature map F is obtained by fusing the output of the third Conv+C3 module and the visible light feature output of the second GFT feature fusion network through the residual structure. R4 , will F R4 After the feature map is extracted by the first Conv+SPP+C3 module, it is input into the third GFT feature fusion network for internal feature fusion of visible light features. The output of the first Conv+SPP+C3 module is then fused with the visible light feature output of the third GFT feature fusion network through a residual structure to obtain the feature map F. R5 .

[0061] like Figure 2 As shown, in the infrared image feature extraction part of the backbone network, the infrared image F... T Input the second Focus module to extract feature map F T1 , will F T1 Input the fourth Conv+C3 module to extract feature map F T2 , will F T2 After the feature map is extracted by the fifth Conv+C3 module, it is input into the first GFT feature fusion network for internal feature fusion of infrared features. The output of the fifth Conv+C3 module is then fused with the infrared feature output of the first GFT feature fusion network through a residual structure to obtain the feature map F. T3 , will F T3 After the feature map is extracted by the sixth Conv+C3 module, it is input into the second GFT feature fusion network for cross-fusion of infrared features and visible light features extracted in the same stage. The feature map F is obtained by fusing the output of the sixth Conv+C3 module and the infrared feature output of the second GFT feature fusion network through the residual structure. T4 , will F T4 After the feature map is extracted by the second Conv+SPP+C3 module, it is input into the third GFT feature fusion network for internal feature fusion of infrared features. The feature map F is obtained by fusing the output of the second Conv+SPP+C3 module with the infrared feature output of the third GFT feature fusion network through the residual structure. T5 .

[0062] Finally, F R3 and F T3 The fused feature P3 will be F R4and F T4 The fusion yields feature P4, which is then used to obtain F. R5 and F T5 Feature P5 is obtained through fusion.

[0063] In this embodiment, the structure of the GFT feature fusion network follows the mid-fusion characteristic. The first GFT feature fusion network (GFT-1) and the third GFT feature fusion network (GFT-3) in the first and last layers perform feature fusion within the two-stream network, and the second GFT feature fusion network (GFT-2) in the middle layer performs feature fusion between the two-stream networks. The feature fusion steps include:

[0064] (1) As Figure 3 As shown, in GFT-1 and GFT-3, the multi-head self-attention module in each layer of the Transformer is replaced with the agent attention module. Based on this, a skip attention structure is introduced into the agent attention module.

[0065] For the skip attention structure, the output of the (l-1)th layer proxy attention module in the Transformer is represented as: The Transformer at layer l is represented as a residual structure:

[0066]

[0067] Z′ l ←MLP(Z l )+Z l

[0068] Where SkipAt(·) is the skip attention structure, MLP(·) is the multilayer perceptron, and Z l Z′ is the attention output of the l-th layer. l The output of the Transformer at layer l is used to skip attention by applying the SkipAt(·) structure. Make The features are provided by the proxy attention module of the previous layer rather than calculated by the proxy attention module of the current layer. In the embodiment, the Transformer contains a total of 8 layers. In GFT-1 and GFT-3, the 4th and 7th layers of the 8 proxy attention layers are selected as the active layers of SkipAt(·).

[0069] The specific formula for the SkipAt(·) function layer is as follows:

[0070]

[0071] The first fully connected layer FC1(·) is used: Expand the channel dimension, where n represents the number of samples for the input or output features, and d represents the feature dimension. Then, a depthwise separable convolutional module DwC(·) is used: The relationships between tokens are obtained, and the results are flattened into a vector and input into the second fully connected layer FC2(·): To restore the channel dimension, the output of the current layer skipping attention is finally calculated using the ECA (Efficient Channel Attention) module ECA(·). The ECA module is implemented using ECA-Net. It aggregates features according to the channel dimension using global average pooling, then performs adaptive 1×1 convolution kernels according to the channel dimension, and finally inputs them into the GELU activation function.

[0072] (2) Figure 4 As shown, GFT-2 employs a Transformer-based cross-attention mechanism. This module facilitates the interaction of feature information between modules, enabling the two-stream network to learn each other's features. Within the Transformer, the query (Q), key (K), and value (V) are fed into a multi-head self-attention layer for different linear projection transformations. The transformed Q, K, and V then yield D values ​​of different dimensions. k and D v Where Q, K, and V are derived from the input sequence The vector matrix obtained by the linear transformation, where i and d represent the sequence length and feature size respectively, is given by the following formula:

[0073]

[0074] Among them, D k D represents the matrix dimensions of Q and K. v W represents the matrix dimension of V. Q W K and W V These are the parameter matrices after linear mapping.

[0075] The attention layer multiplies the transposes of matrices Q and K to obtain a coefficient, then multiplies it by a scaling factor, normalizes it using the Softmax function, and multiplies it by matrix V to obtain a weighted value. The attention function formula using matrices Q, K, and V is as follows:

[0076]

[0077] wherein the superscript T represents matrix transposition.

[0078] In the GFT-2, the query, key and value defining the visible light feature are Q1, K1 and V1 respectively, the query, key and value defining the infrared feature are Q2, K2 and V2 respectively, the attention output of the visible light part is obtained by querying K2 by Q1 and combining with V2 for attention calculation on the visible light feature part, the attention output of the infrared part is obtained by querying K1 by Q2 and combining with V1 for attention calculation on the infrared feature part, and the specific formula is as follows:

[0079] Z RGB ← GFT-2 (Q1, K2, V2)

[0080] Z IR ← GFT-2 (Q2, K1, V1)

[0081] wherein GFT-2(·) represents the GFT-2 module, Z RGB represents the attention output of the visible light part, and Z IR represents the attention output of the infrared part.

[0082] S2, a neck network of the multispectral double-flow detection network, the neck network replaces the Concat cascaded structure, the C3 module and the Conv module of the original YOLOv5 neck network by introducing the BiFPN module, the VoVGSCSP module and the GSConv module, takes the fused features of different stages as the input of the neck network and performs feature enhancement, and the enhanced features of different stages are respectively captured by the RepConv module to capture diversified features.

[0083] In the embodiment, as shown in Figure 5 , the BiFPN (Bi-directional Feature Pyramid Network) module is used to replace the Concat cascaded structure of the original YOLOv5 neck network, to realize the feature interaction from top to bottom and from bottom to top, and to learn the input features of different levels, and the final feature map output after weighted fusion of different level input features can combine the features of the current layer and the upper and lower layers, and the specific formula is as follows:

[0084]

[0085] wherein I i represents the i-th input feature, O represents the fused feature output, w i represents a learnable weight, j represents the index of the learnable weight, and ε represents a small parameter and is used to prevent the output from being unstable due to too small denominator.

[0086] Meanwhile, a Slim-Neck structure is used, which replaces the C3 module of the original YOLOv5 neck network with the VoVGSCSP module and replaces the Conv convolution module in P3 with the GSConv module, thereby improving the ability to detect small targets and balancing the number of parameters and detection accuracy.

[0087] like Figure 5 As shown, in the neck network, P3 undergoes feature extraction sequentially through the first BiFPN module, the first VoVGSCSP module, and the first RepConv module; P4 undergoes feature extraction sequentially through the second BiFPN module, the second VoVGSCSP module, the third BiFPN module, the third VoVGSCSP module, and the second RepConv module; and P5 undergoes feature extraction sequentially through the second GSConv module, the fourth BiFPN module, the fourth VoVGSCSP module, and the third RepConv module. Specifically, the first BiFPN module combines the output of the second VoVGSCSP module with the output after passing through the first GSConv and the first upsampling module; the second BiFPN module combines the output of the second GSConv with the output after passing through the second upsampling module; the third BiFPN module combines P4 with the output of the first VoVGSCSP module after passing through the first Conv module; and the fourth BiFPN module combines the output of the third VoVGSCSP module with the output after passing through the second Conv module.

[0088] (3) For the first RepConv (Re-parameterization Convolutional) module, the second RepConv module, and the third RepConv module designed in the last layer of the detection head, i.e. the neck network, the reparameterization technique is used to optimize the computational efficiency and accuracy of the convolutional layer. The core idea is to improve the model performance by using different convolutional structures in the training and inference phases.

[0089] During the training phase, each RepConv module uses multiple parallel convolutional layers to capture diverse features:

[0090] Y = X * W s +X*W p +X*W d

[0091] Where * denotes a convolution operation, X represents the input feature map, and W represents W. s Standard convolution kernel, W p W represents the pointwise convolution kernel. d This represents convolution kernels with different padding.

[0092] In the inference phase, the convolutional layers in each RepConv module are reparameterized into a single convolutional layer, reducing the amount of computation and speeding up the inference process:

[0093] W rep =W s +W p +W d

[0094] Y=X*W rep

[0095] where W rep represents the single convolutional layer obtained by reparameterization.

[0096] Through this reparameterization technique, the RepConv module only needs to perform one convolution operation in the inference phase, greatly improving the inference speed while maintaining the model's complex feature extraction ability in the training phase.

[0097] S3, build the detection head of the multispectral dual-stream detection network, the detection head receives the output of the RepConv module corresponding to different stages and processes to generate the detection result.

[0098] In the embodiment, the first detection module, the second detection module and the third detection module respectively accept the output from the first RepConv module, the second RepConv module and the third RepConv module.

[0099] S4, build the output layer of the multispectral dual-stream detection network, the output layer fuses the detection results corresponding to different stages to obtain the final target detection result.

[0100] In the embodiment, the final target detection result is obtained by weighted average calculation of the detection results of the first detection module, the second detection module and the third detection module.

[0101] S5, use the Wise IoU v3 loss function to train the multispectral dual-stream detection network.

[0102] In the embodiment, IoU is an index used to measure the overlap degree of anchor box and target box, which avoids the influence of target box size in the form of proportion, and uses to represent the Bounding Box Regression loss function:

[0103]

[0104] where S u =(wh+w gt h gt -W i H i) represents the overlapping area between the anchor box and the target box, W i and H i represent the length and width dimensions of the overlapping area, the anchor box x and y represent the horizontal and vertical coordinate positions of the anchor box, and w and h represent the length and width dimensions of the anchor box, the target box x gt and y gt represent the horizontal and vertical coordinate positions of the anchor box, w gt and h gt represent the length and width dimensions of the anchor box.

[0105] The Wise IoU v3 loss function is improved from the Wise IoU v1 loss function, which is represented as The specific formula is as follows:

[0106]

[0107]

[0108] wherein W g and H g are the length and width dimensions of the minimum bounding box, W g and H g part is separated from the calculation graph to avoid introducing gradients generated by the aspect ratio to affect the calculation of the covariance, and the superscript * represents this operation, According to the distance attention mechanism, the anchor box of ordinary quality is significantly amplified, then when the anchor box and the target box completely coincide, the dependence of the high-quality anchor box can be significantly reduced, and tends to depend on the distance between the center points.

[0109] According to this, the Wise IoU v3 loss function is represented as The specific formula is as follows:

[0110]

[0111] wherein β is the coefficient of the Dynamic non-monotonic Focusing Mechanism (Dynamic non-monotonic Focusing Mechanism), used to describe the proportion of outliers of the anchor box, is a dynamic parameter related to the average momentum, represents the loss function operated, and α and δ are two hyperparameters and take 1.7 and 2.7 respectively in the embodiment.

[0112] For the trained multispectral double-flow detection network, based on the YOLOv5s benchmark weight, the multispectral double-flow detection network containing the GFT network and the detection model containing the CFT network in the prior art are respectively trained and tested on the LLVIP dataset, and the overall accuracy comparison results are as shown in Figure 6 The recall rates are as shown in Figure 7 The recall rate represents the proportion of correct predictions in positive samples, the mAP0.5 is the average accuracy when the value of IoU is 50% as shown in Figure 8 The mAP0.75 is the average accuracy when the value of IoU is 75% as shown in Figure 9 The mAP0.95 is the average accuracy when the value of IoU is 95% as shown in Figure 10 These indicators are all the higher the better. It can be seen that the method provided in the embodiment of the present application has an improvement of about 2% in accuracy and an improvement of about 2% in recall rate, which can prove that the method has more excellent performance and performance.

[0113] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-spectral feature fusion multi-scale target detection method based on a general attention mechanism, characterized in that, The method comprises the following steps: A multi-spectral double-flow detection network including a backbone network, a neck network, a detection head and an output layer is built based on a YOLOv5 model framework; Wherein, the backbone network reads the visible light image and the infrared image to perform parallel double-flow network feature extraction, a GFT feature fusion network based on Transformer and introducing proxy attention, skip attention structure and cross attention is constructed to further extract features of the original features extracted by the double-flow network at different stages and fuse the original features through a residual structure to obtain fusion features at different stages, including: the visible light image F R Input the first Focus module to extract the feature map F R1 , F R1 Input the first Conv+C3 module to extract the feature map F R2 , F R2 After inputting the second Conv+C3 module to extract the feature map, input the first GFT feature fusion network to perform internal feature fusion of the visible light feature, and fuse the output of the second Conv+C3 module with the visible light feature output of the first GFT feature fusion network through a residual structure to obtain the feature map F R3 , F R3 After inputting the third Conv+C3 module to extract the feature map, input the second GFT feature fusion network to perform cross fusion of the visible light feature and the infrared feature extracted at the same stage, and fuse the output of the third Conv+C3 module with the visible light feature output of the second GFT feature fusion network through a residual structure to obtain the feature map F R4 , F R4 After inputting the first Conv+SPP+C3 module to extract the feature map, input the third GFT feature fusion network to perform internal feature fusion of the visible light feature, and fuse the output of the first Conv+SPP+C3 module with the visible light feature output of the third GFT feature fusion network through a residual structure to obtain the feature map F R5 ; the infrared image F T Input the second Focus module to extract the feature map F T1 , F T1 Input the fourth Conv+C3 module to extract the feature map F T2 , F T2 After inputting the fifth Conv+C3 module to extract the feature map, input the first GFT feature fusion network to perform internal feature fusion of the infrared feature, and fuse the output of the fifth Conv+C3 module with the infrared feature output of the first GFT feature fusion network to obtain the feature map F T3 , F T3 After inputting the sixth Conv+C3 module to extract the feature map, input the second GFT feature fusion network to perform cross fusion of the infrared feature and the visible light feature extracted at the same stage, and fuse the output of the sixth Conv+C3 module with the infrared feature output of the second GFT feature fusion network through a residual structure to obtain the feature map F T4 , F T4 The input second Conv+SPP+C3 module extracts a feature map, and then the third GFT feature fusion network is input to perform internal feature fusion of the infrared feature and fuse the output of the second Conv+SPP+C3 module and the infrared feature output of the third GFT feature fusion network through a residual structure to obtain a feature map F T5 ; In the first GFT feature fusion network or the third GFT feature fusion network, the multi-head self-attention module in each layer of the Transformer is replaced by a proxy attention module, and on this basis, a skip attention structure is introduced in the proxy attention module; for the skip attention structure, the output of the proxy attention module in the (l-1)th layer in the Transformer is represented as The (l)th layer of the Transformer is represented as a residual structure: Z l ′ ←MLP(Z l )+Z l Where SkipAt(·) is the skip attention structure, MLP(·) is the multilayer perceptron, and Z l Z is the attention output of the l-th layer. l ′ The output of the Transformer at layer l is used to skip attention by applying the SkipAt(·) structure. Make The features are provided by the proxy attention module of the previous layer, rather than being calculated by the proxy attention module of the current layer. The specific formula for the SkipAt(·) layer is as follows: wherein the first fully connected layer FC1(·) is used to expand the channel dimension, and then the deep separable convolution module DwC(·) is used to obtain the relationship between Tokens, the result is flattened into a vector and input into the second fully connected layer FC2(·) to restore the channel dimension, and finally the output of the current layer skip attention is calculated through the ECA module ECA(·) A BiFPN module, a VoVGSCSP module and a GSConv module are introduced to replace the Concat cascade structure, a C3 module and a Conv module of the original YOLOv5 neck network, the fusion features of different stages are taken as the input of the neck network and are subjected to feature enhancement, and the enhanced features of different stages are subjected to a RepConv module to capture diversified features; The detection head receives the output of the RepConv module corresponding to the features of different stages and processes the output to generate a detection result; The output layer fuses the detection results corresponding to the features of different stages to obtain a final target detection result.

2. The multi-spectral feature fusion multi-scale object detection method based on the pan attention mechanism according to claim 1, characterized in that, The ECA module is implemented by an ECA-Net, aggregates features by using a global average pooling according to the channel dimension size, performs an adaptive 1x1 convolution kernel according to the channel dimension, and inputs a final GELU activation function.

3. The multi-spectral feature fusion multi-scale object detection method based on the pan attention mechanism of claim 1, characterized in that, In the second GFT feature fusion network, comprising: A cross-attention mechanism based on a Transformer is adopted, the query, key and value of the visible light feature are defined as Q1, K1 and V1 respectively, the query, key and value of the infrared feature are defined as Q2, K2 and V2 respectively, the attention output of the visible light part is obtained by querying K2 through Q1 and combining with V2, and the attention output of the infrared part is obtained by querying K1 through Q2 and combining with V1.

4. The multi-spectral feature fusion multi-scale object detection method based on the pan attention mechanism of claim 1, characterized in that, In the neck network, comprising: F R3 and F T3 the fused feature P3, F R4 and F T4 the fused feature P4, and F R5 and F T5 the fused feature P5 are input into the neck network, P3 is sequentially subjected to feature extraction by a first BiFPN module, a first VoVGSCSP module and a first RepConv module, P4 is sequentially subjected to feature extraction by a second BiFPN module, a second VoVGSCSP module, a third BiFPN module, a third VoVGSCSP module and a second RepConv module, and P5 is sequentially subjected to feature extraction by a second GSConv module, a fourth BiFPN module, a fourth VoVGSCSP module and a third RepConv module; The first BiFPN module is further combined with the output of the second VoVGSCSP module after the first GSConv and the first upsampling module, the second BiFPN module is further combined with the output of the second GSConv after the second upsampling module, the third BiFPN module is further combined with P4 and the output of the first VoVGSCSP module after the first Conv module, and the fourth BiFPN module is further combined with the output of the third VoVGSCSP module after the second Conv module.

5. The multi-spectral feature fusion multi-scale object detection method based on the pan attention mechanism according to claim 4, characterized in that, In the detection head, comprising: The first detection module, the second detection module and the third detection module respectively receive the output from the first RepConv module, the second RepConv module and the third RepConv module.

6. The multi-spectral feature fusion multi-scale object detection method based on the pan-attention mechanism according to claim 5, characterized in that, In the output layer, comprising: The final target detection result is obtained by weighted average calculation on the detection results of the first detection module, the second detection module and the third detection module.

7. The multi-spectral feature fusion multi-scale object detection method based on the pan-attention mechanism of claim 4, characterized in that, In the training stage, each RepConv module uses multiple parallel convolution layers to capture diversified features: Y = X * W s + X * W p + X * W d wherein * represents a convolution operation, X represents an input feature map, W s represents a standard convolution kernel, W p represents a point-wise convolution kernel, W d represents a convolution kernel with different padding; In the inference stage, the convolution layers in each RepConv module are re-parameterized into a single convolution layer to accelerate the extraction of diversified features: W rep = W s + W p + W d Y = X * W rep where W rep represents a single volume layer obtained by reparameterization.

8. The multi-spectral feature fusion multi-scale object detection method based on the pan attention mechanism according to any one of claims 1-7, characterized in that, The multi-spectral double-flow detection network is trained by using a Wise IoU v3 loss function.

Images