Multispectral target detection method based on adaptive attention mechanism

Abstract

The invention provides a multispectral target detection method based on a self-adaptive attention mechanism, and relates to the technical field of image processing, and the method comprises the steps: collecting multispectral image data sets of different scenes and different types, and dividing the multispectral image data sets into a training set and a test set; constructing a multispectral target detection model based on an adaptive attention mechanism, and training and testing the multispectral target detection model by using the training set and the test set; and performing multispectral target detection by using the trained and tested multispectral target detection model. According to the method, the problems that in the prior art, in the field of multispectral fusion target detection, features among modes are not aligned, complementary features among the modes cannot be adaptively weighed, and fusion feature information is redundant are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multispectral target detection method based on an adaptive attention mechanism. Background Art

[0002] With the continuous development of computer vision and deep learning, the acquisition and processing of multispectral images has become an important research direction in the field of target detection. Multispectral images can provide richer image information than single spectral images by acquiring data from different spectral bands, such as visible light, infrared, and ultraviolet bands. Among them, visible light images are the spectral range that the human eye can perceive, with a wavelength of approximately between 400nm and 700nm. Its images can capture rich color and texture information, but are easily affected by harsh environments, resulting in poor imaging conditions. Infrared images are outside the wavelength range of visible light and are usually divided into different bands such as near infrared (0.7–1.5μm), mid-infrared (1.5–5μm), and far infrared (8–15μm). Infrared imaging technology achieves insensitivity to environmental factors such as lighting and smoke by capturing the thermal radiation of objects, so that targets can still be stably identified under these harsh conditions, but infrared images will lose the detailed texture information of the target. By combining the advantages of visible light and infrared images, multispectral fusion target detection has broad application prospects in fields such as remote sensing monitoring, military reconnaissance, security monitoring, and environmental monitoring. However, multispectral fusion target detection has the following difficulties: feature misalignment between modalities, inability to adaptively weigh complementary characteristics between modalities, and redundant fusion feature information.

[0003] In order to solve the problem of feature misalignment between modalities, the prior art uses CNN convolutional neural network to realize cross-modal interaction mechanism to effectively combine the features of visible light and infrared images, so that target detection can more accurately process heterogeneous information of different modalities; in addition, the prior art proposes an innovative bilinear convolutional neural network (BCNN) model to extract change information from multispectral images, especially focusing on the change detection task in remote sensing image analysis; although the above methods have made certain contributions to the multispectral fusion target detection task, due to the lack of modeling ability of convolutional neural network for long-range relations, they cannot fully solve the problem of feature misalignment between modalities. The prior art proposes a cross-modal integration network based on Transformer, which effectively fuses the information of RGB and infrared modalities through collaborative spatial attention mechanism and feature interaction strategy, and enhances the model's recognition ability of salient targets. However, this method only utilizes the long-range relationship modeling ability of Transformer, and ignores the interference of noise introduced by incomplete alignment between infrared image and visible light image on the original image, resulting in poor performance of the model in fusing multispectral images.

[0004] In order to solve the problem of being unable to adaptively weigh the complementary characteristics between modalities, the existing technology introduces a multi-scale adaptive fusion module to solve the problem of adaptively adjusting the contribution ratio of feature maps at different scales in the feature fusion task. However, this method does not take into account the problem of adaptive weights of different modalities in different scenarios; the existing technology proposes a feature fusion pyramid network, which uses a Transformer-like strategy to perform multimodal feature fusion, thereby suppressing features that are not conducive to fusion during the fusion process. Due to the quadratic time complexity problem of the Transformer architecture, introducing this architecture into the model will make the model training time-consuming and labor-intensive.

[0005] In order to solve the problem of redundant fused feature information, the existing technologies almost always directly perform additive fusion when fusing and enhancing the fused feature map with the feature maps of each modality, and fusing and outputting the feature maps of different modalities to the detection head. For example, the existing technologies directly perform additive fusion enhancement on the fused feature map and the feature map of the backbone network, and connect them along the channel dimension before inputting them into the detection head to perform target detection. This method does not take into account the information redundancy problem caused by direct additive fusion or cascading by channel dimension, resulting in poor performance of the final model. In order to effectively utilize the feature information of different modalities, a convolutional network can be used to further extract features to suppress redundant information. However, due to the inherent defects of the local receptive field of the CNN network and the lack of the ability to model long-range feature relationships, it cannot suppress redundant information well. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multispectral target detection method based on an adaptive attention mechanism, which solves the problems of misalignment of inter-modal features, inability to adaptively weigh the complementary characteristics between modalities, and redundancy of fusion feature information in the field of multispectral fusion target detection in the prior art.

[0007] In order to achieve the above objectives, the technical solution adopted by the present invention is: a multispectral target detection method based on an adaptive attention mechanism, comprising the following steps:

[0008] S1. Collect multispectral image datasets of different scenes and categories and divide them into training sets and test sets;

[0009] S2. Build a multispectral target detection model based on adaptive attention mechanism, and use training set and test set to train and test the multispectral target detection model;

[0010] S3. Perform multispectral target detection using the trained and tested multispectral target detection model.

[0011] The beneficial effects of the present invention are as follows: the present invention discloses a multispectral target detection method based on an adaptive attention mechanism, which aims to solve the problems of misalignment of inter-modal features, inability to adaptively weigh inter-modal complementary characteristics, and redundant fusion feature information in existing multispectral fusion target detection technologies. The present invention effectively realizes the feature fusion of visible light and infrared modalities in multispectral images by combining the multi-resolution analysis capability of wavelet transform and the inter-modal feature fusion capability of the adaptive cross-attention mechanism, thereby improving the accuracy and robustness of target detection.

[0012] Furthermore, the loss function of the multispectral target detection model is expressed as follows:

[0013] loss=loss cls +loss obj +loss bbox

[0014]

[0015] loss cls =loss obj =H(p,q)

[0016] =-∑ x (p(x)logq(x)+(1-p(x)log(1-q(x))))

[0017]

[0018] Among them, loss represents the loss function of the multispectral target detection model, loss cls Represents classification loss, which is responsible for distinguishing objects of different categories. obj Represents the target loss, which is used to measure the probability of an object existing in a grid cell. bbox represents the bounding box regression loss, which is composed of lossCIoU, an IoU loss function that considers the center point distance and aspect ratio difference of the box, and is used to accurately locate the object. lossCIoU represents a loss function that considers not only the IoU between the predicted box and the true box, but also the center point distance between the predicted box and the true box. IoU represents the intersection-over-union ratio between the predicted box and the true box, ρ represents the Euclidean distance between the two centers, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, b and b gtdenote the center points of the predicted box and the true box respectively, α denotes the weight function, v denotes the similarity used to measure the aspect ratio, H(p,q) denotes the binary cross entropy loss function, x denotes a sample in the sample space, p(x) denotes the probability that the multispectral target detection model predicts that the sample x belongs to the positive class, q(x) denotes the probability that the sample x actually belongs to the positive class, arctan denotes the inverse tangent function, w gt and h gt They represent the width and height of the real box, w and h represent the width and height of the predicted box, respectively.

[0019] The beneficial effects of the above further scheme are: by combining classification loss, target loss and bounding box regression loss, the technical effects of improving detection accuracy, enhancing robustness, optimizing feature fusion, reducing computational complexity, improving the detection ability of small objects and low-contrast objects, adaptive inter-modal complementary characteristics and improving generalization ability are achieved. The multispectral target detection model predicts the object bounding box more accurately through the CIoU loss function, while considering the overlapping area, center point distance and aspect ratio, thereby improving robustness in complex environments and enhancing the generalization ability of the model.

[0020] Furthermore, step S3 includes the following steps:

[0021] S301, input the collected multispectral image into the dual-branch backbone network for information extraction, and obtain feature maps of different scales by downsampling, wherein the dual-branch architecture is based on the Yolov8 architecture, transforming the original single-branch architecture into a dual-branch architecture, the dual branches are infrared and visible light branches, and each branch in the dual-branch backbone network includes a base layer stem and a four-layer feature extraction layer stage i , the input image F0 of each branch is passed through the base layer stem to obtain the feature map F1, and then passes through four layers of feature extraction layers to obtain the feature maps F2, F3, F4 and F5;

[0022] S302, inputting the feature maps of the same scale of the infrared and visible light branches into the adaptive cross-attention feature module to fuse different modal information;

[0023] S303, using the wavelet convolution feature fusion enhancement module, the fused feature map is fused and enhanced with the two feature maps of the dual-branch backbone network respectively;

[0024] S304, input the infrared and visible light feature maps of the last three scales of the backbone network into the feature fusion enhancement module for fusion, and input the fused feature maps into the Yolov8 neck and head network for target detection, thereby completing the detection of multi-spectral targets.

[0025] The beneficial effects of the above further scheme are: to address the problem of feature misalignment between modalities, the present invention designs a feature fusion mechanism based on the cross-attention mechanism, which effectively enhances the interaction between multispectral features and realizes information complementarity; to address the problem of being unable to adaptively weigh the complementary characteristics between modalities, the present invention designs a modality adaptation module, so that the network can adaptively focus on the dominant modality in different scenarios and effectively fuse the effective information between different modalities for target detection; to address the problem of redundant fused feature information, the present invention designs a wavelet convolution fusion module, so that the network can efficiently perform feature fusion enhancement while also removing redundant information between different modalities.

[0026] Furthermore, the expression of the feature map F1 is as follows:

[0027] F1=ConvModule(F0)

[0028] ConvModule(·)=Silu(BatcheNorm(Conv2d(·)))

[0029] Among them, ConvModule(·) represents the module set used for convolution operation, Silu(·) represents the activation function, BatcheNorm(·) represents batch normalization, and Conv2d(·) represents batch normalization, which represents the convolution operation with a kernel size of 3 and a stride of 2.

[0030] The four-layer feature extraction stage i Both include a convolution module and a cross-stage local network module CSPDarknet, where the feature map F1 to the feature map F i+1 The expression is as follows:

[0031] F i (1) ,F i (2) =Split(F i )

[0032]

[0033] CSPDarkNet=Conv2d(Concat(F i (1) ,F res ))

[0034]

[0035] F i+1 =CSPDarkNet(ConvModule(F i )),i=1,2,3

[0036] F i+1 =SPPF(CSPDarkNet(ConvModule(F i ))),i=4

[0037] Among them, F i (1) and F i (2) Represent the feature map F i The first and second parts after segmentation, Split(·) means splitting the input feature map along the channel dimension, F res Represents the second part F of the segmented feature map i (2) implement The output after the operation, Indicates that the input of the last residual block in the cascade of k residual modules is used as the overall output, CSPDarkNet represents the Darknet structure in the cross-stage local network, Conv2d(·) represents the two-dimensional convolution function, Concat(·) represents the concatenation of feature maps along the channel dimension, and F i+1 represents the next layer of feature maps, ConvModule(·) represents the convolution operation, SPPF(·) represents the fast spatial pyramid pooling operation, It means that k maximum pooling operations are performed on the input feature map, and the pooling results are concatenated along the channel dimension.

[0038] The beneficial effects of the above further scheme are: the above steps achieve technical effects such as enhancing feature expression ability, improving feature extraction efficiency, optimizing feature fusion, improving model flexibility, enhancing robustness, improving multi-scale characteristics, and improving computational efficiency by combining multiple operations. The overall design reduces computing resource consumption and improves computing efficiency while maintaining high performance, which together promotes the model to achieve higher accuracy and robustness in multispectral target detection tasks.

[0039] Furthermore, the step S302 includes the following steps:

[0040] S3021, inputting the feature maps of the same scale of the two branches of infrared and visible light into the modality adaptation module, and adjusting the weights of the respective modalities;

[0041] S3022, inputting the weighted feature map into a spatial feature dimensionality reduction module to perform dimensionality reduction processing to obtain a visible light and infrared branch feature map after dimensionality reduction;

[0042] S3023. Input the reduced-dimensional visible light and infrared branch feature maps into the adaptive cross-attention feature module for feature fusion to complete the fusion of different modal information.

[0043] The beneficial effect of the above further scheme is: through the modality adaptation module, the proportion of visible light and infrared images in the fusion process is dynamically adjusted to achieve more robust and efficient multimodal fusion. Furthermore, through the spatial feature dimensionality reduction module, the size of the feature map is reduced while retaining key information, thereby effectively reducing the computational burden of the cross-attention module. Finally, through the adaptive cross-attention feature module, while maximally retaining the feature information of the original modality, it can also effectively integrate the information of the additional modality, thereby achieving a more accurate and comprehensive feature representation.

[0044] Furthermore, the step S3021 includes the following steps:

[0045] S30211. The feature maps F of the same scale of the infrared and visible light branches are T and F R , concatenate according to the channel dimension, and perform average pooling to calculate the weights of each modality;

[0046] S30212. The modal weight is further split into two branches according to the channel dimension, and multiplied with the input feature maps of the original visible light branch and the infrared branch respectively, and the feature maps of the visible light and infrared branches after weight adjustment are output to complete the adjustment of the modal weight:

[0047] F R '=W R *F R

[0048] F T '=W T *F T

[0049] W R ,W T =Split(AvgPool(Concat(F R ,F R )))

[0050] Among them, F R ' and F T 'represent the feature maps of the visible light and infrared branches after adjusting the weights, W R and W T They represent the weights of their respective branches, Split(·) means splitting the input feature map along the channel dimension, AvgPool(·) means the average pooling operation, and Concat(·) means concatenating the feature maps along the channel dimension.

[0051] The beneficial effect of the above further scheme is: by calculating the weights of different feature maps in different scenarios, the proportion of them in subsequent steps can be adaptively adjusted to improve the final fusion effect, thereby achieving more robust and efficient multimodal fusion.

[0052] Furthermore, the expression of the feature map after dimensionality reduction in step S3022 is as follows:

[0053] F d =F max +F avg

[0054] F max =MaxPool(F')

[0055] F avg =AvgPool(F')

[0056] Among them, F d Indicates that F max and F avg The output feature map obtained by adding element by element, F max and F avg They represent the output feature maps after the maximum pooling and average pooling operations on the feature maps, respectively. MaxPool(·) represents the maximum pooling operation, AvgPool(·) represents the average pooling operation, and F' represents any feature map of visible light and infrared.

[0057] The beneficial effect of the above further scheme is: reducing the spatial dimension of the feature map through the operations of maximum pooling and average pooling respectively, while retaining the key information to the greatest extent and reducing the size of the feature map, thereby effectively reducing the computational burden of the cross-attention module.

[0058] Furthermore, the step S3023 includes the following steps:

[0059] S30231. Normalize the visible light and infrared branch feature maps after dimensionality reduction respectively to obtain a normalized visible light branch feature map. And infrared branch feature map

[0060] S30232, visible light branch feature map After two linear layers, the matrix K is generated R and the matrix V R , the infrared branch feature map After a linear layer, the matrix Q is generated T ;

[0061] S30233, according to the matrix Q T and the matrix K R, calculate the similarity matrix of the two modes, and calculate the corresponding similarity score M through the SoftMax function R ;

[0062] S30234, according to the similarity score M R and the matrix V R , calculate the output feature map that incorporates additional modal information and

[0063]

[0064] Among them, D K represents the scaling factor, W V , W K and W Q They are used to generate the matrix V R , K R and Q T The mapping matrix, SoftMax(·) represents the activation function, V T M represents the value matrix obtained by infrared modal mapping. T Represents the similarity score matrix of infrared modalities;

[0065] S30235, feature map Pass through the normalization layer again and compare it with the visible light branch feature map Perform residual connection;

[0066] S30236. Input the residual connection result to the fully connected layer for feature extraction to obtain an output feature map that incorporates additional modal information. and

[0067]

[0068] Among them, Conv() represents the convolution operation, Reshape() represents the upsampling operation, which is used to map the feature map back to the resolution of the original feature map, α, β, γ and λ all represent learnable parameters, FFN() represents the feedforward neural network, which is used to further extract features, and LayerNorm() represents the layer normalization operation. and They represent the output feature maps of visible light and infrared images after the feedforward neural network, respectively. R ' and F T 'represent the feature maps of visible light and infrared branches after adjusting the weights;

[0069] S30237, feature map and Splicing is performed along the channel dimension and fused into a unified feature representation to complete the fusion of different modal information:

[0070]

[0071] Among them, F fused Represents the fused feature map.

[0072] The beneficial effect of the above further scheme is: using the cross-attention mechanism to aggregate the feature maps of different modalities to learn the difference and global feature information between the two modalities, which can effectively deal with the pixel misalignment problem of multispectral images. At the same time, multiple residual connection operations are added to the above scheme. This design not only ensures that the feature information of the original modality can be retained to the maximum extent during the fusion process, but also effectively integrates the information of the additional modalities, thereby achieving a more accurate and comprehensive feature representation.

[0073] Furthermore, step S303 includes the following steps:

[0074] S3031, cascading the fused feature map with the two feature maps of the backbone network of the infrared and visible light branches in the channel dimension;

[0075] S3032. Use wavelet convolution to fuse the cascaded feature maps to complete the fusion enhancement processing of the feature maps.

[0076] The beneficial effect of the above further scheme is: the fused feature map is fused and enhanced with the visible light and infrared feature maps of the backbone network through the wavelet convolution fusion module, and the wavelet convolution operation has the property of a large receptive field, which greatly removes the redundant information of the feature map and ultimately enhances the fusion performance of the feature map.

[0077] Furthermore, the step S304 is specifically as follows:

[0078] The feature maps F3, F4 and F5 of the infrared and visible light branches in the backbone network are taken as input and fused;

[0079] The fused feature map is input into the Yolov8 neck and head network for target detection to complete the detection of multispectral targets:

[0080]

[0081] in, represents the prediction result of the i-th layer, Head represents the decoupled detection head of Yolov8, It represents the output feature map of the dual-branch backbone network of the i-th layer that fuses the visible light and infrared feature maps. PAFPN represents the feature fusion pyramid network structure of Yolov8. WTConv represents the fusion enhancement processing.

[0082] The beneficial effects of the above further scheme are: effective fusion of multi-spectral information is achieved, the accuracy of target detection is enhanced, and the robustness of the model in low-light or complex environments is improved. The PAFPN feature fusion pyramid network structure optimizes feature representation and improves multi-scale adaptability, while the WTConv fusion enhancement processing reduces the amount of calculation while maintaining high accuracy and improves computational efficiency. In addition, this scheme enhances the generalization ability of the model, making it better adaptable to different application scenarios, and the decoupled detection head design of Yolov8 helps to speed up the detection speed and meet the needs of real-time applications, thereby significantly improving accuracy, robustness, computational efficiency and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 The present invention is a flow chart of the method.

[0084] Figure 2 It is a structural diagram of the model framework of the present invention.

[0085] Figure 3 This is a structural diagram of the adaptive cross-attention feature module (MAFB) of the present invention.

[0086] Figure 4 This is a structural diagram of the wavelet convolution fusion module of the present invention. DETAILED DESCRIPTION

[0087] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0088] Example

[0089] The overall framework structure of the present invention is as follows Figure 2 As shown in the figure, a multi-spectral fusion target detection model based on wavelet transform and adaptive cross-attention mechanism is shown. First, the backbone network extracts the features of RGB and infrared images respectively. Here, the Yolov8 architecture is followed and the cross-stage local network CSPDarkNet is used for feature extraction. Subsequently, the intermediate features of the same scale extracted from the two modalities are input into the modality adaptive fusion network proposed in the present invention to aggregate information. Secondly, the fusion of the two modal feature maps is used to extract the features of the infrared image. Respectively with the RGB and infrared feature maps in the backbone network, that is, Finally, in order to detect targets of different scales, the present invention fuses the last three layers of feature maps of the two modal backbone networks through a wavelet convolution module and outputs them into the neck and the detection head for final target detection; the neck network used in the present invention is PAFPN, and the head uses the decoupling head of Yolov8.

[0090] like Figure 1 As shown, the present invention provides a multispectral target detection method based on an adaptive attention mechanism, and its implementation method is as follows:

[0091] S1. Collect multispectral image datasets of different scenes and categories and divide them into training sets and test sets;

[0092] In this embodiment, multispectral image datasets of different scenes and categories are collected and divided into training sets and test sets. In this embodiment, a total of four datasets are collected, namely FLIR, LLVIP, KAIST and M3FD. Specifically, the FLIR dataset images cover a diverse set of scenes from day to night, including 3 categories, 5142 pairs of aligned RGB and infrared images, of which 4129 pairs are used for training and 1013 pairs are used for testing; LLVIP is a dataset focusing on multispectral pedestrian detection in low-light conditions, which contains 15488 aligned RGB-IR image pairs, of which 12025 images are used for training and 3463 images are used for testing; KAIST is a pedestrian annotation dataset, whose training set contains 8892 pairs of images, and the test set contains 2252 pairs of infrared-visible light pairs, and 1455 pairs of images in the test set are taken during the day, and the remaining 797 pairs of images are taken at night; The M3FD dataset contains 6 different categories, a total of 4200 pairs of RGB-IR image pairs, of which 3360 pairs are used for training and 840 pairs are used for testing.

[0093] S2. Build a multispectral target detection model based on adaptive attention mechanism, and use training set and test set to train and test the multispectral target detection model;

[0094] In this embodiment, a multispectral target detection model based on an adaptive attention mechanism is configured to preprocess the collected data, including random enhancement, random flipping, etc. The preprocessed data is input into the multispectral target detection model, and the training is performed using the training set. During the training process, a verification is performed after each round, and the optimal multispectral target detection model is saved. The optimal multispectral target detection model is deployed in the target detection system to perform target detection on the multispectral image.

[0095] S3. Use the trained and tested multispectral target detection model to perform multispectral target detection. The implementation method is as follows:

[0096] S301, input the collected multispectral image into the dual-branch backbone network for information extraction, and obtain feature maps of different scales by downsampling, wherein the dual-branch architecture is based on the Yolov8 architecture, transforming the original single-branch architecture into a dual-branch architecture, the dual branches are infrared and visible light branches, and each branch in the dual-branch backbone network includes a base layer stem and a four-layer feature extraction layer stage i The input image F0 of each branch passes through the base layer stem to obtain the feature map F1, and passes through four feature extraction layers to obtain the feature map F2, feature map F3, feature map F4 and feature map F5.

[0097] In this embodiment, based on the Yolov8 architecture, the original single-branch architecture is transformed into a dual-branch architecture to adapt to the dual-branch input of visible light and infrared images, and the multispectral image is input into the dual-branch backbone network for information extraction, and feature maps of different scales are obtained by downsampling. The dual-branch Yolov8 backbone network is constructed, and each branch includes a base layer stem and a four-layer feature extraction layer stage i , i=1,2,3,4, the input image F0 of each branch first passes through the base layer stem to obtain the feature map F1, and then passes through four layers of feature extraction layers in sequence to obtain the feature map F2, feature map F3, feature map F4 and feature map F5, wherein the feature map F2, feature map F3, feature map F4 will be used as the input of step S302 and step S303, and the feature map F2, feature map F3, feature map F4 and feature map F5 will be used as the input of step S304. The base layer stem is composed of a convolution module, and the formula for obtaining the output feature map F1 from the input image F0 to the base layer stem is as follows:

[0098] F1=ConvModule(F0)

[0099] ConvModule(·)=Silu(BatcheNorm(Conv2d(·)))

[0100] Among them, ConvModule(·) represents the module set used for convolution operation, Silu(·) represents the activation function, BatcheNorm(·) represents batch normalization, and Conv2d(·) represents batch normalization, which represents the convolution operation with a kernel size of 3 and a stride of 2.

[0101] In this embodiment, the four feature extraction stages iThe foundation consists of a convolution module and a CSPDarknet module, in which a fast spatial pyramid pooling module SPPF is attached to the last feature extraction layer, stage 4. i+1 The expression is as follows:

[0102] F i (1) ,F i (2) =Split(F i )

[0103]

[0104] CSPDarkNet=Conv2d(Concat(F i (1) ,F res ))

[0105]

[0106] F i+1 =CSPDarkNet(ConvModule(F i )),i=1,2,3

[0107] F i+1 =SPPF(CSPDarkNet(ConvModule(F i ))),i=4

[0108] Among them, F i (1) and F i (2) Represent the feature map F i The first and second parts after segmentation, Split(·) means splitting the input feature map along the channel dimension, F res Represents the second part F of the segmented feature map i (2) implement The output after the operation, Indicates that the input of the last residual block in the cascade of k residual modules is used as the overall output, CSPDarkNet represents the Darknet structure in the cross-stage local network, Conv2d(·) represents the two-dimensional convolution function, Concat(·) represents the concatenation of feature maps along the channel dimension, and F i+1 represents the next layer of feature maps, ConvModule(·) represents the convolution operation, SPPF(·) represents the fast spatial pyramid pooling operation, It means that k maximum pooling operations are performed on the input feature map, and the pooling results are spliced ​​along the channel dimension. Darknet is an open source network neural architecture.

[0109] S302: Input the feature maps of the same scale of the infrared and visible light branches into the adaptive cross-attention feature module to fuse different modal information. The implementation method is as follows:

[0110] S3021, input the feature maps of the same scale of the two branches of infrared and visible light into the modality adaptation module, and adjust the weights of each modality. The implementation method is as follows:

[0111] S30211. The feature maps F of the same scale of the infrared and visible light branches are T and F R , concatenate according to the channel dimension, and perform average pooling to calculate the weights of each modality;

[0112] S30212, splitting the modal weight into two branches according to the channel dimension, and performing multiplication operations with the input feature maps of the original visible light branch and the infrared branch respectively, outputting the feature maps of the visible light and infrared branches after adjusting the weights, and completing the adjustment of the modal weights;

[0113] In this embodiment, the feature maps F of the visible light and infrared image branches are T and F R , first concatenate according to the channel dimension and then perform average pooling to calculate the weights of each modality, and then split the weights into two branches according to the channel dimension and multiply them with the input feature maps of each branch to obtain the final output feature map. The formula for this process is as follows:

[0114] F R '=W R *F R

[0115] F T '=W T *F T

[0116] W R ,W T =Split(AvgPool(Concat(F R ,F R )))

[0117] Among them, F R ' and F T 'represent the feature maps of the visible light and infrared branches after adjusting the weights, W R and W TThey represent the weights of their respective branches, Split(·) means splitting the input feature map along the channel dimension, AvgPool(·) means the average pooling operation, and Concat(·) means concatenating the feature maps along the channel dimension.

[0118] S3022, inputting the weighted feature map into a spatial feature dimensionality reduction module to perform dimensionality reduction processing to obtain a visible light and infrared branch feature map after dimensionality reduction;

[0119] In this embodiment, the visible light and infrared branch feature maps F after weight adjustment are R ' and F T ' Input the spatial feature dimensionality reduction module respectively, where the maximum pooling and average pooling operations are performed on any feature map F', and then the results of the two operations are added to obtain the reduced feature map F d The process is expressed in the following formula:

[0120] F d =F max +F avg

[0121] F max =MaxPool(F')

[0122] F avg =AvgPool(F')

[0123] Among them, F d Indicates that F max and F avg The output feature map obtained by adding element by element, F max and F avg They represent the output feature maps after the maximum pooling and average pooling operations on the feature maps, respectively. MaxPool(·) represents the maximum pooling operation, AvgPool(·) represents the average pooling operation, and F' represents any feature map of visible light and infrared.

[0124] S3023, inputting the reduced dimension visible light and infrared branch feature maps into the adaptive cross-attention feature module for feature fusion to complete the fusion of different modal information, and the implementation method is as follows:

[0125] S30231. Normalize the visible light and infrared branch feature maps after dimensionality reduction respectively to obtain a normalized visible light branch feature map. And infrared branch feature map

[0126] S30232, visible light branch feature map After two linear layers, the matrix K is generated R and the matrix V R, the infrared branch feature map After a linear layer, the matrix Q is generated T ;

[0127] S30233, according to the matrix Q T and the matrix K R , calculate the similarity matrix of the two modes, and calculate the corresponding similarity score M through the SoftMax function R ;

[0128] S30234, according to the similarity score M R and the matrix V R , calculate the output feature map that incorporates additional modal information and

[0129] S30235, feature map Pass through the normalization layer again and compare it with the visible light branch feature map Perform residual connection;

[0130] S30236. Input the residual connection result to the fully connected layer for feature extraction to obtain an output feature map that incorporates additional modal information. and

[0131] S30237, feature map and The information is spliced ​​along the channel dimension and fused into a unified feature representation to complete the fusion of different modal information.

[0132] In this embodiment, the above-mentioned visible light and infrared branch feature maps after dimensionality reduction are and The two modal feature maps are input into two cross-attention feature fusion modules at the same time to obtain the fusion feature maps dominated by each modality. For this process, the following will be described when visible light is the dominant modality. First, the two modal feature maps are input into the LayerNorm layer for normalization to reduce the differences between different modalities and image noise. This process can be expressed by the following formula:

[0133]

[0134]

[0135] Among them, k∈[1,c] represents the kth channel of the current feature map, c represents the number of feature map channels, H represents the feature map height, W represents the feature map width, i represents the i-th row of the current channel of the feature map, and j represents the j-th column of the current channel of the feature map. Represents the feature map F ij The kth channel ofij represents the current feature map, μ represents the mean of the current channel, σ represents the variance of the current channel, ε is a very small constant used to prevent the denominator from being 0, γ, β represent learnable parameters, represents the kth channel of the normalized feature map, The kth channel of the output feature map of the layer normalization is represented. Finally, the output visible light branch feature map of the LayerNorm operation And infrared branch feature map It will be used as the input of the cross attention mechanism.

[0136] In this embodiment, the visible light branching characteristic diagram Will pass through two linear layers to generate the matrix K R and the matrix V R , infrared branch After a linear layer, the matrix Q is generated T , then, through the matrix Q T Perform matrix multiplication on the matrix to calculate the similarity matrix of the two modes, and calculate the corresponding similarity score M through the SoftMax function R , and then the similarity score and matrix V R Multiply them together to get the final output feature map that incorporates additional modal information The above process can be expressed by the following formula:

[0137]

[0138] Among them, W V ,W K ,W Q They are used to generate the matrix V R ,K R ,Q T The mapping matrix, SoftMax(·) represents the activation function, and Represent the input visible light and infrared feature maps respectively, and They represent the output feature maps that incorporate additional modal information, D K Represents the scaling factor, which aims to scale the result of the dot product to ensure that its expected value remains within a smaller range, thereby avoiding the problem of gradient disappearance. Indicates K R The matrix obtained by transposing the operation, V T represents the value matrix obtained by infrared modal mapping, M T Represents the similarity score matrix of infrared modalities.

[0139] In this embodiment, in order to further extract and fuse the feature information of the two modes, the above output matrix is ​​passed through a LayerNorm layer again and combined with the visible light branch feature map The residual connection is performed, and then the output is input into a FFN fully connected layer for final feature extraction. In order to effectively integrate the additional modal information and retain the original modal features to the maximum extent, the feature maps after subsequent operations are all processed with residual connections. The above process can be expressed by the following formula:

[0140]

[0141] Among them, Conv() represents the convolution operation, α, β, γ and λ all represent learnable parameters, FFN() represents the feedforward neural network, which is used to further extract features, and LayerNorm() represents the layer normalization operation. and They represent the output feature maps of visible light and infrared images after the feedforward neural network, respectively. R ' and F T 'represents the feature maps of the visible light and infrared branches after weight adjustment, Reshape() represents the upsampling operation, which is used to map the feature map back to the resolution of the original feature map, so that the feature map after dimensionality reduction is restored to the scale of the original feature map. Similar operations are performed on the infrared dominant branch, and finally two output feature maps are obtained, which are respectively integrated with additional modal information. and

[0142] In this embodiment, in order to merge the features of the two modes into a unified feature representation, the output feature map and Connect along the channel dimension and use a convolution module to fused output. The process is expressed by the following formula:

[0143]

[0144] Among them, F fused Represents the fused feature map.

[0145] S303, using the wavelet convolution feature fusion enhancement module, the fused feature map is fused and enhanced with the two feature maps of the dual-branch backbone network respectively, and the implementation method is as follows:

[0146] S3031, cascading the fused feature map with the two feature maps of the backbone network of the infrared and visible light branches in the channel dimension;

[0147] S3032. Use wavelet convolution to fuse the cascaded feature maps to complete the fusion enhancement processing of the feature maps.

[0148] In this embodiment, Figure 4 As shown, first, the fusion feature map F obtained in the above steps is fused The feature maps of the two branches are cascaded in the channel dimension, and then the two feature maps are fused using wavelet convolution. The following is a description of the visible light branch. Specifically, the input feature map is first decomposed into low-frequency and high-frequency components using wavelet convolution; then, the obtained low-frequency component cascade is further processed; finally, the bottom feature map and the upper feature map are added and aggregated for output through inverse wavelet transform. The above process can be expressed by the following formula:

[0149]

[0150] Among them, WT represents the wavelet convolution operation, represents the low-frequency feature map of the i-1th layer, Y (i) represents the inverse wavelet convolution output feature map of the i-th layer, IWT represents an inverse wavelet convolution operation, Y (ii+1) represents the inverse wavelet convolution output feature map of the i+1th layer, represents the set of high-frequency feature maps of the i-th layer, F LL ,F LH ,F HL ,F HH The input image is processed by four mutually orthogonal convolution kernels to extract the low-frequency and high-frequency components of the image. LL for The low frequency component of F LH ,F HL ,F HH They are the horizontal, vertical and diagonal high-frequency components respectively. In order to expand the receptive field of the feature map, the low-frequency component F LL Finally, the low-frequency and high-frequency components are combined to perform inverse wavelet transform using the inverse operation of wavelet convolution, where: represents the F of the i-th layer R All high frequency components generated, the final Y (0) As the final output of the wavelet convolution feature fusion enhancement module.

[0151] S304, input the infrared and visible light feature maps of the last three scales of the backbone network into the feature fusion enhancement module for fusion, and input the fused feature maps into the Yolov8 neck and head network for target detection, thereby completing the detection of multi-spectral targets.

[0152] In this embodiment, the feature maps F3, F4 and F5 of the two branches in the backbone network are used as inputs, and are first further fused by the wavelet convolution feature fusion enhancement module described in the above steps, and then the fused feature maps are input into the neck network and head module of Yolov8 for target detection. This process can be expressed by the following formula:

[0153]

[0154] in, represents the prediction result of the i-th layer, Head represents the decoupled detection head of Yolov8, It represents the output feature map of the dual-branch backbone network of the i-th layer that fuses the visible light and infrared feature maps. PAFPN represents the feature fusion pyramid network structure of Yolov8. WTConv represents the fusion enhancement processing.

[0155] In this embodiment, the loss function includes three parts: classification loss, confidence loss and bounding box regression loss. The classification loss and confidence loss both use cross entropy loss, and the bounding box regression loss uses C-IOU loss. The final loss is the sum of the three losses, and its formula is expressed as follows:

[0156] loss=loss cls +loss obj +loss bbox

[0157]

[0158] loss cls =loss obj =H(p,q)

[0159] =-∑ x (p(x)logq(x)+(1-p(x)log(1-q(x))))

[0160]

[0161] Among them, loss represents the loss function of the multispectral target detection model, loss cls Represents classification loss, which is responsible for distinguishing objects of different categories. obj Represents the target loss, which is used to measure the probability of an object existing in a grid cell. bboxrepresents the bounding box regression loss, which is composed of lossCIoU, an IoU loss function that considers the center point distance and aspect ratio difference of the box, and is used to accurately locate the object. lossCIoU represents a loss function that considers not only the IoU between the predicted box and the true box, but also the center point distance between the predicted box and the true box. IoU represents the intersection-over-union ratio between the predicted box and the true box, ρ represents the Euclidean distance between the two centers, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, b and b gt denote the center points of the predicted box and the true box respectively, α denotes the weight function, v denotes the similarity used to measure the aspect ratio, H(p,q) denotes the binary cross entropy loss function, x denotes a sample in the sample space, p(x) denotes the probability that the multispectral target detection model predicts that the sample x belongs to the positive class, q(x) denotes the probability that the sample x actually belongs to the positive class, arctan denotes the inverse tangent function, w gt and h gt They represent the width and height of the real box, w and h represent the width and height of the predicted box, respectively.

[0162] For classification loss and confidence loss, the smaller the cross entropy value, the closer the actual output is to the expected output.

[0163] In this embodiment, in order to further verify that the multi-spectral fusion target detection method provided by the embodiment of the present invention is superior to the existing methods, the performance of the present invention and similar cutting-edge methods in recent years are evaluated and compared using evaluation indicators on the data set collected in this embodiment.

[0164] In this embodiment, the evaluation indicators used for the FLIR, LLVIP, and M3FD datasets are specifically:

[0165] The intersection-over-union (IOU) of the predicted box A and the real box B is defined as follows:

[0166]

[0167] TP: The target object is correctly predicted (the IOU between the predicted box and the true box is greater than the preset threshold); FP: The object that does not belong to the target is incorrectly predicted, that is, the negative sample is predicted as a positive sample; FN: The target object is not predicted, that is, the positive sample is regarded as a negative sample. The formulas for precision P and recall R are as follows:

[0168]

[0169] In order to measure the precision and recall rate, the AP indicator is used. The definition of the AP indicator is as follows:

[0170]

[0171] Among them, when the IOU thresholds are 0.5, 0.75 and 0.5:0.95 (step size 0.05), the AP indicators are AP 0.5 ,AP 0.75 ,AP 0.5:0.95 ,The average value of AP index for different classes is the corresponding Map. The larger the evaluation index value, the better performance of the model.

[0172] The specific evaluation indicators used for the KAIST dataset are:

[0173] The false positive per picture (FPPI) is defined as:

[0174]

[0175] Where N represents the total number of images, and the missed detection rate MR is defined as:

[0176] MR=1-R

[0177] Logarithmic mean missed detection rate MR -2 Defined as: Under the FPPI threshold sampled uniformly at logarithmic intervals, calculate the corresponding missed detection rate MR i Then calculate the average:

[0178]

[0179] Here, ln represents natural logarithm, and exp represents exponent.

[0180] In the comparative experiment of KAIST data set, the logarithmic mean missed detection rate MR is used. -2 The smaller the value of the indicator, the better the performance of the model.

[0181] In summary, after multiple sets of experimental verification, the effectiveness of the method has been verified on multiple standard data sets, and it has good generalization ability.

Claims

1. A multispectral target detection method based on adaptive attention mechanism, characterized in that: The following steps are involved: S1. Collect multispectral image datasets of different scenes and categories and divide them into training sets and test sets; S2. Build a multispectral target detection model based on adaptive attention mechanism, and use training set and test set to train and test the multispectral target detection model; S3. Perform multispectral target detection using the trained and tested multispectral target detection model.

2. The multispectral target detection method based on the adaptive attention mechanism according to claim 1 is characterized in that: The loss function of the multispectral target detection model is expressed as follows: loss=loss cls +loss obj +loss bbox loss cls =loss obj =H(p,q)=-∑ x (p(x)logq(x)+(1-p(x)log(1-q(x)))) Among them, loss represents the loss function of the multispectral target detection model, loss cls Represents classification loss, which is responsible for distinguishing objects of different categories. obj Represents the target loss, which is used to measure the probability of an object existing in a grid cell. bbox represents the bounding box regression loss, which is composed of lossCIoU, an IoU loss function that considers the center point distance and aspect ratio difference of the box, and is used to accurately locate the object. lossCIoU represents a loss function that considers not only the IoU between the predicted box and the true box, but also the center point distance between the predicted box and the true box. IoU represents the intersection-over-union ratio between the predicted box and the true box, ρ represents the Euclidean distance between the two centers, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, b and b gt denote the center points of the predicted box and the true box respectively, α denotes the weight function, v denotes the similarity used to measure the aspect ratio, H(p,q) denotes the binary cross entropy loss function, x denotes a sample in the sample space, p(x) denotes the probability that the multispectral target detection model predicts that the sample x belongs to the positive class, q(x) denotes the probability that the sample x actually belongs to the positive class, arctan denotes the inverse tangent function, w gt and h gt They represent the width and height of the real box, w and h represent the width and height of the predicted box, respectively.

3. The multispectral target detection method based on adaptive attention mechanism according to claim 1, characterized in that: The step S3 comprises the following steps: S301, input the collected multispectral image into the dual-branch backbone network for information extraction, and obtain feature maps of different scales by downsampling, wherein the dual-branch architecture is based on the Yolov8 architecture, transforming the original single-branch architecture into a dual-branch architecture, the dual branches are infrared and visible light branches, and each branch in the dual-branch backbone network includes a base layer stem and a four-layer feature extraction layer stage i , the input image F0 of each branch is passed through the base layer stem to obtain the feature map F1, and then passes through four layers of feature extraction layers to obtain the feature maps F2, F3, F4 and F5; S302, inputting the feature maps of the same scale of the infrared and visible light branches into the adaptive cross-attention feature module to fuse different modal information; S303, using a wavelet convolution feature fusion enhancement module, the fused feature map is fused and enhanced with the two feature maps of the dual-branch backbone network respectively; S304, input the infrared and visible light feature maps of the last three scales of the backbone network into the feature fusion enhancement module for fusion, and input the fused feature maps into the Yolov8 neck and head network for target detection, thereby completing the detection of multi-spectral targets.

4. The multispectral target detection method based on adaptive attention mechanism according to claim 3 is characterized in that: The expression of the feature map F1 is as follows: F1=ConvModule(F0) ConvModule(·)=Silu(BatcheNorm(Conv2d(·))) Among them, ConvModule(·) represents the module set used for convolution operation, Silu(·) represents the activation function, BatcheNorm(·) represents batch normalization, and Conv2d(·) represents batch normalization, which represents the convolution operation with a kernel size of 3 and a stride of 2. The four-layer feature extraction stage i Both include a convolution module and a cross-stage local network module CSPDarknet, where the feature map F1 to the feature map F i+1 The expression is as follows: F i (1) ,F i (2) =Split(F i ) CSPDarkNet=Conv2d(Concat(F i (1),F res )) F i+1 =CSPDarkNet(ConvModule(F i )),i=1,2,3 F i+1 =SPPF(CSPDarkNet(ConvModule(F i ))),i=4 Among them, F i (1) and F i (2) represent the feature map F i The first and second parts after segmentation, Split(·) means splitting the input feature map along the channel dimension, F res Represents the second part F of the segmented feature map i (2) Execution The output after the operation is Indicates that the input of the last residual block in the cascade of k residual modules is used as the overall output, CSPDarkNet represents the Darknet structure in the cross-stage local network, Conv2d(·) represents the two-dimensional convolution function, Concat(·) represents the concatenation of feature maps along the channel dimension, and F i+1 represents the next layer of feature maps, ConvModule(·) represents the convolution operation, SPPF(·) represents the fast spatial pyramid pooling operation, It means that k maximum pooling operations are performed on the input feature map, and the pooling results are concatenated along the channel dimension.

5. The multispectral target detection method based on adaptive attention mechanism according to claim 2, characterized in that: The step S302 includes the following steps: S3021, inputting the feature maps of the same scale of the two branches of infrared and visible light into the modality adaptation module, and adjusting the weights of the respective modalities; S3022, inputting the weighted feature map into a spatial feature dimensionality reduction module to perform dimensionality reduction processing to obtain a visible light and infrared branch feature map after dimensionality reduction; S3023. Input the reduced-dimensional visible light and infrared branch feature maps into the adaptive cross-attention feature module for feature fusion to complete the fusion of different modal information.

6. The multispectral target detection method based on adaptive attention mechanism according to claim 5, characterized in that: The step S3021 includes the following steps: S30211. The feature maps F of the same scale of the infrared and visible light branches are T and F R , concatenate according to the channel dimension, and perform average pooling to calculate the weights of each modality; S30212. The modal weight is further split into two branches according to the channel dimension, and multiplied with the input feature maps of the original visible light branch and the infrared branch respectively, and the feature maps of the visible light and infrared branches after weight adjustment are output to complete the adjustment of the modal weight: F R '=W R *F R F T '=W T *F T W R ,W T =Split(AvgPool(Concat(F R ,F R ))) Among them, F R ' and F T 'represent the feature maps of the visible light and infrared branches after adjusting the weights, W R and W T They represent the weights of their respective branches, Split(·) means splitting the input feature map along the channel dimension, AvgPool(·) means the average pooling operation, and Concat(·) means concatenating the feature maps along the channel dimension.

7. The multispectral target detection method based on adaptive attention mechanism according to claim 5, characterized in that: The expression of the feature map after dimensionality reduction in step S3022 is as follows: F d =F max +F avg F max =MaxPool(F') F avg =AvgPool(F') Among them, F d Indicates that F max and F avg The output feature map obtained by adding element by element, F max and F avg They represent the output feature maps after the maximum pooling and average pooling operations on the feature maps, respectively. MaxPool(·) represents the maximum pooling operation, AvgPool(·) represents the average pooling operation, and F' represents any feature map of visible light and infrared.

8. The multispectral target detection method based on adaptive attention mechanism according to claim 5, characterized in that: The step S3023 includes the following steps: S30231. Normalize the visible light and infrared branch feature maps after dimensionality reduction respectively to obtain a normalized visible light branch feature map. And infrared branch feature map S30232, visible light branch feature map After two linear layers, the matrix K is generated R and the matrix V R , the infrared branch feature map After a linear layer processing, the matrix Q is generated T ; S30233, according to the matrix Q T and matrix K R , calculate the similarity matrix of the two modes, and calculate the corresponding similarity score M through the SoftMax function R ; S30234, according to the similarity score M R and the matrix V R , calculate the output feature map that incorporates additional modal information and Among them, D K represents the scaling factor, W V , W K and W Q They are used to generate the matrix V R , K R and Q T The mapping matrix, SoftMax(·) represents the activation function, V T represents the value matrix obtained by infrared modal mapping, M T Represents the similarity score matrix of infrared modalities; S30235, feature map Pass through the normalization layer again and compare it with the visible light branch feature map Perform residual connection; S30236. Input the residual connection result to the fully connected layer for feature extraction to obtain an output feature map that incorporates additional modal information. and Among them, Conv() represents the convolution operation, Reshape() represents the upsampling operation, which is used to map the feature map back to the resolution of the original feature map, α, β, γ and λ all represent learnable parameters, FFN() represents the feedforward neural network, which is used to further extract features, and LayerNorm() represents the layer normalization operation. and They represent the output feature maps of visible light and infrared images after the feedforward neural network, respectively. R ' and F T 'represent the feature maps of visible light and infrared branches after adjusting the weights; S30237, feature map and Splicing is performed along the channel dimension and fused into a unified feature representation to complete the fusion of different modal information: Among them, F fused Represents the fused feature map.

9. The multispectral target detection method based on adaptive attention mechanism according to claim 3, characterized in that: The step S303 includes the following steps: S3031, cascading the fused feature map with the two feature maps of the backbone network of the infrared and visible light branches in the channel dimension; S3032. Use wavelet convolution to fuse the cascaded feature maps to complete the fusion enhancement processing of the feature maps.

10. The multispectral target detection method based on adaptive attention mechanism according to claim 3, characterized in that: The step S304 is specifically as follows: The feature maps F3, F4 and F5 of the infrared and visible light branches in the backbone network are taken as input and fused; The fused feature map is input into the Yolov8 neck and head network for target detection to complete the detection of multispectral targets: in, represents the prediction result of the i-th layer, Head represents the decoupled detection head of Yolov8, It represents the output feature map of the dual-branch backbone network of the i-th layer that fuses the visible light and infrared feature maps. PAFPN represents the feature fusion pyramid network structure of Yolov8. WTConv represents the fusion enhancement processing.

Images