Multi-modal feature target detection method based on dynamic convolution and attention mechanism

Abstract

The present application relates to a multi-modal feature target detection method based on dynamic convolution and attention mechanism, belonging to the field of image recognition. The method has two data streams at the beginning of the Backbone of YOLOv5, respectively inputting visible light images and infrared light images, and using a dynamic convolution module ODConv and a multispectral convolution attention feature fusion module MS-CBAM and a residual network for feature extraction operation. The present application has the advantages of combining the features of visible light images and infrared images, combining multiple attention mechanisms and architectures, greatly improving the multi-modal and small target target detection accuracy, and solving the problem of weak target detection performance in dim environments. Compared with other multi-modal fusion target detection, the present application has fast training speed and low hardware resource consumption.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and relates to a multimodal feature target detection method based on dynamic convolution and attention mechanisms. Background Technology

[0002] Object detection is a crucial technology in computer vision tasks, directly impacting detection accuracy and computational efficiency. Therefore, this field has consistently attracted attention from academia and industry. This invention discusses object detection that aims to improve overall network performance using novel modal data and modal fusion methods. For example, at night, traffic systems may face insufficient light for surveillance cameras, making it difficult to capture images of most violations, monitor pedestrians and vehicles, and automatically issue accident alarms from a single spectral data source. Enhancing the visible light image features of vehicles and pedestrians at night using infrared images captured by infrared cameras can significantly improve nighttime object detection accuracy. Therefore, how to utilize large amounts of multispectral image data to improve the performance of object recognition and detection models is a highly valuable and challenging research task. The multimodal feature fusion dual-stream neural network integrates information from these two different modalities into a deep learning neural network, greatly improving the training accuracy and precision of object detection in addressing the aforementioned problems. However, existing CNNs can only fuse information in local regions through convolutional receptive fields. Two-stream convolutional neural networks cannot effectively utilize the complementarity between different modalities. Simply stacking feature maps increases the learning difficulty of the neural network, exacerbates modal imbalance, and leads to performance degradation. This invention modifies the existing YOLOv5 neural network model by introducing an improved channel attention, spatial attention, and dynamic convolution to form a modal fusion module. This allows for more comprehensive cross-modal fusion, learning, and prediction of the two modalities under multiple attention levels. Simultaneously, the NWD localization loss function is used to enhance the accuracy of small target detection. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a multimodal feature target detection method based on dynamic convolution and attention mechanisms.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A multimodal feature-based target detection method based on dynamic convolution and attention mechanisms includes the following steps:

[0006] S1: Establish a neural network model based on YOLOv5 dual-stream convolutional detection network, where Backbone uses convolutional operations and feature fusion modules for modality fusion and feature learning;

[0007] S2: The multispectral module MS-CBAM is composed of channel attention and spatial attention. Channel attention is used to weight the features of the visible light and infrared light images respectively. Then, the infrared light and visible light images are stacked together and spatial attention is used to weight the features of the feature map. Finally, a residual network is used to refine the features.

[0008] S3: Introduce a multi-head attention mechanism to the convolutional structure. By assigning different attention coefficient matrices to the four dimensions of input channel dimension, output channel dimension, spatial dimension and convolutional kernel, a dynamic convolution ODConv module is established.

[0009] S4: Set the MS-CBAM module as the larger position of the 80×80×256 feature map for output, and the ODConv module as the medium and small positions of the 40×40×512 and 20×20×1024 feature maps for output; output the three feature maps of different sizes into the Neck layer, i.e., the feature pyramid, for feature extraction, predict the output features, and output the prediction results.

[0010] S5: During the training phase, visible light and infrared light data undergo specific Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling before being fed into the dual-stream neural network for training. YOLO v5l pre-trained weights are used for initialization, and stochastic gradient descent is used to learn the network parameters.

[0011] In the prediction phase, a softmax classifier is used to obtain the final classification probability of the class.

[0012] During the optimization phase, a joint optimization approach using localization loss, classification loss, and confidence loss is adopted to reduce the error between the true and predicted values. NWD is introduced into the localization loss to improve the accuracy of small target detection. S5 is repeated continuously until the set number of iterations is reached, at which point the model training is complete, and the target detection task is performed.

[0013] Optionally, in S1, the input of the YOLO v5-based two-stream convolutional object detection network framework is a pair of images of different modalities, the backbone is a two-stream convolutional network, and the two-stream neural network model includes a backbone, a neck, and a prediction layer.

[0014] Let the input visible light feature map be X. V The input infrared feature map is X. T The length, width, and number of channels of the feature map are H, W, and C, respectively;

[0015] The feature extraction network structure uses three feature fusion modules and a residual network to form a three-stage feature extraction loop and refinement structure. The i-th feature fusion calculation process is represented as follows:

[0016]

[0017] Where σ is the feature fusion function, and the input feature map of the visible light image is X. V The infrared light image input feature map is X. T F is the feature fusion module, which performs batch normalization operations; it fuses feature maps. The length, width, and number of channels are H, W, and 2C, respectively; then, the fused features and the original features are used to construct a residual network:

[0018]

[0019]

[0020] To obtain new feature maps for visible and infrared light f t i and

[0021] Optionally, in S2, for the input images of visible light and infrared light, channel attention mechanism is calculated for both respectively, and then the feature maps are superimposed according to the channel dimension, and then input into the spatial attention for calculation.

[0022] The calculations of the MS-CBAM module are represented as follows:

[0023] X = M S [concat[M C (X V ),M C (X T )]]

[0024] Among them, M C Represents the channel attention mechanism, M S Represents spatial attention mechanism; Concat indicates stacking feature maps along the channel dimension;

[0025] Next, a residual network is constructed for X to refine the model. The process is represented as follows:

[0026] X' V =X V +X

[0027] X' T =X T +X

[0028] The final feature map obtained is X' V ∈V B×C×H×W 、X' T ∈T B×C×H×W , indicating the final output of the MS-CBAM module.

[0029] Optionally, in S3, a multi-head self-attention mechanism is introduced during the convolution process. Different attention coefficient matrices ODConv are assigned to the convolution in four dimensions: input channel dimension, output channel dimension, spatial dimension, and convolution kernel, thereby improving feature extraction capabilities. The overall operation of the ODConv module is represented as follows:

[0030] X'=ODConv(concat(X V ,X T ))

[0031] Among them, X V and X T The inputs are the feature maps of the visible light and infrared light modes, respectively. concat means that the two inputs are superimposed along the channel number dimension, and ODConv means dynamic convolution operation.

[0032] Its dynamic convolution formula, which integrates four dimensions, is expressed as follows:

[0033] y=(α w1 ⊙α f1 ⊙α c1 ⊙α s1 ⊙W1+...+α wn ⊙α fn ⊙α cn ⊙α sn ⊙W n )*x

[0034] W is the kernel dimension. i Attention coefficient matrix, and These represent the paths along the convolution kernel W. i The dynamic convolutional attention coefficient matrix in the spatial dimension, input channel dimension, and output channel dimension of the kernel space, where ⊙ represents the multiplication operation along different dimensions of the kernel space.

[0035] Optionally, the inputs and outputs of the MS-CBAM module and the ODConv module are both visible light and infrared light feature maps, and the output will form a residual network with the input;

[0036] The loss functions for localization loss, classification loss, and confidence loss are expressed as follows:

[0037] L total =L box +L cls +L conf

[0038] The localization loss uses the NWD loss function; the NWD loss function uses the Normalized Wasserstein Distance calculation method to calculate similarity through the corresponding Gaussian distribution.

[0039] The beneficial effects of the present invention are as follows: the present invention can effectively optimize target detection under conditions of insufficient brightness in the whole or part of the image, and has greater advantages in terms of prediction accuracy and reliability when applied to target detection systems.

[0040] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0042] Figure 1 This is a flowchart illustrating the overall architecture of the present invention;

[0043] Figure 2 This is a flowchart of the Backbone process of the present invention;

[0044] Figure 3 This is a structural diagram of the dynamic convolutional feature fusion module;

[0045] Figure 4 This is a structural diagram of the MS-CBAM feature fusion module. Detailed Implementation

[0046] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0047] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0048] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0049] This invention provides a target detection method based on dynamic convolution and attention mechanisms and a YOLO v5 two-stream network, such as... Figure 1 As shown, the method includes the following steps:

[0050] Step 1: Construct a basic two-stream neural network model in Step 1, such as... Figure 2 As shown, YOLO v5 includes data processing, Backbone, Neck, prediction layer, and the present invention designs a feature extraction and fusion approach based on dynamic convolution feature fusion module, MS-CBAM multimodal cyclic fusion and refinement, repeating the fusion operation multiple times and performing residual processing to increase the consistency of multispectral features.

[0051] Step 2: In Step 2, construct a dynamic convolution ODConv that assigns different attention coefficient matrices to the convolution based on four dimensions: input channel, output channel, spatial distribution, and convolution kernel. For example... Figure 3 As shown, a multi-head attention mechanism and parallel strategy are used to learn modal complementary attention in four dimensions of the convolutional kernel.

[0052] Step 3: Construct an MS-CBAM module based on channel attention and spatial attention, such as... Figure 4 As shown, the feature maps are weighted in both the channel and spatial dimensions, and a residual network is used to refine the features.

[0053] Step 4: Set the MS-CBAM module to output the larger 80×80×256 feature map, and the ODConv module to output the medium and small 40×40×512 and 20×20×1024 feature maps, respectively. Input the feature maps into the YOLO v5 feature pyramid and continue with YOLO v5 feature fusion and prediction.

[0054] Step 5: Train the neural network with the determined parameters using training samples until the training conditions are met, and then test the trained neural network using a test set;

[0055] In step 1, the present invention performs prediction based on the Neck and Head layers of YOLO v5 and establishes a baseline network based on a dual-stream convolutional network. The feature is that the convolutional network is first used to extract the local features of the visible light and infrared light dual modes, and then the feature fusion module is used to perform feature weighted fusion operation.

[0056] First, the visible light and infrared light images processed in step 1 are subjected to three convolution operations respectively. The convolutional visible light and infrared light feature maps are represented as X. V X T .

[0057] This invention designs a feature fusion method using a residual network composed of an MS-CBAM module and an ODConv module for feature fusion. For example... Figure 2 As shown, the module and the residual network jointly construct a feature extraction and fusion approach that involves cyclic fusion and refinement of features. This invention performs feature fusion operations at three locations in the YOLO v5 network: 80×80×256, 40×40×512, and 20×20×1024. Figure 2 P3, P4, and P5, representing large, medium, and small feature maps respectively, are incorporated into the feature pyramid. The feature cyclic fusion and refinement structure of this invention can increase the consistency of multispectral features. Let's assume that in the i-th fusion module, to obtain the new fused feature f, the visible light image feature X... V With infrared image features X T The fusion process can be represented as:

[0058]

[0059] Where σ is the feature fusion function and F is the feature fusion module.

[0060] To avoid overfitting, the operations F in all loops share weights, and then the fused features are used to construct a residual network with the original features:

[0061]

[0062]

[0063] To prevent the vanishing gradient problem when learning network parameters and to better fuse multispectral features, an auxiliary semantic segmentation task is used to bring separate information to each refined spectral feature.

[0064] The similarity between modes increases with the number of iterations, while the consistency and complementarity of spectral features increase with the increase in similarity. Consistency among multispectral features is crucial; conversely, excessive consistency can lead to sharp increases or decreases in eigenvalues, rendering redundant iterations meaningless. Experiments show that feature fusion performance begins to decline after the fourth iteration. Therefore, in practice, we use three iterations to balance consistency and complementarity.

[0065] Meanwhile, the three feature fusion modules will input three types of processed feature maps—large, medium, and small—into the feature pyramid in three separate steps.

[0066] Furthermore, in step 2, this invention is illustrated using dynamic convolution with multi-head self-attention mechanism operations on the kernel dimension. For a dynamic convolutional layer, it uses a linear combination of n convolutional kernels, dynamically weighted through an attention mechanism, making the convolution operation dependent on the input feature map. The overall operation of ODConv can be expressed as:

[0067] X'=ODConv(concat(X v ,X T ))

[0068] Among them, X V and X T The inputs are the feature maps for the visible light and infrared light modes, respectively. concat represents the superposition of the two inputs along the channel number dimension, and ODConv represents the dynamic convolution operation.

[0069] Specifically, mathematically, a dynamic convolution operation in a single dimension can be defined as:

[0070] y=(α w1 W1+...+α wn W n )*x

[0071] in, and represents the input and output of a feature map matrix with height h, width w, and number of channels c, respectively. i Indicates the output convolution filter The i-th convolutional kernel is formed, m = 1, ..., c out . The attention coefficient matrix is ​​of convolution kernel dimension, which is composed of the attention function π conditioned on the input features. wi (x) calculates; * indicates convolution operation, where the bias term is omitted.

[0072] According to the equation for dynamic convolution, dynamic convolution has two basic components: given n convolution kernels, the convolution kernel W... i and the attention function used to compute its attention scalar The corresponding kernel space has four dimensions with a kernel size of k×k, and each convolutional kernel has c input channels. in and the number of output channels c out .

[0073] The ODConv module in this invention simultaneously considers the kernel dimension, spatial dimension, input channel dimension, and output channel dimension, making the multimodal feature fusion in convolution operations more comprehensive. The formula for each dimension is similar to that of dynamic convolution in the kernel dimension. For example... Figure 3 As shown, its dynamic convolution formula, which integrates four dimensions, can be expressed as:

[0074] y=(α w1 ⊙α f1 ⊙α c1 ⊙α s1 ⊙W1+...+α wn ⊙α fn ⊙α cn ⊙α sn ⊙W n )*x

[0075] For convolution kernel W i Attention coefficient matrix, and These represent the paths along the convolution kernel W. i The dynamic convolutional attention coefficient matrix in the spatial dimension, input channel dimension, and output channel dimension of the kernel space, where ⊙ represents the multiplication operation along different dimensions of the kernel space.

[0076] Where, α si Different attention scalars are assigned to each convolutional filter at k×k spatial locations; α ci For each convolution filter W i m c in Different attention scalars are assigned to channels; α fi For each convolution filter W i m c out Different attention scalars are assigned to channels; α wiAn attention scalar is assigned to the entire convolutional kernel. This is achieved by multiplying the attention coefficient matrix of these four dimensions with the corresponding dimensions of the n convolutional kernels to obtain the module's output.

[0077] Specifically, the input X is first compressed into a form with c by a global average pooling operation. in The feature vector of length is passed through a fully connected layer and a ReLU unit. The fully connected layer maps the compressed feature vector to a low-dimensional space with a reduction rate r. Then, it goes through four branches, each corresponding to one of the four dimensions mentioned above, and each branch has an output of size k×k, c. in ×1、c out Full-joint (FC) layers of ×1 and n×1, along with a Softmax or Sigmoid function, generate normalized attention coefficient matrices α, respectively. si α ci α fi and α wi .

[0078] Because these four dimensions are complementary and can capture rich contextual cues, ODConv can significantly enhance the feature extraction capabilities of basic convolutional operations in CNNs.

[0079] Furthermore, in step 3, an MS-CBAM module based on channel attention and spatial attention is established, which weights the feature map in the channel dimension and the spatial dimension respectively, and uses a residual network to refine the features.

[0080] For the input feature map X V ∈V B×C×H×W X T ∈T B×C×H×W Where V represents the visible light image, T represents the infrared image, B represents the batch size, C represents the number of channels, and H and W represent the length and width of the feature map, respectively, in pixels. The calculation of the MS-CBAM module can be expressed as:

[0081] X = M s [concat[M c (X V ),M c (X T )]]

[0082] Among them, M c Represents the channel attention mechanism, M s This represents the spatial attention mechanism. Concat indicates stacking feature maps along the channel dimension, and X represents the module output. By combining channel attention and spatial attention, features can be weighted in both the channel and spatial dimensions. This can reduce the adverse effects of using only one type of pooling operation and improve the accuracy performance of the neural network.

[0083] The Channel Attention Module (CAM) improves the representational power of feature maps by learning the interactions between each channel. Specifically, the CAM first performs max pooling and average pooling operations on each channel of the input feature map sequentially, obtaining max-pooled and average-pooled feature maps. These two feature maps are then used as input, passed through two fully connected layers and a sigmoid function to obtain the weights for each channel. The channel weights are then multiplied by the original feature map to obtain a weighted feature map. The channel attention mechanism can be expressed as:

[0084]

[0085] In the formula, and These represent average pooling and max pooling, respectively.

[0086] The Spatial Attention Module (SAM) improves the representational power of feature maps by learning the interactions between each pixel in the feature map. The input feature map to this module is the feature map output by the Channel Attention Module. First, for an input feature map, the SAM performs max pooling and average pooling operations to obtain max-pooled and average-pooled feature maps. These two feature maps are then concatenated, passed through a convolutional layer and a sigmoid function to obtain the weights for each pixel, and multiplied by the pixel weights with the original feature map to obtain a weighted feature map. Next, average pooling and max pooling are performed on the channel dimensions of the visible light and infrared light feature maps, respectively, resulting in two feature maps of size 1. These two feature maps are then concatenated along the channel dimension to obtain a single feature map of size 1. Finally, this feature map is reduced to one channel through a 7×7 convolutional operation and then activated by a sigmoid activation function to generate spatial attention features.

[0087] Finally, the output features of spatial attention are multiplied element-wise with the input features to obtain the final generated features. The spatial attention mechanism can be represented as:

[0088]

[0089] In the formula, and These represent average pooling and max pooling, respectively.

[0090] This invention uses channel attention and spatial attention, and then refines X by constructing a residual network. The process can be represented as follows:

[0091] X' V =X V +X

[0092] X' T =X T +X

[0093] The final feature map obtained is X' V ∈V B×C×H×W 、X' T ∈T B×C×H×W , indicating the final output of the MS-CBAM module.

[0094] Furthermore, in step 4, the feature map sizes H, W, and C are set to 80×80×256, 40×40×512, and 20×20×1024, respectively. Figure 2 The feature maps at positions P3, P4, and P5 were fused using MS-CBAM, ODConv, and ODConv, respectively. Then, these three large, medium, and small feature maps were input into the YOLO v5 Neck feature pyramid for further feature fusion and extraction.

[0095] In step 5, the loss function is divided into localization loss, classification loss, and confidence loss, which can be expressed as:

[0096]

[0097] The localization loss uses NWD, while the other losses use the default YOLO v5 loss function.

[0098] NWD uses a Wasserstein distance-based metric, which significantly improves the performance of small target detection.

[0099] For small objects, the bounding box will always contain some background pixels because a real object is unlikely to be exactly a rectangle. Within the bounding box, foreground pixels are generally concentrated in the center, while background pixels are generally concentrated on the edges. To better weight each pixel within the bounding box, it can be modeled as a 2D Gaussian distribution. Specifically, for a horizontal bounding box R = (cx, cy, w, h), it can be represented using an inscribed ellipse as follows:

[0100]

[0101] Among them, (μ x ,μ y ) is the center point of the ellipse, (σ) x ,σ y () represents the radius of the x and y axes. Corresponding to the bounding box:

[0102] μ x =cx,μ y =cy,

[0103] The probability density function of a 2D Gaussian distribution is:

[0104]

[0105] Where X, μ, and ∑ represent the coordinates (x, y), mean, and variance, respectively. When:

[0106]

[0107] This ellipse represents the outline of a 2D Gaussian distribution. Therefore, the horizontal bounding box R = (cx, cy, w, h) can be modeled as a 2D Gaussian distribution:

[0108]

[0109] In this way, the similarity between two bounding boxes can be represented by the distance between these two Gaussian distributions.

[0110] Next, this invention uses the Wasserstein distance from optimal transport theory to calculate the distance between the two distributions. For two 2D Gaussian distributions, their second-order Wasserstein distance can be defined as:

[0111]

[0112] Right now:

[0113]

[0114] For the two bounding boxes:

[0115]

[0116] However, this is a distance metric and cannot be directly used for similarity. We use the normalized exponent to obtain a new metric called the normalized Wasserstein distance:

[0117]

[0118] Here, C is a constant that depends on the dataset.

[0119] Next, the model is trained on the pre-constructed input dataset. At each epoch, the model parameters for that epoch are saved, and the classification accuracy of the current epoch is compared with the classification accuracy of the previous best model. When the set maximum number of epochs is reached, the pedestrian target recognition model with the highest recognition accuracy is output. The trained model can detect and recognize targets in low-light conditions, including people, animals, cars, other vehicles, and obstacles.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal feature target detection method based on dynamic convolution and attention mechanisms, characterized in that: The method includes the following steps: S1: Establish a neural network model based on YOLO v5 dual-stream convolutional detection network, where Backbone uses convolutional operations and feature fusion modules for modality fusion and feature learning; S2: The multispectral module MS-CBAM is composed of channel attention and spatial attention. Channel attention is used to weight the features of the visible light and infrared light images respectively. Then, the infrared light and visible light images are stacked together and spatial attention is used to weight the features of the feature map. Finally, a residual network is used to refine the features. S3: Introduce a multi-head attention mechanism to the convolutional structure. By assigning different attention coefficient matrices to the four dimensions of input channel dimension, output channel dimension, spatial dimension and convolutional kernel, a dynamic convolution ODConv module is established. S4: Set the MS-CBAM module as the large 80×80×256 feature map for output, and the ODConv module as the medium and small 40×40×512 and 20×20×1024 feature maps for output; output the three feature maps of different sizes into the Neck layer, i.e., the feature pyramid, for feature extraction, predict the output features, and output the prediction results. S5: During the training phase, visible light and infrared light data undergo Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling before being fed into the dual-stream neural network for training. YOLO v5l pre-trained weights are used for initialization, and stochastic gradient descent is used to learn the network parameters. In the prediction phase, a softmax classifier is used to obtain the final classification probability of the class. During the optimization phase, a joint optimization approach using localization loss, classification loss, and confidence loss is adopted to reduce the error between the true and predicted values. NWD is introduced into the localization loss to improve the accuracy of small target detection. S5 is repeated continuously until the set number of iterations is reached, at which point the model training is complete, and the target detection task is performed.

2. The multimodal feature target detection method based on dynamic convolution and attention mechanism according to claim 1, characterized in that: In S1, the input to the YOLO v5-based dual-stream convolutional object detection network framework is image pairs of different modalities. The backbone is a dual-stream convolutional network, and the dual-stream neural network model includes a backbone, a neck, and a prediction layer. Let the input visible light feature map be... The input infrared light feature map is The length, width, and number of channels of the feature map are H, W, and C, respectively; The feature extraction network structure uses three feature fusion modules and a residual network to form a three-stage feature extraction loop and refinement structure. i The sub-feature fusion calculation process is represented as follows: in The feature fusion function is the visible light image input feature map. The infrared light image input feature map is , F For the feature fusion module, perform batch normalization operations; fuse feature maps. The length, width, and number of channels are H, W, and 2C, respectively; then, the fused features and the original features are used to construct a residual network: Acquiring new feature maps for visible and infrared light and .

3. The multimodal feature target detection method based on dynamic convolution and attention mechanism according to claim 2, characterized in that: In S2, for the input images of visible light and infrared light, channel attention mechanism is calculated for both respectively. Then, the feature maps are superimposed according to the channel dimension and then input into the spatial attention for calculation. The calculations of the MS-CBAM module are represented as follows: in, Represents the channel attention mechanism. Represents spatial attention mechanisms; Concat represents stacking feature maps along the channel dimension; After that The process of constructing a residual network for refinement is represented as follows: The final feature map obtained is , , indicating the final output of the MS-CBAM module.

4. The multimodal feature target detection method based on dynamic convolution and attention mechanism according to claim 3, characterized in that: In S3, a multi-head self-attention mechanism is introduced during the convolution process. Different attention coefficient matrices ODConv are assigned to the convolution in four dimensions: input channel dimension, output channel dimension, spatial dimension, and convolution kernel, thereby improving feature extraction capabilities. The overall operation of the ODConv module is represented as follows: in, and The inputs are the feature maps of the visible light and infrared light modes, respectively. concat means that the two inputs are superimposed along the channel number dimension, and ODConv means dynamic convolution operation. Its dynamic convolution formula, which integrates four dimensions, is expressed as follows: For the convolution kernel dimension Attention coefficient matrix, and These represent the paths along the convolution kernel. The dynamic convolutional attention coefficient matrix in the spatial dimension, input channel dimension, and output channel dimension. This represents multiplication operations along different dimensions of the kernel space.

5. The multimodal feature target detection method based on dynamic convolution and attention mechanism according to claim 4, characterized in that: The MS-CBAM module and ODConv module both input and output are visible light and infrared light feature maps, and the output will form a residual network with the input. The loss functions for localization loss, classification loss, and confidence loss are expressed as follows: The localization loss uses the NWD loss function; the NWD loss function uses the Normalized Wasserstein Distance calculation method to calculate similarity through the corresponding Gaussian distribution.

6. The multimodal feature target detection method based on dynamic convolution and attention mechanism according to claim 5, characterized in that: The NWD loss function is expressed as: in, For Wasserstein distance, Let C be the distance between two Gaussian distributions, and let C be a fixed constant related to the dataset to improve the detection performance of small targets.

Images