A method for target detection and recognition based on dual-mode image fusion

Abstract

The present invention provides a method for target detection and recognition based on dual-mode image fusion. First, a dual-mode encoder is used to extract features from the acquired visible light image and infrared image; then, independent component analysis is performed on the extracted features and weighted fusion is performed; finally, a Co-DETR target detection model is constructed, and the trained model is used to complete the final target detection and recognition. The present invention can greatly improve the accuracy and robustness of target detection, and take into account both real-time and robustness.

Description

Technical Field

[0001] The present invention belongs to the field of radar signal processing, and in particular relates to a target detection and recognition method based on dual-mode image fusion. Background Art

[0002] With the development of artificial intelligence technology and sensor technology, image-based target detection and recognition has become an important research direction in the field of computer vision, and is widely used in many fields such as autonomous driving, security monitoring, and medical image analysis. Traditional target detection methods usually rely on single-mode sensors, such as visible light cameras or infrared cameras. However, in complex environments, single-mode image information is limited by the influence of the external environment, such as lighting conditions, occlusion, and bad weather, making it difficult for detection accuracy and robustness to meet actual needs.

[0003] In order to overcome the limitations of single-mode sensors, multimodal data fusion technology has gradually attracted widespread attention. Multimodal image fusion can effectively enhance the detectability and recognition accuracy of targets by combining the information of visible light and infrared imaging. Among them, dual-mode image fusion technology is particularly important because visible light images can provide rich color and texture information, while infrared images can provide thermal radiation information of targets in low light or at night when visible light fails. By effectively fusing the image information of these two modes, the performance of target detection and recognition can be significantly improved, especially in complex and changing environments.

[0004] At present, the target detection and recognition technology of dual-mode image fusion is mainly carried out in the following ways: image layer fusion, feature layer fusion and decision layer fusion. Image layer fusion refers to the fusion directly on the original image, such as the fusion of visible light and infrared images through weighted average, Laplace pyramid and other methods. Feature layer fusion is to extract features from the two modal images separately and then fuse the features of different modalities to improve the detection performance of the target. Decision layer fusion is to fuse the detection results of different modalities after processing the data of each modality independently, and finally obtain more accurate detection and recognition results.

[0005] However, dual-mode image fusion still faces some technical challenges. In terms of how to effectively extract and fuse features of different modalities, existing methods lack accuracy in complex scenes, especially when processing subtle features and edge information of the target. It is often difficult to simultaneously retain detail information in visible light images and thermal feature information in infrared images. Secondly, in the heterogeneous fusion of multimodal information, the weight distribution is not adaptable enough, and it is difficult to make reasonable adjustments according to dynamic changes in the environment. Furthermore, the computational complexity of the fusion algorithm is high, and it is difficult to achieve real-time performance while ensuring accuracy. That is, ensuring both real-time and robustness of information fusion is still a problem to be solved. These problems jointly limit the practical application effect of dual-mode image fusion technology. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a target detection and recognition method based on dual-mode image fusion. First, a dual-mode encoder is used to extract features from the acquired visible light image and infrared image; then, independent component analysis is performed on the extracted features and weighted fusion is performed; finally, a Co-DETR target detection model is constructed, and the trained model is used to complete the final target detection and recognition. The present invention can greatly improve the accuracy and robustness of target detection, and take into account both real-time and robustness.

[0007] A method for target detection and recognition based on dual-mode image fusion, characterized by the following steps:

[0008] Step 1: Use a visible light camera to obtain a visible light image of the target, and use an infrared thermal imager to obtain an infrared image of the target;

[0009] Step 2: Use a dual-mode encoder to extract features of the visible light image and the infrared image, wherein the dual-mode encoder includes a visible light encoder and an infrared encoder. The visible light image is input into the visible light encoder to extract detail features of the visible light image, and the infrared image is input into the infrared encoder to extract detail features of the infrared image.

[0010] Step 3: Use the independent component analysis method to perform independent component analysis on the features of the visible light image and the infrared image extracted in step 2 to obtain independent feature components of the visible light image and the infrared image;

[0011] Step 4: Perform weighted fusion on the independent feature components of the visible light image and the infrared image to obtain the fused features;

[0012] Step 5: Using the fused features as input and the predicted target location and category as output, the Co-DETR deep learning target detection network is trained using a joint loss function;

[0013] The Co-DETR deep learning target detection network is mainly composed of three parts: encoder, decoder, and target detection head; the encoder uses multi-head self-attention operation to extract high-level features containing global context information of input features; the decoder uses a cross-attention mechanism to decode the encoder output features and target query information, and outputs features related to the target; the target detection head maps the features output by the decoder to the location information and category information of the target; the target query information includes the target location and category;

[0014] Step 6: Input the fused features into the trained Co-DETR deep learning target detection network, and output the final predicted target location and category to complete target detection and recognition.

[0015] Specifically, the processing process of the visible light encoder is as follows: input a visible light image, first pass a 1×1 convolution layer to perform channel compression on the image, and obtain a feature map F1; then, input the feature map F1 into the mini residual module for local spatial feature extraction, and obtain a feature map F2; then, input the feature map F2 into the coordinate attention module for cross-channel feature extraction and fusion, and obtain the weighted feature F weight Finally, the weight feature Fweight is input into the reweighting module and multiplied pixel by pixel with the feature map F2 to obtain the final visible light image feature F VS ;

[0016] The mini residual module includes 1 MaxPool layer, 1 1×1 convolution layer and 1 3×3 convolution layer; the specific processing process of the coordinate attention module is as follows: the feature map F2 is input into the coordinate attention module, firstly, the global average pooling feature extraction is performed along the horizontal direction to obtain a one-dimensional vector g X , perform global average pooling feature extraction along the vertical direction to obtain a one-dimensional vector g Y ; Then, vector g X and g Y Splice along the channel dimension and use a 1×1 convolution layer to compress the spliced ​​features to obtain the compressed feature F. compressed ; Next, the feature F compressed Through a BN layer and a nonlinear activation function layer in turn, the activated feature F is obtained activited ; F activited The network is divided into two parts, and the number of channels of each part is adjusted to the same number of channels as the input feature map F2 through two independent 1×1 convolutional layers, and two features F adjust1 and F adjust2 ; Finally, for feature F adjust1 and F adjust2After fusion, the activation function Sigmoid is used for normalization to obtain the weight feature F weight .

[0017] Specifically, the processing process of the infrared encoder is as follows: the infrared image is input, and the channel is first compressed by convolution processing to obtain a feature map F1; then, the feature map F1 is input into the mini residual module for local spatial feature extraction to obtain a feature map F2; then, the feature map F2 is input into the similarity adaptive module to extract different local spatial information to obtain a feature map F3, and the activation function Sigmoid is used to normalize F3 to obtain a weighted feature map F weight ; Finally, the weight feature F weight Input to the reweighting module, multiply it with the feature map F2 pixel by pixel to obtain the final infrared image feature F IR ;

[0018] Among them, the similarity adaptive module calculates the importance of each neuron based on the energy function constructed by the neural network, and generates a three-dimensional attention weight of the feature map, and uses the obtained weight to weight the input feature map to obtain features with local spatial information.

[0019] Specifically, the weighted fusion formula described in step 4 is as follows:

[0020] F fusion =αS VS +βS IR (1)

[0021] Among them, F fusion represents the fused features, S VS Represents the independent characteristic components of the visible light image, S IR It represents the independent characteristic component of the infrared image, α and β are weighting coefficients, satisfying α, β>0 and α+β=1.

[0022] Specifically, the target detection head consists of a bounding box regression branch and a category prediction branch. The bounding box regression branch is used to predict the bounding box of the target. A multi-layer perceptron combined with position encoding is used to regress the input features to obtain bounding box information including position coordinates and size. The category prediction branch is used to predict the category of the target. A multi-layer perceptron and a Softmax activation function are used to classify the input features to obtain the probability value of belonging to each category.

[0023] Furthermore, the joint loss function is expressed as follows:

[0024] l=λ box l box +λ cls l cls (2)

[0025] Among them, l is the joint loss, l box is the L1 loss for bounding box regression, l cls is the cross entropy loss for category prediction, λ box and λ cls is the weight coefficient, satisfying λ box , cls >0 and λ box +λ cls =1.

[0026] Furthermore, position encoding is introduced during bounding box regression to add spatial position information to each pixel in the image.

[0027] The beneficial effects of the present invention are as follows: due to the use of a dual-modal encoder, encoders are designed separately to extract feature information of visible light images and infrared images, which can better extract the detail features of visible light images and the spatial features of infrared images, maximize the exploration of target features in images of different modalities, and help improve the accuracy of final target detection; due to the use of ICA (independent component analysis) feature fusion method, deep fusion of multi-modal features can be achieved, and the complementarity of the two modalities is fully utilized to improve the robustness and accuracy of target detection and recognition; due to the construction of the Co-DETR deep learning target detection network, it is a lightweight, collaborative hybrid task detection model with higher detection speed and better end-to-end learning ability, which can achieve fast and high-precision detection of targets; the present invention has high target detection accuracy and robustness, and takes into account the real-time and robustness of the method, especially in complex environments and low-light conditions, can obtain better target detection effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a target detection and recognition method based on dual-mode image fusion of the present invention;

[0029] Figure 2 is a structural diagram of a dual-mode encoder of the present invention;

[0030] Among them, (a) - schematic diagram of the overall structure of the dual-modal encoder, (b) - schematic diagram of the Mini Residual module, and (c) - schematic diagram of the CoordAtt module. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0032] The present invention provides a target detection and recognition method based on dual-mode image fusion, such as Figure 1 As shown, the specific implementation process is as follows:

[0033] 1. Image data acquisition

[0034] Use a visible light camera to obtain a visible light image X of the target VS , and use the infrared thermal imager to obtain the infrared image X of the same target IR The two images capture different characteristic information of the target, with the visible light image providing the texture and details of the target, and the infrared image capturing the temperature distribution of the target through thermal radiation. Through synchronous acquisition by optoelectronic devices, visible light images and infrared images are finally obtained, providing input data for subsequent image fusion and target detection.

[0035] 2. Use bimodal encoder to extract image features

[0036] Due to the different characteristics of visible light and infrared images and the differences in imaging, different networks are needed to extract the features of each image. The present invention designs a dual-mode encoder to extract the features of each image. The dual-mode encoder consists of a visible light encoder and an infrared encoder. The overall structure of the dual-mode encoder is as follows: Figure 2 (a) shown.

[0037] Among them, the structure and processing process of the visible light encoder are as follows:

[0038] (1) Input visible light image: First, the visible light image X VS Input into a 1×1 convolution layer to compress the image channels and obtain the feature map F1.

[0039] (2) Mini Residual Module: Input F1 into the Mini Residual module to extract local spatial features and obtain feature map F2. Figure 2 As shown in (b), the module includes: a MaxPool layer to downsample the input feature map; a 1×1 convolution layer to perform linear transformation on the channel; and a 3×3 convolution layer to extract local spatial features.

[0040] (3) CoordAtt module: Input the feature map F2 into the CoordAtt module, such as Figure 2 As shown in (c), the processing process of this module is as follows:

[0041] 1) Global average pooling in the horizontal direction (X AvgPool): Perform a global average pooling operation on F2 to extract features in the horizontal direction to obtain a one-dimensional vector g X ;

[0042] 2) Global average pooling in the vertical direction (Y AvgPool): Perform a global average pooling operation on F2 to extract features in the vertical direction to obtain a one-dimensional vector g Y ;

[0043] 3) Feature fusion and compression: g X and g Y Concatenate along the channel dimension to form feature F concat Then, a 1×1 convolutional layer is used to compress the concatenated features and obtain the compressed features F compressed ;

[0044] 4) BN layer and activation layer (BN-Non-Linear): F compressed Through a BN layer and a nonlinear activation function layer in turn, the activated feature F is obtained activited ;

[0045] 5) Feature segmentation and adjustment: F activited It is divided into two parts, and two independent 1×1 convolution layers are used to adjust the number of channels of each part to the same number of channels as the input feature F2, and two features F adjust1 and F adjust2 ;

[0046] 6) Feature fusion and normalization: adjust1 and F adjust2 Perform feature fusion, use the activation function Sigmoid for normalization, and generate the weight feature F weight ;

[0047] (4) Weight assignment: Assign the weight feature F weight Input the Re-weight module and multiply it pixel by pixel with feature F2 to get the final visible light image feature F VS .

[0048] The structure and processing of the infrared encoder are as follows:

[0049] (1) Input infrared image X IR :Infrared encoder uses infrared image X IR As input, the input infrared image is compressed through a 1×1 convolution layer to obtain the convolved feature F1;

[0050] (2) The local spatial features are extracted through the Mini Residual module to obtain the feature map F2;

[0051] (3) Use the similarity adaptive module (SimAM module) to extract different local spatial information in the channel. This module calculates the importance of each neuron based on the energy function constructed by the neural network:

[0052] Energy function It is used to measure the differentiation between neurons. The lower the energy, the greater the difference between the current neuron and the surrounding neurons. The calculation method of the energy function is shown in the following formula:

[0053]

[0054] Among them, λ is a hyperparameter, and t is the single channel input feature of the target neuron. and is the energy function factor, and the specific calculation process is as follows:

[0055]

[0056] Among them, x i is the single channel input feature of the i-th neuron, M = H × W is the number of neurons on the channel, and H and W are the length and width of the channel respectively.

[0057] pass Obtain the weight of each neuron, that is, the three-dimensional attention weight of the feature map, and use the weight to weight the input feature F2 to obtain the feature F3 with local spatial information;

[0058] (4) The feature F3 output by the SimAM module is normalized using the activation function Sigmoid to generate the weight feature F weight ;

[0059] (5) Weight assignment: Assign the weight feature F weight Input the Re-weight module (re-weighted module) and multiply it pixel by pixel with the original feature F2 to obtain the final infrared image feature F IR .

[0060] 3. Feature fusion based on independent component analysis (ICA)

[0061] Independent component analysis (ICA) is a classic statistical technique for multidimensional signal processing, which aims to extract independent source signals from multiple observed signals. It is widely used in feature extraction, blind source separation and other fields. The core idea of ​​ICA is to assume that the input data is a linear mixture of several independent source signals. By separating these independent components, it is possible to remove redundancy and extract deep features of the input data.

[0062] The present invention adopts a feature fusion method based on ICA, which can effectively extract independent feature components from visible light images and infrared images, and realize deep fusion of multimodal features through weighted fusion. This method can make full use of the complementarity of the two modes and improve the robustness and accuracy of target detection and recognition. The specific process is:

[0063] First, the unimodal features F extracted by the visible light encoder and the infrared encoder are VS and F IR As the input signal of ICA, independent component analysis is performed through the separation matrix W to obtain the independent characteristic components S of visible light and infrared images. VS and S IR :

[0064] [S VS ,S IR ]=W[F VS ,F IR ] (6)

[0065] Among them, S VS represents the independent characteristic components of the visible light image after ICA separation, S IR Represents the independent characteristic components of the infrared image after ICA separation.

[0066] Then, the independent feature components are weighted fused to retain the key information of different modalities. The fused features are expressed as:

[0067] F fusion =αS VS +βS IR (7)

[0068] Among them, F fusion represents the fused features; α and β are weighting coefficients, representing the feature weights of visible light and infrared images, satisfying α, β>0 and α+β=1. In low-light or nighttime scenes, infrared images are usually more important, so α<β can be set; in strong-light or daytime scenes, visible light information is dominant, so α>β can be set.

[0069] The weighted fusion process enables the fusion features to retain the detailed features in the visible light image and obtain the temperature radiation features of the target from the infrared image. This feature fusion method can effectively improve the accuracy and robustness of target detection in complex environments.

[0070] Through ICA feature fusion, independent features that do not interfere with each other can be extracted from visible light and infrared images, avoiding the redundancy and conflict of multimodal features. The independent component separation mechanism of ICA allows the features of different modalities to be processed independently, and during the fusion process, the system can flexibly adjust the weights of different modalities to ensure the best detection performance in different scenarios.

[0071] In addition, since ICA can effectively separate non-Gaussian signals, it is particularly suitable for processing image fusion tasks in complex scenes. For example, in bad weather, low light and high interference environments, the ICA-based feature fusion method can retain and amplify key features and improve the robustness of target detection.

[0072] 4. Object detection based on Co-DETR deep learning object detection network

[0073] Co-DETR (Collaborative Detection Transformer) is a deep learning target detection network based on the Transformer architecture. Through collaborative training strategies and combined with multimodal feature input, it can perform end-to-end efficient target detection based on multimodal image fusion. Its core idea is to use the self-attention mechanism in the Transformer model, input the multimodal fused features into the encoder and decoder, extract the global context information of the target, and realize target detection by predicting the location and category of the target.

[0074] (1) Co-DETR Deep Learning Object Detection Network

[0075] The Co-DETR deep learning target detection network mainly consists of three parts: encoder, decoder, and target detection head. The encoder uses multi-head self-attention operation to extract high-level features containing global context information of input features; the decoder uses cross-attention mechanism to decode encoder output features and target query information (target location, category), and outputs features related to the target; the target detection head maps the features output by the decoder to the location information and category information of the target.

[0076] 1) Encoder

[0077] The role of the encoder is to fusion feature F fusion Perform global context modeling and output higher-level feature representation F enc The core of this mechanism is the multi-head self-attention mechanism. fusion Perform multi-head self-attention calculation, expressed as:

[0078]

[0079] In the formula, Q, K, and V are query matrix, key value matrix, and value matrix, respectively. They are expressed by feature F fusion Obtained by linear transformation; d k is the dimension of the key-value matrix; Softmax is used to calculate the attention weights.

[0080] The role of the self-attention mechanism is to assign global weights to the features at each position, thereby capturing the dependencies between features.

[0081] In order to improve the expressiveness of the model, the encoder usually adopts a multi-head attention mechanism to project the input features into multiple different subspaces. The calculation formula of multi-head attention is:

[0082] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (9)

[0083] In the formula, head i =Attention(QW i Q ,KW i K ,VW i V ) is the self-attention calculated for each attention head in each subspace. i Q ,W i K ,W i V is the corresponding linear projection matrix, W O is the output linear projection matrix.

[0084] The attention output is further processed by a feed-forward neural network (FFN) to obtain an updated feature representation F enc FFN usually consists of two fully connected layers and a ReLU activation function.

[0085] 2) Decoder

[0086] The decoder decodes the features according to the target query information and outputs the features related to the target. The target query information is usually embedded in the vector Q query It is represented by and decoded by the cross-attention mechanism. The calculation formula of the cross-attention mechanism is similar to that of self-attention:

[0087]

[0088] Where Q = Q queryis the target query embedding; K, V are the encoder output features F enc Linear transformation of .

[0089] The cross-attention output is processed by the feedforward neural network to finally obtain the decoded feature representation F dec , this feature representation will be used to predict the bounding box and category of the object.

[0090] 3) Target detection head

[0091] Obtain target-related features F through the decoder dec , these features are fused with the encoder output F enc The context information generated by the cross-attention mechanism with the target query vector. These decoder features are the input for further predicting bounding boxes and categories, that is, the task of the target detection head is to map the features output by the decoder into the location information and category information of the target. The target detection head usually consists of a bounding box regression branch and a category prediction branch. The bounding box regression branch is used to predict the bounding box of the target, and the category prediction branch is used to predict the category of the target.

[0092] (a) Multi-layer Perceptron (MLP) for bounding box regression

[0093] The features F generated by the decoder dec Based on this, a multi-layer perceptron (MLP) is used to perform the regression task. The prediction of the bounding box can be expressed by the following formula:

[0094] B=MLP box (F dec ) (11)

[0095] Where B is the bounding box of the final output, including the center point coordinates (x, y) as well as the width w and height h. The specific MLP is usually composed of several fully connected layers, each of which contains the activation function:

[0096]

[0097] In the formula, is the weight matrix in the MLP, is the bias term corresponding to each layer, σ is the activation function, ReLU activation function is used, and the output B is [x, y, w, h], which represents the coordinates and size of the bounding box.

[0098] In order to deal with the bounding box regression problem of targets at different positions in the image, Co-DETR introduces position encoding, which adds spatial position information to each pixel in the image, allowing the model to better handle situations where targets are at different positions:

[0099] F' dec =Fdec +P (13)

[0100] Where P is the position encoding vector, which is encoded based on the position of the target in the image.

[0101] In order to compare with the true bounding box when calculating the loss, the output of the bounding box regression is normalized using the Sigmoid activation function to map the predicted value to between [0,1]:

[0102] B=σ(MLP box (F' dec ))·image_size (14)

[0103] Where σ is the Sigmoid activation function, which is used to limit the output to the interval [0,1], image_size is the size of the image, and MLP box (F' dec ) represents the MLP module processing of bounding box regression.

[0104] (b) Multi-layer Perceptron (MLP) for category prediction

[0105] The task of class prediction is to decode the feature F dec Processing is performed to predict the category of the target. The category prediction task also uses a multi-layer perceptron (MLP), but its output is not the bounding box coordinates, but the score of each category. The formula is as follows:

[0106] C=MLP cls (F dec ) (15)

[0107] Similar to bounding box regression, the MLP for category prediction also contains several fully connected layers:

[0108]

[0109] In the formula, is the weight matrix in the MLP, is the bias term corresponding to each layer, and σ is the activation function.

[0110] The final output of the category prediction is processed through the Softmax activation function, which converts the output value of each category into a probability distribution, representing the likelihood of each category:

[0111] C prob =Softmax(C) (17)

[0112] The formula of the Softmax function is:

[0113]

[0114] In the formula, C[i] represents the original score of category i, K is the number of categories, and C prob [i] is the probability of belonging to class i.

[0115] (2) Loss Function

[0116] In order to optimize the object detection model, a joint loss function is used to train the bounding box regression and category prediction branches. Commonly used loss functions include L1 loss and classification cross entropy loss.

[0117] 1) Bounding Box Loss

[0118] L1 loss is used for bounding box regression, which calculates the difference between the predicted bounding box and the true bounding box. box It is expressed as:

[0119] l box =L1(B pred ,B gt ) (19)

[0120] In the formula, B pred is the bounding box predicted by the model, B gt is the true bounding box, and the L1 loss is used to directly calculate the absolute error between the bounding box prediction and the true value.

[0121] 2) Category prediction loss

[0122] The classification cross entropy loss is used for the loss of category prediction, calculating the difference between the predicted category and the true category. cls It is expressed as:

[0123]

[0124] In the formula, y i is the true category label.

[0125] 3) Total loss

[0126] The total loss function l is expressed as:

[0127] l=λ box l box +λ cls l cls (twenty one)

[0128] In the formula, λ box and λ cls is a weight coefficient used to balance the impact of bounding box and category losses, satisfying λ box ,λ cls >0;λ box +λ cls =1.

[0129] The fused features are used as input and the predicted target location and category are used as output. The Co-DETR deep learning target detection network is trained using a joint loss function. The fused features are input into the trained Co-DETR deep learning target detection network, and the final predicted target location and category are output to complete target detection and recognition.

[0130] The Co-DETR deep learning target detection network can make full use of the multimodal information in the ICA fusion features to improve the accuracy and efficiency of target detection. Experimental results show that the present invention not only improves the accuracy and robustness of target detection, but also takes into account real-time and robustness. The detection accuracy in complex environments reaches 92%, which is 10%-15% higher than that of single-modal detection methods. In low-light and bad weather conditions, the introduction of infrared modality enables the system to better identify targets, significantly improving the detection capability when visible light is insufficient.

Claims

1. A method for target detection and recognition based on dual-mode image fusion, characterized in that Here are the steps: Step 1: Use a visible light camera to obtain a visible light image of the target, and use an infrared thermal imager to obtain an infrared image of the target; Step 2: Use a dual-mode encoder to extract features of the visible light image and the infrared image, wherein the dual-mode encoder includes a visible light encoder and an infrared encoder. The visible light image is input into the visible light encoder to extract detail features of the visible light image, and the infrared image is input into the infrared encoder to extract detail features of the infrared image. Step 3: Use the independent component analysis method to perform independent component analysis on the features of the visible light image and the infrared image extracted in step 2 to obtain independent feature components of the visible light image and the infrared image; Step 4: Perform weighted fusion on the independent feature components of the visible light image and the infrared image to obtain the fused features; Step 5: Using the fused features as input and the predicted target location and category as output, the Co-DETR deep learning target detection network is trained using a joint loss function; The Co-DETR deep learning target detection network is mainly composed of three parts: encoder, decoder, and target detection head; the encoder uses multi-head self-attention operation to extract high-level features containing global context information of input features; the decoder uses a cross-attention mechanism to decode the encoder output features and target query information, and outputs features related to the target; the target detection head maps the features output by the decoder to the location information and category information of the target; the target query information includes the target location and category; Step 6: Input the fused features into the trained Co-DETR deep learning target detection network, and output the final predicted target location and category to complete target detection and recognition.

2. The target detection and recognition method based on dual-mode image fusion according to claim 1, characterized in that: The processing process of the visible light encoder is as follows: input a visible light image, first pass a 1×1 convolution layer to perform channel compression on the image, and obtain a feature map F1; then, input the feature map F1 into the mini residual module for local spatial feature extraction, and obtain a feature map F2; then, input the feature map F2 into the coordinate attention module for cross-channel feature extraction and fusion, and obtain the weighted feature F weight Finally, the weight feature Fweight is input into the reweighting module and multiplied pixel by pixel with the feature map F2 to obtain the final visible light image feature F VS ; The mini residual module includes 1 MaxPool layer, 1 1×1 convolution layer and 1 3×3 convolution layer; the specific processing process of the coordinate attention module is as follows: the feature map F2 is input into the coordinate attention module, firstly, the global average pooling feature extraction is performed along the horizontal direction to obtain a one-dimensional vector g X , perform global average pooling feature extraction along the vertical direction to obtain a one-dimensional vector g Y ; Then, vector g X and g Y Splice along the channel dimension and use a 1×1 convolution layer to compress the spliced ​​features to obtain the compressed feature F. compressed ; Next, the feature F compressed Through a BN layer and a nonlinear activation function layer in turn, the activated feature F is obtained activited ; F activited The network is divided into two parts, and the number of channels of each part is adjusted to the same number of channels as the input feature map F2 through two independent 1×1 convolutional layers, and two features F adjust1 and F adjust2 ; Finally, for feature F adjust1 and F adjust2 After fusion, the activation function Sigmoid is used for normalization to obtain the weight feature F weight .

3. The target detection and recognition method based on dual-mode image fusion according to claim 1, characterized in that: The processing process of the infrared encoder is as follows: input the infrared image, first perform channel compression through convolution processing, and obtain a feature map F1; then, input the feature map F1 into the mini residual module for local spatial feature extraction, and obtain a feature map F2; then, input the feature map F2 into the similarity adaptive module to extract different local spatial information, and obtain a feature map F3, and use the activation function Sigmoid to normalize F3 to obtain a weighted feature map F weight ; Finally, the weight feature F weight Input to the reweighting module, multiply it with the feature map F2 pixel by pixel to obtain the final infrared image feature F IR ; Among them, the similarity adaptive module calculates the importance of each neuron based on the energy function constructed by the neural network, and generates a three-dimensional attention weight of the feature map, and uses the obtained weight to weight the input feature map to obtain features with local spatial information.

4. The target detection and recognition method based on dual-mode image fusion according to claim 1, characterized in that: The weighted fusion formula described in step 4 is as follows: F fusion =αS VS +βS IR (1) Among them, F fusion represents the fused features, S VS Represents the independent characteristic components of the visible light image, S IR It represents the independent characteristic component of the infrared image, α and β are weighting coefficients, satisfying α, β>0 and α+β=1.

5. The target detection and recognition method based on dual-mode image fusion according to claim 1, characterized in that: The target detection head consists of a bounding box regression branch and a category prediction branch. The bounding box regression branch is used to predict the bounding box of the target. A multi-layer perceptron combined with position encoding is used to perform regression processing on the input features to obtain bounding box information including position coordinates and size, and a Sigmoid function is used for normalization processing. The category prediction branch is used to predict the category of the target. A multi-layer perceptron and a Softmax activation function are used to classify the input features to obtain the probability value belonging to each category.

6. The target detection and recognition method based on dual-mode image fusion according to claim 1 or 5, characterized in that: The joint loss function expression is as follows: in, For joint losses, is the L1 loss for bounding box regression, is the cross entropy loss for category prediction, λ box and λ cls is the weight coefficient, satisfying λ box , cls >0 and λ box +λ cls =1.

7. The method for target detection and recognition based on dual-mode image fusion according to claim 5, characterized in that: Position encoding is introduced during bounding box regression to add spatial position information to each pixel in the image.

Images