An image fusion detection method based on double-flow attention guidance and related devices

Abstract

The application belongs to the technical field of target detection in computer vision, and discloses a kind of image fusion detection method and related device based on double-flow attention guide;Among them, the image fusion detection method based on double-flow attention guide includes the following steps: obtaining the visible light image and infrared image of the target detection area to be acquired;Based on the visible light image and infrared image of the target detection area to be acquired, a pre-trained deep target detection network is used for target detection, and a target detection result is obtained;Wherein, the deep target detection network includes a feature extraction module, a double-flow attention guide module and a detection module.The technical scheme disclosed by the application can solve the technical problems of poor precision and low practicality caused by insufficient guidance of visible light and infrared image fusion on features in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology in computer vision, and specifically relates to an image fusion detection method and related apparatus based on dual-stream attention guidance. Background Technology

[0002] Visible light and infrared image fusion detection refers to a technique that uses modal fusion to fuse visible light and infrared images of a target (exemplarily, ships such as aircraft carriers, destroyers, and cruisers), locates it using bounding boxes, and classifies the target. Further illustratively, marine scenes are often complex, with multiple environmental interferences (exemplarily, fish scale light interference, towed decoys, chaff interference, deception interference, and suppression interference). Visible light images are difficult to distinguish in such environments, making target locations deceptive. Infrared images often lack clear target outlines, limiting the effective information available for extraction and impacting the location and detection of maritime targets. Additionally, the small size of maritime vessels (exemplarily, when the image is scaled to a 640-pixel square, the target's pixels are less than 16×16) makes them easily confused with other maritime targets (exemplarily, islands, lighthouses, and other common marine targets). Furthermore, the structure of maritime targets is often indistinct, with similar outlines and smaller pixel sizes compared to other targets, making them less noticeable in the visible light modality and difficult to distinguish in the infrared modality. In conclusion, the fusion detection of visible light and infrared images at sea is an extremely challenging task.

[0003] Currently, existing visible light and infrared image fusion detection methods can be divided into two categories based on their technical processes and development stages: traditional visible light and infrared image fusion detection and deep learning-based visible light and infrared image fusion detection. Traditional visible light and infrared image fusion detection primarily relies on manual extraction of low-level features (such as edges, colors, and textures), manual fusion of different modalities, and manual detection combined with machine learning classification and regression algorithms. This approach depends on a large amount of prior knowledge with significant interference, requires substantial manual intervention, and suffers from low reliability, accuracy, and efficiency, as well as high labor costs. Furthermore, the fusion process depends on manual alignment, and as the types of targets become more refined and numerous, the effort required for fusion and annotation increases dramatically, making it unsuitable for large-scale production lines. Deep learning-based visible light and infrared image fusion detection, on the other hand, uses deep target detection neural networks to complete feature extraction, feature fusion, defect localization, and classification, achieving end-to-end automated detection with high detection accuracy, efficiency, and reliability.

[0004] Existing deep learning-based methods for visible light and infrared image fusion detection primarily rely on Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). CNNs utilize the translation invariance of images as prior knowledge, aggregating pixel information layer by layer through convolution operations to effectively learn visible light and infrared image features. While these existing approaches offer high detection accuracy, efficiency, and reliability, they are ill-suited for environments with complex interference. Directly using convolutional methods cannot effectively distinguish blurred contours and similar targets in both modalities, cannot adequately fit the differences between visible light and infrared images, and cannot avoid feature confusion during the fusion process. Therefore, there is an urgent need to establish an effective visual attention fusion module to enhance the attention to different features of visible light and infrared images, thereby improving the mean average precision (mAP) of visible light and infrared image fusion detection. Visual Transformers model long-range dependencies based on Multi-Head Self-Attention (MHSA) mechanisms. In existing multi-stage visual Transformer classification tasks, feature maps for detection are often output through stage-by-stage downsampling. However, MHSA computation often directly computes the pre-attention matrix of query, key, and value formed by linear mapping of feature maps, without considering the attention differences between different modalities, thus hindering feature extraction from visible light and infrared images. Furthermore, the pre-attention matrix of query, key, and value formed by linear mapping of feature maps does not pay attention to guidance between different modalities, but only operates within its own modality, thus hindering feature fusion and detection of visible light and infrared images. Therefore, there is an urgent need to propose a cross-attention guidance mechanism that considers internal information enhancement and inter-feature information interaction to improve the overall average accuracy of visual Transformers in the fusion detection of visible light and infrared images. Summary of the Invention

[0005] The purpose of this invention is to provide an image fusion detection method and related apparatus based on dual-stream attention guidance to solve one or more of the aforementioned technical problems. Specifically, this invention provides a visible light and infrared image fusion detection scheme based on dual-stream attention guidance, which can solve the technical problems of poor accuracy and low practicality caused by insufficient feature guidance in visible light and infrared image fusion in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an image fusion detection method based on dual-stream attention guidance, comprising the following steps:

[0008] Acquire visible light and infrared images of the target detection area;

[0009] Based on the acquired visible light and infrared images of the target detection area, a pre-trained deep target detection network is used to perform target detection and obtain the target detection results.

[0010] The deep target detection network includes:

[0011] The feature extraction module is used to take visible light images and infrared images as input, extract features, and output feature maps of the visible light mode and the infrared mode.

[0012] The dual-stream attention guidance module is used to input feature maps of the visible light mode and the infrared mode, perform feature fusion, and output the fused feature map.

[0013] The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modality fusion; wherein, the target detection result includes the target anchor box position and the target classification result.

[0014] A further improvement of the present invention is that,

[0015] The feature extraction module includes N feature downsampling stages of dual streams, and outputs N pairs of visible light image feature maps and infrared image feature maps;

[0016] In each feature downsampling stage, the visible light image and the infrared image are downsampled by a factor of 2, and features are extracted through convolution operations, resulting in a single layer of feature map.

[0017] A further improvement of the present invention is that,

[0018] In the detection module, the feature map after the fusion of visible light and infrared light is used. After N upsampling and layer-by-layer downsampling, the enhancement and detection of the two modes of visible light and infrared light are completed through the feature pyramid structure, so as to obtain the target detection result after mode fusion.

[0019] A further improvement of the present invention is that,

[0020] The dual-stream attention guidance module includes: a modal self-attention enhancement module, a modal cross-attention guidance module, and a modal adaptive dynamic adjustment module; wherein...

[0021] The modal self-attention enhancement module includes: a feature normalization module, a multi-head self-attention module, and a skip connection layer; wherein, the feature normalization module is used to input the feature maps of the visible light mode and the infrared mode and perform a normalization operation, and output the normalized feature map; the multi-head self-attention module is used to input the normalized feature map and form a query Q, key K, and value V through linear mapping to obtain attention weights; the skip connection layer is used to input the feature maps of the visible light mode and the infrared mode and add them to the obtained attention weights, and output the enhanced feature maps of the visible light mode and the infrared mode;

[0022] The modal cross-attention guidance module includes a cross-attention module and a modal hybrid connection module. The cross-attention module is used to input the enhanced feature maps of the visible light mode and the infrared mode, and after different combinations of queries Q, keys K, and values ​​V, obtain the attention weights after cross-guidance. The modal hybrid connection module is used to input the enhanced feature maps of the visible light mode and the infrared mode, add them to the obtained attention weights after cross-guidance, and output the feature maps of the visible light mode and the infrared mode after modal interaction.

[0023] The modality adaptive dynamic adjustment module includes: a modality channel and spatial aggregation module and a dynamic convolution size adjustment module; wherein, the modality channel and spatial aggregation module is used to input the feature maps of the visible light modality and the infrared modality after modal interaction and calculate the spatial attention weights and channel attention weights, and multiply the obtained spatial attention weights and channel attention weights with the initial convolution kernel to obtain the optimal convolution kernel weights; the dynamic convolution size adjustment module is used to input the obtained optimal convolution kernel weights, and perform convolution operations on the feature maps of the visible light modality and the infrared modality after modal interaction based on the optimal convolution kernel weights to obtain the fused feature map.

[0024] A further improvement of the present invention is that,

[0025] The modal self-attention enhancement module is represented as follows:

[0026]

[0027] Q(X rgb )~U(-r,r),Q(X ir )~U(-r,r);

[0028] K(X rgb )=Conv 1×1 (X rgb ),V(X rgb )=Conv 1×1 (X rgb );

[0029] K(X ir )=Conv 1×1 (X ir ),V(X ir )=Conv 1×1 (X ir );

[0030] In the formula, X rgb is the input feature map of the visual attention module for the visible light modality; softmax(·) is the exponential normalization function; T For matrix transpose; Q(X) rgb ), K(X) rgb ), V(X) rgb ) represent the query, key, and value matrices respectively in the attention encoding of the visible light modality; d is the embedding dimension; X ir The input feature map for the visual attention module in the infrared modality; Q(X) ir ), K(X) ir ), V(X) ir ) represent the query, key, and value matrices respectively in the attention encoding of the infrared modality; U(-r,r) is a uniform distribution with parameter r; Conv 1×1 (·) represents a convolution operation with a kernel size of 1×1.

[0031] A further improvement of the present invention is that,

[0032] In the modal cross-attention guidance module, during the steps of querying Q, key K, and value V after different combinations, the query Q(X) of visible light and infrared attention is... rgb ) and Q(X) ir ) are applied to the K(X) bond in infrared and visible light respectively. ir ) and K(X rgb In ), it is represented as:

[0033]

[0034] In the formula, MLP(·) represents a multilayer perceptron; The visible light characteristics after fusion; It is the fused infrared signature.

[0035] A further improvement of the present invention is that,

[0036] The loss function used in the training process of the deep object detection network is expressed as follows:

[0037]

[0038] In the formula, To predict the loss difference between the anchor frame and the actual anchor frame; The loss is determined by whether the target category is correct or not; Whether or not the loss is for the target; λ is the dynamic convolution kernel loss; λ is a hyperparameter used for balancing.

[0039] in,

[0040]

[0041] S rgb|ir =sign(K) rgb -K ir );

[0042] S ir|rgb =sign(K) ir -K rgb );

[0043] In the formula, μ is a preset hyperparameter used to balance the two losses; D js (·) represents the JS divergence between two features; K rgb K represents the kernel of a visible light image. ir Represents the kernel of an infrared image; S rgb|ir and S ir|rgb These represent the unique feature information in visible light images and infrared images, respectively; sign(·) is the sign function.

[0044] A second aspect of the present invention provides an image fusion detection system based on dual-stream attention guidance, comprising:

[0045] The image acquisition module is used to acquire visible light and infrared images of the area to be detected.

[0046] The target detection module is used to perform target detection based on the acquired visible light and infrared images of the target detection area, using a pre-trained deep target detection network, and obtain the target detection results.

[0047] The deep target detection network includes:

[0048] The feature extraction module is used to take visible light images and infrared images as input, extract features, and output feature maps of the visible light mode and the infrared mode.

[0049] The dual-stream attention guidance module is used to input feature maps of the visible light mode and the infrared mode, perform feature fusion, and output the fused feature map.

[0050] The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modality fusion; wherein, the target detection result includes the target anchor box position and the target classification result.

[0051] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the image fusion detection method based on dual-stream attention guidance as described in any one of the first aspects of the present invention.

[0052] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image fusion detection method based on dual-stream attention guidance as described in any one of the first aspects of the present invention.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention provides a dual-stream attention-guided visible light and infrared image fusion detection method that uses a pre-trained deep target detection network for target detection. This solves the technical problems of poor accuracy and low practicality caused by insufficient feature guidance in visible light and infrared image fusion in existing technologies. Explained, the deep target detection network includes a feature extraction module, a dual-stream attention-guided module, and a detection module. The dual-stream attention-guided module introduces an attention mechanism, enabling the network to focus on important regions and features in the image, thereby enhancing the capabilities of the feature extraction module. This helps to extract key information from visible light and infrared images more accurately, providing stronger support for subsequent target detection tasks. During image fusion, the dual-stream attention-guided module guides the effective fusion of features from visible light and infrared images. By focusing on the similarities and differences between the two, this module can generate a fused image containing richer and more accurate information, thereby improving the accuracy and practicality of target detection. The dual-stream attention-guided module makes the entire deep target detection network more adaptable to environmental factors such as lighting and weather. Even in complex and changing scenes, the network can work stably and provide reliable target detection results. In summary, this invention utilizes the characteristic of attention to explore local features, strengthens the information connection between visible light and infrared modes, and introduces a cross-attention guidance mechanism into modal learning, which can effectively improve the average accuracy of all types of maritime detection.

[0055] In a preferred embodiment of the present invention, the dual-stream attention guidance module includes a modal self-attention enhancement module, a modal cross-attention guidance module, and a modal adaptive dynamic adjustment module. The modal self-attention enhancement module and the modal cross-attention guidance module promote effective information fusion and background information suppression between the two modalities. The modal adaptive dynamic adjustment module introduces dynamic convolutional kernel loss to fit the fusion features of the visible light and infrared modalities, simultaneously strengthening the feature information within each modality and the fusion effect between modalities. This effectively improves the detection performance of multimodal targets in complex environments and increases the overall average accuracy of target detection. In summary, the method of the present invention, based on the calculation between different attention levels and using dynamic convolutional kernels to enable the neural network to learn more effective modal interaction features, can effectively improve the detection accuracy of visible light and infrared images of targets. Each module is highly independent, without complex coupling relationships, and has strong practicality. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating an image fusion detection method based on dual-stream attention guidance, as described in an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the structure of the deep target detection network in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the modal self-attention enhancement module in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the modal cross-attention guidance module in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the modal adaptive dynamic adjustment module in an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram illustrating the detection results of visible light and infrared targets at sea in a specific embodiment of the present invention; wherein, Figure 6 (a) is a schematic diagram of the detection results of maritime targets under visible light conditions. Figure 6 (b) is a schematic diagram of the detection results of maritime targets under infrared conditions;

[0063] Figure 7This is a schematic diagram of an image fusion detection system based on dual-stream attention guidance in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0065] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0066] Please see Figure 1 This invention provides an image fusion detection method based on dual-stream attention guidance, specifically a method for fusion detection of visible light and infrared images at sea based on dual-stream attention guidance, comprising the following steps:

[0067] Step 1: Acquire the visible light and infrared images of the sea to be detected; interpretably, the visible light and infrared images of the sea to be detected can be modally aligned corresponding images or misaligned modal images, and can contain interference (fish scale light interference, towed decoys, chaff interference, deception interference, suppression interference, etc.) or pairs of images without interference.

[0068] Step 2: Based on the visible light and infrared images of the sea to be detected, a pre-trained depth image detection network is used to locate and classify the sea target to obtain the detection results; interpretably, the detection results are output as the bounding box of the target and the category probability.

[0069] Please see Figure 2 In this embodiment of the invention, the deep object detection network includes a feature extraction module, a two-stream attention guidance module, and a detection module connected in sequence; wherein,

[0070] The feature extraction module is used to extract features from visible light and infrared images and output corresponding feature maps for the two modalities; wherein, the feature extraction module uses a convolutional neural network as the basic visual feature calculation module;

[0071] The dual-stream attention guidance module is used to input the different modalities and output the fusion result. The dual-stream attention guidance module includes a modal self-attention enhancement module, a modal cross-attention guidance module, and a modal adaptive dynamic adjustment module. The modal self-attention enhancement module includes a feature normalization module, a multi-head self-attention module, and a skip connection layer. The feature normalization module is used to input the feature maps of the visible light modality and the infrared modality, perform normalization operations, and output the normalized feature map. The multi-head self-attention module is used to input the normalized feature map and perform linear mapping to form a query Q, a key K, and a value V, obtaining attention weights. The skip connection layer is used to input the feature maps of the visible light modality and the infrared modality, add them to the obtained attention weights, and output the enhanced feature maps of the visible light modality and the infrared modality. Further explanation: the skip connection layer is a normalization layer that adds the input visible light and infrared features to the obtained attention.

[0072] The modal cross-attention guidance module includes a cross-attention module and a modal hybrid connection module. The cross-attention module is used to input the enhanced feature maps of the visible light mode and the infrared mode, and after different combinations of queries Q, keys K, and values ​​V, obtain the attention weights after cross-guidance. The modal hybrid connection module is used to input the enhanced feature maps of the visible light mode and the infrared mode, add them to the obtained attention weights after cross-guidance, and output the feature maps of the visible light mode and the infrared mode after modal interaction.

[0073] The modality adaptive dynamic adjustment module includes: a modality channel and spatial aggregation module and a dynamic convolution size adjustment module; wherein, the modality channel and spatial aggregation module is used to input the feature maps of the visible light modality and the infrared modality after modal interaction and calculate the spatial attention weights and channel attention weights, and multiply the obtained spatial attention weights and channel attention weights with the initial convolution kernel to obtain the optimal convolution kernel weights; the dynamic convolution size adjustment module is used to input the obtained optimal convolution kernel weights, and perform convolution operations on the feature maps of the visible light modality and the infrared modality after modal interaction based on the optimal convolution kernel weights to obtain the fused feature map;

[0074] The detection module is used to input the feature map after feature fusion output by the multimodal feature fusion module, and output the detection results after fusion of different modes; wherein, the detection results include the anchor frame position and the classification results of the types of ships at sea.

[0075] In a specific embodiment of this invention, a method for detecting visible and infrared targets at sea based on dual-stream attention guidance is proposed. This method is used to extract information from visible and infrared modalities and accurately locate their positions under complex environmental interference. A dual-stream attention guidance module is designed to simultaneously consider and utilize the differences and shared effective information between visible and infrared modalities. Addressing the information loss problem caused by the failure of existing visible and infrared image fusion methods to fully consider the modal enhancement effect of self-attention and the interactive guidance effect of cross-attention on information from different modalities, this invention effectively improves the overall average accuracy of detecting dual-modal targets at sea under complex interference. Furthermore, the proposed dual-stream attention guidance module and modal adaptive adjustment module are both based on convolution and matrix operations, with consistent image naming and data format to avoid complex dependencies. These modules can be added to various depth target detection models and applied to different datasets, demonstrating strong generalization ability and broad application prospects.

[0076] Please see Figures 2 to 5 In a specific embodiment of the present invention, the specific process for fusion detection of visible light and infrared images at sea is performed according to the following steps, including:

[0077] Step 1: Obtain and construct a dataset of visible light and infrared images at sea, where the visible light and infrared images are 1280×1064 pixels, and the corresponding modalities in the dataset are named with the same name;

[0078] Step 2: Construct a deep object detection network, which includes a feature extraction module, a two-stream attention guidance module, and a detection module connected in sequence.

[0079] like Figure 2 As shown, the feature extraction module includes five feature downsampling stages in the dual-stream; each stage downsamples the visible light and infrared images by a factor of 2, extracts features through convolution operations, and outputs a layer of feature maps; a total of five pairs of visible light and infrared image feature maps are obtained, and the paired features are input into the dual-stream attention guidance module.

[0080] The dual-stream attention guidance module includes a modal self-attention enhancement module, a modal cross-attention guidance module, and a modal adaptive dynamic adjustment module; among them,

[0081] like Figure 3 As shown, the modal self-attention enhancement module enhances the interpretability and saliency of the target through modal self-attention modules within visible light and infrared images. The modal self-attention module is represented as follows:

[0082]

[0083] Q(X rgb)~U(-r,r),Q(X ir )~U(-r,r);

[0084] K(X rgb )=Conv 1×1 (X rgb ),V(X rgb )=Conv 1×1 (X rgb );

[0085] K(X ir )=Conv 1×1 (X ir ),V(X ir )=Conv 1×1 (X ir );

[0086] In the formula, X rgb X is the input feature map for the visual attention module in the visible light modality. ir For the input feature map of the visual attention module in the infrared modality, Conv 1×1 (·) represents a convolution operation with a kernel size of 1×1. For matrix transpose, softmax(·) is the exponential normalization function used to normalize features to an interval, Q(X rgb ), K(X) rgb ), V(X) rgb ) represent the query, key, and value matrices corresponding to the attention encoding of the visible light mode, respectively, Q(X ir ), K(X) ir ), V(X) ir ) represent the query, key, and value matrices corresponding to the attention encoding of the infrared modality, respectively, d = h × w is the embedding dimension, h and w are the height and width of the visible light and infrared images, respectively, and U(-r,r) is a uniform distribution with parameter r;

[0087] The modal self-attention enhancement module takes the visible light and infrared images obtained from the feature extraction module as inputs, passes them through the self-attention module, and then uses a skip connection to connect the input with the attention result. The enhanced modal features are then input into the modal cross-attention guidance module.

[0088] like Figure 4 As shown, in the modal cross-attention guidance module, the query Q(X) for visible light and infrared attention... rgb ) and Q(X) ir ) are applied to the K(X) bond in infrared and visible light respectively. ir ) and K(X rgb )middle:

[0089]

[0090]

[0091] In the formula, MLP(·) represents a multilayer perceptron. The visible light characteristics after fusion It is the fused infrared signature;

[0092] Explained, the cross-attention guidance module exchanges queries from visible light and infrared images, enabling the visible light and infrared images to adjust the feature points of the corresponding modalities based on the effective information of the target in their own images, thereby enhancing the interaction and fusion of different modalities. Then, the guided features are used by the modality adaptive dynamic adjustment module to find the differences and saliences between modalities.

[0093] like Figure 5 As shown, the modal adaptive dynamic adjustment module receives visible light and infrared input features after modal interaction. It has a height H, a width W, and a channel C. in Along V convolutional kernels K1, K2, ... K V The adaptive dynamic adjustment module constructs two feature attention weights, including spatial attention weights. Channel attention weight Perform the following calculation:

[0094] α t ∈φ t (X);

[0095] Where t∈{s,c}, φ represents the sequential operation of the Global Average Pooling (GAP) layer, fully connected layers (FC), Rectified Linear Unit (ReLU) activation function, and the head branch; then, the weights are applied to V convolutional kernels to obtain the aggregated optimal dynamic convolutional kernel K, which is calculated as follows:

[0096]

[0097] in, Represents α t The v-th element in the array; ⊙ represents element-wise multiplication;

[0098] Then, the optimal convolution kernel is used to perform convolution operations with the visible light and infrared input features after modal interaction to obtain the fused feature map.

[0099] Step 3: Data preprocessing. Before being input into the detection network, the visible light and infrared images are uniformly scaled to 640×640 pixels and pixel normalization is performed. For image enhancement, a 50% horizontal and vertical image flip is performed, and an image mask with a probability of 0.4 is added.

[0100] Step 4: Training process. The visible light and infrared images are first downsampled 5 times to obtain feature maps of 5 stages. The feature maps of the last three stages are then input into the dual-stream attention guidance module. The output dual-stream features are then input into the pyramid neural network for feature aggregation. Finally, the features are input into the detection head to obtain the target localization and classification results.

[0101] The neural network training is optimized based on the difference between the output and the true value. Jensen-Shannon (JS) divergence is used to measure the information difference between visible light and infrared features. JS divergence assesses the similarity between two distributions, approaching zero when they are similar and reaching 1 when they are dissimilar. The loss is convergently trained by minimizing the distribution difference, and its calculation is as follows:

[0102]

[0103] Among them, D js (·) represents the JS divergence between two features, K rgb K represents the kernel of a visible light image. ir The kernel representing the infrared image.

[0104] At the same time, the unique feature information in visible and infrared images is better enhanced by calculating the difference between the two kernels, denoted as S. rgb|ir and S ir|rgb This corresponds to the feature differences in visible light and infrared images, and is calculated as follows:

[0105] S rgb|ir =sign(K) rgb -K ir ),S ir|rgb =sign(K) ir -K rgb );

[0106] The sign(·) function sets all negative values ​​to 0, ensuring that only significant values ​​are calculated.

[0107] Next, the dynamic convolution kernel loss... It was ultimately set as follows:

[0108]

[0109] μ is set to 0.4.

[0110] Another example is that, during each training step, backpropagation is performed starting from the loss function value; a stochastic gradient descent (SGD) optimizer is used to automatically calculate the loss and update the network model parameters through the framework, enabling the network model to learn the best target features step by step.

[0111] In this embodiment of the invention, the loss function used in the training process of the deep object detection network is expressed as follows:

[0112]

[0113] In the formula, To predict the loss difference between the anchor frame and the actual anchor frame; The loss is determined by whether the target category is correct or not; Whether or not the loss is for the target; λ is the dynamic convolution kernel loss; λ is a hyperparameter used for balancing.

[0114] In a specific embodiment of this invention: the marine visible light and infrared image dataset used contains 2484 dual-modal images captured by visible light and infrared cameras across 20 categories, with 1242 images for each of the visible light and infrared modes. The image size is 1280*1068. These images are randomly divided into a training set of 2236 visible light and infrared image pairs and a test set of 248 visible light and infrared image pairs. The images are scaled to 640×640 pixels during the preprocessing stage. In this embodiment of the invention, the initialization method for the detection network parameters is as follows: the parameters of the feature extraction module are extracted from a model pre-trained on a public dataset, while the other network parameters are initialized randomly. In this embodiment of the invention, the training time on a Gold 6626R@2.90GHz CPU, 8GB of memory, and an NVIDIA GeForce RTX3090 GPU is approximately 6 hours.

[0115] In this embodiment of the invention, the specific implementation steps include: first, setting relevant training parameters, setting the optimizer used for network updates in the technical solution of this invention as a stochastic gradient descent optimizer, with an initial learning rate of 0.001, weight decay of 0.0005, and momentum of 0.937. The deep target detection network of this embodiment consists of a feature extraction module with 5 stages, 3 dual-stream attention-guided modules, and 3 detection heads. The feature extraction module contains 5 downsampling networks; the network input is pairs of visible light and infrared images. First, the feature extraction module extracts features from the paired images, then the features from the latter three stages are input into the dual-stream attention-guided modules to fuse the features, and finally input into the detection head to obtain the location and category results of the maritime target. Next, when training the network using the divided dataset, 32 images are selected from the training set and input into the network each time. The selected stochastic gradient descent optimizer is used to update the parameters, and training is completed after 100 iterations on the dataset. Finally, the images from the test set are input into the network for detection to obtain the results of the embodiment of the invention, such as... Figure 6 As shown, Figure 6 Figure (a) shows the detection performance under visible light conditions. Figure 6 Figure (b) shows the detection effect under infrared conditions.

[0116] In summary, this invention discloses a method for marine visible and infrared image fusion detection based on dual-stream attention guidance. This method leverages the ability of attention to explore local features, strengthening the information connection between visible and infrared modalities. Through cross-attention guidance of difference analysis, the network can effectively learn the prominent features of different modalities and overcome the complex interference of the marine environment, achieving robust detection of marine targets. This invention introduces a cross-attention guidance mechanism into the field of multimodal detection of marine targets, effectively improving the overall average accuracy of marine detection. The modules are highly independent, simple to implement, and have a moderate number of parameters, demonstrating strong applicability and progressive potential.

[0117] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0118] Please see Figure 7 In this embodiment of the invention, an image fusion detection system based on dual-stream attention guidance is provided, comprising:

[0119] The image acquisition module is used to acquire visible light and infrared images of the area to be detected.

[0120] The target detection module is used to perform target detection based on the acquired visible light and infrared images of the target detection area, using a pre-trained deep target detection network, and obtain the target detection results.

[0121] The deep target detection network includes:

[0122] The feature extraction module is used to take visible light images and infrared images as input, extract features, and output feature maps of the visible light mode and the infrared mode.

[0123] The dual-stream attention guidance module is used to input feature maps of the visible light mode and the infrared mode, perform feature fusion, and output the fused feature map.

[0124] The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modality fusion; wherein, the target detection result includes the target anchor box position and the target classification result.

[0125] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to execute the operation of an image fusion detection method based on dual-stream attention guidance.

[0126] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the image fusion detection method based on dual-stream attention guidance in the above embodiments.

[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for image fusion detection based on dual-stream attention guidance, characterized in that, The method comprises the following steps: obtaining a visible light image and an infrared image of a target detection region; based on the obtained visible light image and infrared image of the target detection region, using a pre-trained deep target detection network to perform target detection, and obtaining a target detection result; wherein the deep target detection network comprises: a feature extraction module for inputting the visible light image and the infrared image and performing feature extraction, outputting a feature map of the visible light mode and a feature map of the infrared mode; a dual-flow attention guide module for inputting the feature map of the visible light mode and the feature map of the infrared mode and performing feature fusion, outputting a fused feature map; a detection module for inputting the fused feature map and performing target detection, outputting a target detection result after mode fusion; wherein the target detection result comprises a target anchor box position and a target classification result; the dual-flow attention guide module comprises a modal self-attention enhancement module, a modal cross-attention guide module and a modal adaptive dynamic adjustment module; wherein The modal self-attention enhancement module comprises a feature normalization module, a multi-head self-attention module and a skip connection layer; wherein the feature normalization module is used for inputting the feature map of the visible light mode and the feature map of the infrared mode and performing a normalization operation to output a normalized feature map; the multi-head self-attention module is used for inputting the normalized feature map and performing linear mapping to form queries , keys , values to obtain attention weights; and the skip connection layer is used for inputting the feature map of the visible light mode and the feature map of the infrared mode and adding the obtained attention weights to output the enhanced feature map of the visible light mode and the feature map of the infrared mode. The modal cross-attention guiding module comprises a cross-attention module and a modal mixed connection module; wherein the cross-attention module is used for inputting the enhanced feature map of the visible light modality and the feature map of the infrared modality and performing different combinations on the query , key , value to obtain cross-guided attention weights; the modal mixed connection module is used for inputting the enhanced feature map of the visible light modality and the feature map of the infrared modality and adding the obtained cross-guided attention weights, and outputting the feature map of the visible light modality and the feature map of the infrared modality after modal interaction; the modal adaptive dynamic adjustment module comprises a modal channel and spatial aggregation module and a dynamic convolution size adjustment module; wherein the modal channel and spatial aggregation module is used for inputting the visible light mode feature map and the infrared mode feature map after modal interaction and calculating to obtain spatial attention weights and channel attention weights, multiplying the obtained spatial attention weights and channel attention weights with the initial convolution kernel to obtain the best convolution kernel weight; the dynamic convolution size adjustment module is used for inputting the obtained best convolution kernel weight, and based on the best convolution kernel weight, performing convolution operation on the visible light mode feature map and the infrared mode feature map after modal interaction to obtain the fused feature map.

2. The image fusion detection method based on dual-flow attention guide according to claim 1, wherein The feature extraction module comprises two streams of N feature down-sampling stages, outputting N a pair of visible light image feature maps and infrared image feature maps; each feature down-sampling stage performs 2 times down-sampling on the visible light image and the infrared image, performs feature extraction through convolution operation, and outputs a layer of feature map.

3. The image fusion detection method based on dual-flow attention guide according to claim 2, wherein In the detection module, a feature map fused by visible light and infrared is used, and through upsampling and layer-by-layer downsampling, a feature pyramid structure is used to complete the enhancement and detection of visible light and infrared modalities, and a target detection result after modality fusion is obtained. N In the detection module, a feature map fused by visible light and infrared is used, and through upsampling and layer-by-layer downsampling, a feature pyramid structure is used to complete the enhancement and detection of visible light and infrared modalities, and a target detection result after modality fusion is obtained.

4. The image fusion detection method based on dual-flow attention guide according to claim 1, wherein the modal self-attention enhancement module is represented as: ； ； ； ； ； wherein, is the input feature map of the visual attention module of the visible light modality; is an exponential normalization function; is a matrix transpose operation; , , respectively represent the corresponding query, key, value matrices in the attention encoding of the visible light modality; is the embedding dimension; is the input feature map of the visual attention module of the infrared modality; , , respectively represent the corresponding query, key, value matrices in the attention encoding of the infrared modality; is a uniform distribution with parameters ; is a convolution operation with a convolution kernel size of .

5. The image fusion detection method based on dual-flow attention guide according to claim 4, wherein In the modal cross-attention guidance module, queries are processed through different combinations. ,key ,value In the steps, the query of visible light and infrared attention is performed. and Bonds applied to infrared and visible light respectively and In, it is represented as: ； ； ； ； In the formula, is a multi-layer perceptron; is the fused visible light feature; is the fused infrared feature.

6. The image fusion detection method based on dual-flow attention guide according to claim 1, wherein in the training process of the deep target detection network, the loss function used is represented as: ； In the formula, is the loss of the difference between the anchor frame and the real anchor frame; is the loss of whether the category of the target is correct or not; is the loss of whether the target is or not; is the loss of the dynamic convolution kernel; is a hyperparameter for balancing; wherein ； ； ； ； wherein, is a preset hyper-parameter used to balance the two losses; represents the divergence between two features; JS represents the kernel of the visible light image; represents the kernel of the infrared image; and respectively represent the unique feature information in the visible light image and the infrared image; is a sign function.​ 7. A dual-stream attention-guided based image fusion detection system, characterized in that, comprises: an image acquisition module for acquiring a visible light image and an infrared image of a target detection region; a target detection module for, based on the acquired visible light image and infrared image of the target detection region, using a pre-trained deep target detection network to perform target detection, and obtaining a target detection result; wherein the deep target detection network comprises: a feature extraction module for inputting the visible light image and the infrared image and performing feature extraction, outputting a feature map of the visible light mode and a feature map of the infrared mode; The dual-flow attention guide module is configured to input the feature map of the visible light modality and the feature map of the infrared modality and perform feature fusion, and output a fused feature map. The detection module is configured to input the fused feature map and perform target detection, and output a target detection result after modality fusion; wherein the target detection result includes a target anchor box position and a target classification result. The dual-flow attention guide module includes a modality self-attention enhancement module, a modality cross-attention guide module, and a modality adaptive dynamic adjustment module; wherein The modal self-attention enhancement module comprises a feature normalization module, a multi-head self-attention module and a skip connection layer; wherein the feature normalization module is used for inputting the feature map of the visible light mode and the feature map of the infrared mode and performing a normalization operation to output a normalized feature map; the multi-head self-attention module is used for inputting the normalized feature map and performing linear mapping to form a query , key , value to obtain an attention weight; and the skip connection layer is used for inputting the feature map of the visible light mode and the feature map of the infrared mode and adding the obtained attention weight to output an enhanced feature map of the visible light mode and an enhanced feature map of the infrared mode. The modal cross-attention guiding module comprises a cross-attention module and a modal mixed connection module; wherein the cross-attention module is used for inputting the enhanced feature map of the visible light modality and the feature map of the infrared modality and performing different combinations on the query , key , value to obtain cross-guided attention weights; the modal mixed connection module is used for inputting the enhanced feature map of the visible light modality and the feature map of the infrared modality and adding the obtained cross-guided attention weights, and outputting the feature map of the visible light modality and the feature map of the infrared modality after modal interaction; The modality adaptive dynamic adjustment module includes a modality channel and spatial aggregation module and a dynamic convolution size adjustment module; wherein the modality channel and spatial aggregation module is configured to input the feature map of the visible light modality and the feature map of the infrared modality after modality interaction and calculate spatial attention weights and channel attention weights, multiply the obtained spatial attention weights and channel attention weights with initial convolution kernels to obtain optimal convolution kernel weights; the dynamic convolution size adjustment module is configured to input the obtained optimal convolution kernel weights, perform convolution operation on the feature map of the visible light modality and the feature map of the infrared modality after modality interaction based on the optimal convolution kernel weights, and obtain a fused feature map.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the image fusion detection method based on dual-flow attention guide according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the image fusion detection method based on dual-flow attention guide according to any one of claims 1 to 6.

Images