Image fusion detection method based on double-flow attention guidance and related device

Abstract

The invention belongs to the technical field of target detection in computer vision, and discloses an image fusion detection method based on double-flow attention guidance and a related device. The image fusion detection method based on double-flow attention guidance comprises the following steps: acquiring a visible light image and an infrared image of a to-be-detected area; based on the obtained visible light image and infrared image of the to-be-detected target area, performing target detection by using a pre-trained deep target detection network to obtain a target detection result; wherein the deep target detection network comprises a feature extraction module, a double-flow attention guidance module and a detection module. According to the technical scheme disclosed by the invention, the technical problems of relatively poor precision and relatively low practicability caused by insufficient guidance of visible light and infrared image fusion on features in the prior art can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection in computer vision, and in particular relates to an image fusion detection method based on dual-stream attention guidance and related devices. Background Art

[0002] Visible light and infrared image fusion detection refers to a technique that uses a modal fusion method to fuse the visible light and infrared images of a target (for example, a ship target such as an aircraft carrier, destroyer, cruiser, etc.) and uses a rectangular frame to locate and classify the target. Further illustrative examples show that the ocean scene is too complex and there are multiple environmental interferences (for example, such as fish scale light interference, towed bait, foil interference, deception interference, suppression interference, etc.). The visible light image is not easy to distinguish in this environment, the target position is deceptive, the target outline of the infrared image is not clear, and the effective information available for extraction is limited, which affects the positioning and detection of marine targets. In addition, the small size of marine ship targets (for example, when the edge of the image is scaled to a square of 640 pixels, the pixel of the target is less than 16×16) is easily confused with other targets at sea (for example, common targets in the marine environment such as islands and lighthouses). In addition, the structure of the marine target is not obvious, and it has a similar outline and smaller pixel size to other targets, which determines that it is not easy to be noticed in the visible light mode and difficult to be distinguished in the infrared mode. In summary, the fusion detection of visible light and infrared images at sea is a very challenging task.

[0003] At present, the existing visible light and infrared image fusion detection methods can be divided into two categories according to the technical process and development stage, namely traditional visible light and infrared image fusion detection and visible light and infrared image fusion detection based on deep learning. Traditional visible light and infrared image fusion detection mainly extracts low-level features (such as edges, colors, textures, etc.) manually, fuses different modalities manually, and performs detection with machine learning classification and regression algorithms. The above scheme relies on a large amount of prior knowledge of interference and a large amount of manual participation. The detection reliability, accuracy and efficiency are low and the labor cost is high. In addition, the fusion process relies on manual alignment. As the types of targets are refined and increased, the energy required for fusion and annotation increases significantly, which is not suitable for large-scale production lines. Visible light and infrared image fusion detection based on deep learning completes the feature extraction, feature fusion, defect location and classification process through a deep target detection neural network to achieve end-to-end automatic detection with high detection accuracy, efficiency and reliability.

[0004] Existing deep learning-based visible light and infrared image fusion detection methods are mainly based on convolutional neural networks (CNNs) and vision transformers (ViTs); among them, convolutional neural networks use the prior knowledge of image translation invariance to effectively learn visible light and infrared image features by aggregating pixel information layer by layer through convolution operations. Although the above existing schemes have high detection accuracy, efficiency and reliability, in environments with complex interference, the direct use of convolution methods cannot effectively distinguish blurred contours and similar targets in the two modalities, cannot fit the differences between visible light and infrared images well, and cannot avoid feature confusion problems caused during the fusion process; therefore, it is urgent to establish an effective visual attention fusion module to enhance the attention to different features of visible light and infrared, so as to improve the mean average precision (mAP) of visible light and infrared image fusion detection. Visual Transformer models long-range dependencies based on the Multi-Head Self-Attention (MHSA) mechanism. In the existing multi-stage visual Transformer classification tasks, it is often downsampled stage by stage and finally outputs the feature map for detection; however, the multi-head self-attention calculation often directly calculates the pre-attention matrix of query, key, and value formed by the linear mapping of the feature map, without considering the attention difference of different modalities, so it is not conducive to the feature extraction of visible light and infrared images; in addition, the pre-attention matrix of query, key, and value formed by the linear mapping of the feature map does not pay attention to the guidance between different modalities, but only performs it within each modality, so it is not conducive to the feature fusion and detection of visible light and external images; therefore, it is urgent to propose a cross-attention guidance mechanism that considers internal information enhancement and information interaction between features to improve the average accuracy of the visual Transformer in the fusion detection of visible light and infrared images. Summary of the invention

[0005] The purpose of the present invention is to provide an image fusion detection method and related devices based on dual-stream attention guidance to solve one or more of the above-mentioned technical problems. The present invention specifically provides a visible light and infrared image fusion detection solution based on dual-stream attention guidance, which can solve the technical problems of poor accuracy and low practicality caused by insufficient guidance of features by visible light and infrared image fusion in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an image fusion detection method based on dual-stream attention guidance, comprising the following steps:

[0008] Acquire visible light images and infrared images of the target detection area;

[0009] Based on the obtained visible light image and infrared image of the target detection area, a pre-trained deep target detection network is used to perform target detection to obtain target detection results;

[0010] Wherein, the deep object detection network includes:

[0011] A feature extraction module, used for inputting a visible light image and an infrared image and performing feature extraction, and outputting a feature map of a visible light modality and a feature map of an infrared modality;

[0012] The dual-stream attention guidance module is used to input the feature map of the visible light modality and the feature map of the infrared modality, perform feature fusion, and output the fused feature map;

[0013] The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modal fusion; wherein the target detection result includes the target anchor box position and the target classification result.

[0014] A further improvement of the present invention is that

[0015] The feature extraction module includes N dual-stream feature downsampling stages, outputting N pairs of visible light image feature maps and infrared image feature maps;

[0016] In each feature downsampling stage, the visible light image and infrared image are downsampled by 2 times, features are extracted through convolution operations, and a layer of feature maps is output.

[0017] A further improvement of the present invention is that

[0018] In the detection module, the feature map after fusion of visible light and infrared is used. After N upsampling and layer-by-layer downsampling, the enhancement and detection of the two modalities of visible light and infrared are completed through the feature pyramid structure to obtain the target detection result after modality fusion.

[0019] A further improvement of the present invention is that

[0020] The dual-stream attention guidance module includes: a modality self-attention enhancement module, a modality cross-attention guidance module and a modality adaptive dynamic adjustment module; wherein,

[0021] The modality self-attention enhancement module includes: a feature normalization module, a multi-head self-attention module and a jump connection layer; wherein the feature normalization module is used to input a feature map of a visible light modality and a feature map of an infrared modality and perform a normalization operation, and output a normalized feature map; the multi-head self-attention module is used to input a normalized feature map and form a query Q, a key K, and a value V through linear mapping to obtain an attention weight; the jump connection layer is used to input a feature map of a visible light modality and a feature map of an infrared modality and add the obtained attention weights to output an enhanced feature map of a visible light modality and a feature map of an infrared modality;

[0022] The modal cross-attention guidance module includes: a cross-attention module and a modal hybrid connection module; wherein the cross-attention module is used to input the enhanced feature map of the visible light modality and the feature map of the infrared modality and obtain the attention weight after cross-guidance through different combinations of query Q, key K, and value V; the modal hybrid connection module is used to input the enhanced feature map of the visible light modality and the feature map of the infrared modality and add the obtained cross-guided attention weight, and output the feature map of the visible light modality and the feature map of the infrared modality after modal interaction;

[0023] The modality adaptive dynamic adjustment module includes: a modality channel and space aggregation module and a dynamic convolution size adjustment module; wherein the modality channel and space aggregation module is used to input the feature map of the visible light modality and the feature map of the infrared modality after modality interaction and calculate the spatial attention weight and the channel attention weight, and multiply the obtained spatial attention weight and channel attention weight with the initial convolution kernel to obtain the optimal convolution kernel weight; the dynamic convolution size adjustment module is used to input the obtained optimal convolution kernel weight, and perform a convolution operation on the feature map of the visible light modality and the feature map of the infrared modality after modality interaction based on the optimal convolution kernel weight to obtain a fused feature map.

[0024] A further improvement of the present invention is that

[0025] The modality self-attention enhancement module is expressed as:

[0026]

[0027] Q(X rgb )~U(-r,r),Q(X ir )~U(-r,r);

[0028] K(X rgb )=Conv 1×1 (X rgb ),V(X rgb )=Conv 1×1 (X rgb );

[0029] K(X ir )=Conv 1×1 (X ir ),V(X ir )=Conv 1×1 (X ir );

[0030] In the formula, X rgb is the input feature map of the visual attention module in the visible light modality; softmax(·) is the exponential normalization function; T is the matrix transpose operation; Q(X rgb )、K(X rgb )、V(X rgb ) represent the corresponding query, key, and value matrices in the attention encoding of the visible light modality; d is the embedding dimension; X ir is the input feature map of the visual attention module of the infrared modality; Q(X ir )、K(X ir )、V(X ir ) represent the corresponding query, key, and value matrices in the attention encoding of the infrared modality; U(-r,r) is a uniform distribution with parameter r; Conv 1×1 (·) is a convolution operation with a kernel size of 1×1.

[0031] A further improvement of the present invention is that

[0032] In the modal cross attention guidance module, in the steps of query Q, key K, value V after different combinations, the query Q(X rgb ) and Q(X ir ) are applied to infrared and visible light keys K(X ir ) and K(X rgb ), expressed as:

[0033]

[0034] Where MLP(·) is a multi-layer perceptron; is the visible light feature after fusion; It is the fused infrared signature.

[0035] A further improvement of the present invention is that

[0036] During the training process of the deep object detection network, the loss function used is expressed as:

[0037]

[0038] In the formula, is the difference loss between the predicted anchor box and the true anchor box; The loss of whether the target category is correct or not; Whether it is the loss of the target; is the dynamic convolution kernel loss; λ is a hyperparameter used for balancing;

[0039] in,

[0040]

[0041] S rgb|ir =sign(K rgb -K ir );

[0042] S ir|rgb =sign(K ir -K rgb );

[0043] Where μ is a preset hyperparameter used to balance the two losses; D js (·) represents the JS divergence between two features; K rgb Represents the kernel of the visible light image; K ir represents the kernel of the infrared image; S rgb|ir and S ir|rgb They represent the unique feature information in the visible light image and infrared image respectively; sign(·) is the sign function.

[0044] In a second aspect, the present invention provides an image fusion detection system based on dual-stream attention guidance, comprising:

[0045] An image acquisition module is used to acquire visible light images and infrared images of the target detection area;

[0046] The target detection module is used to perform target detection based on the acquired visible light image and infrared image of the target detection area using a pre-trained deep target detection network to obtain a target detection result;

[0047] Wherein, the deep object detection network includes:

[0048] A feature extraction module, used for inputting a visible light image and an infrared image and performing feature extraction, and outputting a feature map of a visible light modality and a feature map of an infrared modality;

[0049] The dual-stream attention guidance module is used to input the feature map of the visible light modality and the feature map of the infrared modality, perform feature fusion, and output the fused feature map;

[0050] The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modal fusion; wherein the target detection result includes the target anchor box position and the target classification result.

[0051] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image fusion detection method based on dual-stream attention guidance as described in any one of the first aspect of the present invention is implemented.

[0052] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image fusion detection method based on dual-stream attention guidance as described in any one of the first aspect of the present invention is implemented.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] In the visible light and infrared image fusion detection method based on dual-stream attention guidance provided by the present invention, a pre-trained deep target detection network is used for target detection, which solves the technical problems of poor accuracy and low practicality caused by insufficient guidance of features by the fusion of visible light and infrared images in the prior art. Explanatory, the deep target detection network includes a feature extraction module, a dual-stream attention guidance module and a detection module; wherein the dual-stream attention guidance module enables the network to focus on important areas and features in the image by introducing an attention mechanism, thereby enhancing the ability of the feature extraction module, which helps to more accurately extract key information in visible light and infrared images and provide more powerful support for subsequent target detection tasks; in the image fusion process, the dual-stream attention guidance module can guide the features of visible light and infrared images to be effectively fused, and by focusing on the similarities and differences between the two, the module can generate a fused image containing richer and more accurate information, thereby improving the accuracy and practicality of target detection; the setting of the dual-stream attention guidance module makes the entire deep target detection network more adaptable to environmental factors such as lighting and weather, and the network can work stably even in complex and changeable scenes to provide reliable target detection results. In summary, the present invention utilizes the characteristic of attention to explore local features, strengthens the information connection between visible light and infrared modalities, and introduces the cross-attention guidance mechanism into modal learning, which can effectively improve the average accuracy of all categories of marine detection.

[0055] In the preferred technical solution of the present invention, the dual-stream attention guidance module includes a modal self-attention enhancement module, a modal cross-attention guidance module and a modal adaptive dynamic adjustment module; wherein, the modal self-attention enhancement module and the modal cross-attention guidance module are used to promote effective information fusion and background information suppression between the two modalities, and the dynamic convolution kernel loss is introduced through the modal adaptive dynamic adjustment module to fit the fusion characteristics of the visible light and infrared modalities, while strengthening the feature information within the modality and the fusion effect between the modalities, which can effectively improve the detection effect of multimodal targets in complex environments, and can improve the average accuracy of target detection in all categories. In summary, the method of the present invention is based on the calculation between different attentions, and uses dynamic convolution kernels to encourage neural networks to learn more effective modal interaction features, which can effectively improve the detection accuracy of visible light and infrared images of targets. Each module has strong independence, no complex coupling relationship, and strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 is a flowchart of an image fusion detection method based on dual-stream attention guidance in an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of the structure of a deep object detection network in an embodiment of the present invention;

[0059] Figure 3 is a structural schematic diagram of a modality self-attention enhancement module in an embodiment of the present invention;

[0060] Figure 4 is a structural schematic diagram of a modal cross-attention guidance module in an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of a modal adaptive dynamic adjustment module in an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of the detection results of visible light and infrared targets at sea in a specific embodiment of the present invention; wherein, Figure 6 (a) is a schematic diagram of the detection results of marine targets under visible light conditions. Figure 6 (b) is a schematic diagram of the detection results of sea targets in infrared conditions;

[0063] Figure 7It is a schematic diagram of an image fusion detection system based on dual-stream attention guidance in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0065] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] See also Figure 1 The present invention provides an image fusion detection method based on dual-stream attention guidance, specifically a marine visible light and infrared image fusion detection method based on dual-stream attention guidance, comprising the following steps:

[0067] Step 1: Obtain visible light and infrared images of the sea to be detected; explanatory, the visible light and infrared images of the sea to be detected may be corresponding images with modal alignment, or may be unaligned modal images, may contain interference (fish scale light interference, towed bait, chaff interference, deception interference, suppression interference, etc.), or may be paired images without interference;

[0068] Step 2: Based on the visible light and infrared images to be detected at sea, a pre-trained deep image detection network is used to locate and classify marine targets to obtain detection results; interpretably, the detection results are output as the target's bounding box and category probability.

[0069] See also Figure 2 In an embodiment of the present invention, a deep object detection network includes a feature extraction module, a dual-stream attention guidance module and a detection module connected in sequence; wherein,

[0070] The feature extraction module is used to input visible light and infrared images for feature extraction and output corresponding feature maps of the two modalities; wherein the feature extraction module uses a convolutional neural network as a basic visual feature calculation module;

[0071] The dual-stream attention guidance module is used to input the different modalities and output a fusion result; wherein the dual-stream attention guidance module includes a modality self-attention enhancement module, a modality cross-attention guidance module and a modality adaptive dynamic adjustment module; wherein the modality self-attention enhancement module includes a feature normalization module, a multi-head self-attention module and a jump connection layer; wherein the feature normalization module is used to input a feature map of a visible light modality and a feature map of an infrared modality and perform a normalization operation, and output a normalized feature map; the multi-head self-attention module is used to input a normalized feature map and form a query Q, a key K, and a value V through linear mapping to obtain an attention weight; the jump connection layer is used to input a feature map of a visible light modality and a feature map of an infrared modality and add the obtained attention weights, and output an enhanced feature map of a visible light modality and a feature map of an infrared modality; further explanatory, the jump connection layer is a normalization layer that adds the input visible light and infrared features to the obtained attention;

[0072] The modal cross-attention guidance module includes: a cross-attention module and a modal hybrid connection module; wherein the cross-attention module is used to input the enhanced feature map of the visible light modality and the feature map of the infrared modality and obtain the attention weight after cross-guidance through different combinations of query Q, key K, and value V; the modal hybrid connection module is used to input the enhanced feature map of the visible light modality and the feature map of the infrared modality and add the obtained cross-guided attention weight, and output the feature map of the visible light modality and the feature map of the infrared modality after modal interaction;

[0073] The modality adaptive dynamic adjustment module includes: a modality channel and space aggregation module and a dynamic convolution size adjustment module; wherein the modality channel and space aggregation module is used to input the feature map of the visible light modality and the feature map of the infrared modality after modality interaction and calculate the spatial attention weight and the channel attention weight, and multiply the obtained spatial attention weight and channel attention weight with the initial convolution kernel to obtain the optimal convolution kernel weight; the dynamic convolution size adjustment module is used to input the obtained optimal convolution kernel weight, and perform a convolution operation on the feature map of the visible light modality and the feature map of the infrared modality after modality interaction based on the optimal convolution kernel weight to obtain a fused feature map;

[0074] The detection module is used to input the feature map after feature fusion output by the multimodal feature fusion module, and output the detection result after fusion of different modalities; wherein the detection result includes the anchor frame position and the classification result of the marine vessel type.

[0075] In a specific embodiment of the present invention, a method for fusion detection of visible light and infrared images at sea based on dual-stream attention guidance is proposed, which is used to extract information of visible light and infrared modalities and accurately locate their positions in the presence of complex environmental interference, and a dual-stream attention guidance module is designed to simultaneously consider and utilize the differences and common effective information of visible light and infrared modalities. In view of the problem of information loss caused by the failure of existing visible light and infrared image fusion to fully consider the characteristics of each modality, the present invention fully considers the enhancement effect of self-attention on modality and the interactive guidance effect of cross-attention on different modal information, which can effectively improve the average accuracy of all categories of dual-modal target detection at sea under complex interference. In addition, the dual-stream attention guidance module and modality adaptive adjustment module proposed in the present invention are both based on convolution and matrix operations, and the image naming is consistent with the data form, avoiding complex dependencies, and can be added as modules to various deep target detection models. At the same time, it can be applied in different data sets, with strong generalization ability and broad application prospects.

[0076] See also Figures 2 to 5 In a specific embodiment of the present invention, the specific process of fusion detection of visible light and infrared images at sea is performed according to the following steps, including:

[0077] Step 1: Obtain and construct a dataset of visible light and infrared images at sea. The visible light and infrared images are 1280×1064 pixels, and the corresponding modalities in the dataset are named with the same name.

[0078] Step 2: Build a deep object detection network, which includes a feature extraction module, a two-stream attention guidance module, and a detection module connected in sequence.

[0079] like Figure 2 As shown in the figure, the feature extraction module includes five dual-stream feature downsampling stages; each stage downsamples the visible light and infrared images by 2 times, extracts features through convolution operations, and outputs a layer of feature maps; a total of five paired visible light and infrared image feature maps are obtained, and the paired features will be input into the dual-stream attention guidance module.

[0080] The dual-stream attention guidance module includes a modality self-attention enhancement module, a modality cross-attention guidance module, and a modality adaptive dynamic adjustment module; among them,

[0081] like Figure 3 As shown, the modality self-attention enhancement module enhances the interpretability and saliency of the target through the modality self-attention module inside the visible light and infrared images. The modality self-attention module is expressed as:

[0082]

[0083] Q(X rgb)~U(-r,r),Q(X ir )~U(-r,r);

[0084] K(X rgb )=Conv 1×1 (X rgb ),V(X rgb )=Conv 1×1 (X rgb );

[0085] K(X ir )=Conv 1×1 (X ir ),V(X ir )=Conv 1×1 (X ir );

[0086] In the formula, X rgb is the input feature map of the visual attention module in the visible light modality, X ir is the input feature map of the visual attention module of the infrared modality, Conv 1×1 (·) is the convolution operation with a kernel size of 1×1, is the matrix transposition operation, softmax(·) is the exponential normalization function, which is used to normalize the features to an interval, Q(X rgb )、K(X rgb )、V(X rgb ) represent the corresponding query, key, and value matrices in the attention encoding of the visible light modality, Q(X ir )、K(X ir )、V(X ir ) represent the corresponding query, key, and value matrices in the attention encoding of the infrared modality, d = h × w is the embedding dimension, h and w are the height and width of the visible light and infrared images, respectively, and U(-r, r) is a uniform distribution with parameter r;

[0087] The modality self-attention enhancement module inputs the visible light and infrared images obtained by the feature extraction module respectively, passes through the self-attention module, and then connects the input with the attention result through a jump connection. The enhanced modality features will be input into the modality cross-attention guidance module.

[0088] like Figure 4 As shown, in the modality cross-attention guidance module, the query Q(X rgb ) and Q(X ir ) are applied to infrared and visible light keys K(X ir ) and K(X rgb )middle:

[0089]

[0090]

[0091] In the formula, MLP(·) is a multi-layer perceptron, is the fused visible light feature, It is the fused infrared signature;

[0092] Explanativiously, the cross-attention guidance module exchanges visible light and infrared queries, allowing the visible light and infrared images to adjust the feature points of the corresponding modalities based on the effective information of the target in their own images, thereby enhancing the interaction and fusion of different modalities. The guided features are then used by the modality adaptive dynamic adjustment module to find the differences and prominence between the modalities.

[0093] like Figure 5 As shown, the modal adaptive dynamic adjustment module accepts the visible light and infrared input features after modal interaction. It has a height H, a width W and a channel C in , along V convolution kernels K1, K2, ...K V , the adaptive dynamic adjustment module constructs two feature attention weights, including spatial attention weight Channel attention weight Perform the following calculation method:

[0094] α t ∈φ t (X);

[0095] Where t∈{s,c}, φ represents the sequential operation of the global average pooling layer (GAP), fully connected layers (FC), rectified linear unit (ReLU) and head branch; then, the weights are applied to V convolution kernels to obtain the optimal dynamic convolution kernel K after aggregation, which is calculated as follows:

[0096]

[0097] in, Represents α t The vth element in ; ⊙ represents element-by-element multiplication;

[0098] Then, the optimal convolution kernel is used to perform a convolution operation with the visible light and infrared input features after modal interaction to obtain a fused feature map.

[0099] Step 3: Data preprocessing. Before entering the detection network, the visible light and infrared images are uniformly scaled to 640×640 size, and the pixels are normalized. For image enhancement, 50% of the images are flipped horizontally and vertically, and an image mask with a probability of 0.4 is added.

[0100] Step 4: During the training process, the visible light and infrared images are first downsampled five times to obtain feature maps of five stages. The feature maps of the last three stages are input into the two-stream attention guidance module. The output two-stream features are input into the pyramid neural network for feature aggregation and finally input into the detection head to obtain the target positioning and classification results.

[0101] The neural network training is optimized based on the difference between the output results and the true value, and the JensenShannon (JS) divergence is used to measure the information difference between the visible light features and the infrared features. JS divergence evaluates the similarity between the two distributions, approaching zero when they are similar and reaching 1 when they are not similar. By minimizing the distribution difference, the convergence training of the loss is completed, which is calculated as follows:

[0102]

[0103] Among them, D js (·) represents the JS divergence between two features, K rgb represents the kernel of the visible light image, K ir Kernel representing the infrared image.

[0104] At the same time, the unique feature information in the visible light and infrared images is better enhanced by calculating the difference between the two kernels, denoted as S rgb|ir and S ir|rgb , which corresponds to the feature difference between the visible and infrared images, is calculated as follows:

[0105] S rgb|ir =sign(K rgb -K ir ),S ir|rgb =sign(K ir -K rgb );

[0106] Where sign(·) is a sign function that sets all negative values ​​to 0 to ensure that only the significant part is calculated.

[0107] Next, the dynamic convolution kernel loss Finally it is set as follows:

[0108]

[0109] where μ is set to 0.4.

[0110] In addition, exemplary, in each step of training, back propagation is performed starting from the loss function value; a stochastic gradient descent (SGD) optimizer is used to automatically calculate the loss and update the network model parameters through the framework, so that the network model learns the best target features step by step.

[0111] In the embodiment of the present invention, the loss function used in the training process of the deep object detection network is expressed as:

[0112]

[0113] In the formula, is the difference loss between the predicted anchor box and the true anchor box; The loss of whether the target category is correct or not; Whether it is the loss of the target; is the dynamic convolution kernel loss; λ is a hyperparameter used for balancing.

[0114] In a specific example of the present invention: the marine visible light and infrared image dataset used contains 20 categories and a total of 2484 dual-modal images taken by visible light and infrared cameras, with 1242 visible light and infrared images respectively, and the image size is 1280*1068; it is randomly divided into a training set of 2236 visible light and infrared image pairs and a test set of 248 visible light and infrared image pairs, and the images will be scaled to 640×640 pixels in the preprocessing stage; the initialization method of the detection network parameters in the embodiment of the present invention is that the parameters of the feature extraction module are extracted from the model pre-trained on the public dataset, and the initialization method of the remaining network parameters is random initialization; the training time of the embodiment of the present invention on the Gold 6626R@2.90GHz CPU, 8G memory and NVIDIAGeForce RTX3090 GPU is about 6 hours.

[0115] In the embodiment of the present invention, the specific implementation steps include: first setting relevant training parameters, setting the optimizer used for network update in the technical solution of the present invention to a stochastic gradient descent optimizer, with an initial learning rate of 0.001, a weight decay of 0.0005, and a momentum of 0.937. The deep target detection network of the embodiment of the present invention is composed of a feature extraction module including 5 stages, 3 dual-stream attention guidance modules and 3 detection heads, and the feature extraction module includes 5 downsampling networks; the input of the network is a pair of visible light and infrared images, and the features of the paired images are first extracted by the feature extraction module, and then the features of the last three stages are input into the dual-stream attention guidance module to fuse the features, and finally input into the detection head to obtain the positioning and classification results of the maritime targets. Next, when the divided data set is used for network training, 32 pictures are selected from the training set each time to input into the network, and the selected stochastic gradient descent optimizer is used to update the parameters as much as possible, and the training is completed after 100 rounds of iterations on the data set. Finally, the pictures of the test set are input into the network for detection to obtain the results of the embodiment of the invention, such as Figure 6 As shown, Figure 6 (a) shows the detection effect under visible light. Figure 6 (b) shows the detection effect in the infrared case.

[0116] In summary, the embodiment of the present invention discloses a method for fusion detection of visible light and infrared images at sea based on dual-stream attention guidance. The method utilizes the characteristic of attention to explore local features, strengthens the information connection between visible light and infrared modalities, and guides the difference analysis through cross-attention. The network can effectively learn the salient features of different modalities and overcome the complex mixed interference of the marine environment to achieve robust detection of marine targets. The method of the present invention introduces the cross-attention guidance mechanism into the field of multimodal detection of marine targets, which can effectively improve the average accuracy of all categories of marine detection. Each module has strong independence, is simple to implement, has a moderate number of parameters, and has strong applicability and progressiveness.

[0117] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0118] See also Figure 7 In an embodiment of the present invention, an image fusion detection system based on dual-stream attention guidance is provided, comprising:

[0119] An image acquisition module is used to acquire visible light images and infrared images of the target detection area;

[0120] The target detection module is used to perform target detection based on the acquired visible light image and infrared image of the target detection area using a pre-trained deep target detection network to obtain a target detection result;

[0121] Wherein, the deep object detection network includes:

[0122] A feature extraction module, used for inputting a visible light image and an infrared image and performing feature extraction, and outputting a feature map of a visible light modality and a feature map of an infrared modality;

[0123] The dual-stream attention guidance module is used to input the feature map of the visible light modality and the feature map of the infrared modality, perform feature fusion, and output the fused feature map;

[0124] The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modal fusion; wherein the target detection result includes the target anchor box position and the target classification result.

[0125] In one embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the image fusion detection method based on dual-stream attention guidance.

[0126] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the image fusion detection method based on dual-stream attention guidance in the above embodiment.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) that contain computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An image fusion detection method based on dual-stream attention guidance, characterized in that: The following steps are involved: Acquire visible light images and infrared images of the target detection area; Based on the obtained visible light image and infrared image of the target detection area, a pre-trained deep target detection network is used to perform target detection to obtain target detection results; Wherein, the deep object detection network includes: A feature extraction module, used for inputting a visible light image and an infrared image and performing feature extraction, and outputting a feature map of a visible light modality and a feature map of an infrared modality; The dual-stream attention guidance module is used to input the feature map of the visible light modality and the feature map of the infrared modality, perform feature fusion, and output the fused feature map; The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modal fusion; wherein the target detection result includes the target anchor box position and the target classification result.

2. According to claim 1, the image fusion detection method based on dual-stream attention guidance is characterized in that: The feature extraction module includes N dual-stream feature downsampling stages, outputting N pairs of visible light image feature maps and infrared image feature maps; In each feature downsampling stage, the visible light image and infrared image are downsampled by 2 times, features are extracted through convolution operations, and a layer of feature maps is output.

3. According to claim 2, the image fusion detection method based on dual-stream attention guidance is characterized in that: In the detection module, the feature map after fusion of visible light and infrared is used. After N upsampling and layer-by-layer downsampling, the enhancement and detection of the two modalities of visible light and infrared are completed through the feature pyramid structure to obtain the target detection result after modality fusion.

4. According to claim 1, the image fusion detection method based on dual-stream attention guidance is characterized in that: The dual-stream attention guidance module includes: a modality self-attention enhancement module, a modality cross-attention guidance module and a modality adaptive dynamic adjustment module; wherein, The modality self-attention enhancement module includes: a feature normalization module, a multi-head self-attention module and a jump connection layer; wherein the feature normalization module is used to input a feature map of a visible light modality and a feature map of an infrared modality and perform a normalization operation, and output a normalized feature map; the multi-head self-attention module is used to input a normalized feature map and form a query Q, a key K, and a value V through linear mapping to obtain an attention weight; the jump connection layer is used to input a feature map of a visible light modality and a feature map of an infrared modality and add the obtained attention weights to output an enhanced feature map of a visible light modality and a feature map of an infrared modality; The modal cross-attention guidance module includes: a cross-attention module and a modal hybrid connection module; wherein the cross-attention module is used to input the enhanced feature map of the visible light modality and the feature map of the infrared modality and obtain the attention weight after cross-guidance through different combinations of query Q, key K, and value V; the modal hybrid connection module is used to input the enhanced feature map of the visible light modality and the feature map of the infrared modality and add the obtained cross-guided attention weight, and output the feature map of the visible light modality and the feature map of the infrared modality after modal interaction; The modality adaptive dynamic adjustment module includes: a modality channel and space aggregation module and a dynamic convolution size adjustment module; wherein the modality channel and space aggregation module is used to input the feature map of the visible light modality and the feature map of the infrared modality after modality interaction and calculate the spatial attention weight and the channel attention weight, and multiply the obtained spatial attention weight and channel attention weight with the initial convolution kernel to obtain the optimal convolution kernel weight; the dynamic convolution size adjustment module is used to input the obtained optimal convolution kernel weight, and perform a convolution operation on the feature map of the visible light modality and the feature map of the infrared modality after modality interaction based on the optimal convolution kernel weight to obtain a fused feature map.

5. According to claim 4, the image fusion detection method based on dual-stream attention guidance is characterized in that: The modality self-attention enhancement module is expressed as: Q(X rgb )~U(-r,r),Q(X ir )~U(-r,r); K(X rgb )=Conv 1×1 (X rgb ),V(X rgb )=Conv 1×1 (X rgb ); K(X ir )=Conv 1×1 (X ir ),V(X ir )=Conv 1×1 (X ir ); Where, X rgb is the input feature map of the visual attention module in the visible light modality; softmax(·) is the exponential normalization function; T is the matrix transpose operation; Q(X rgb )、K(X rgb )、V(X rgb ) represent the corresponding query, key, and value matrices in the attention encoding of the visible light modality; d is the embedding dimension; X ir is the input feature map of the visual attention module of the infrared modality; Q(X ir )、K(X ir )、V(X ir ) represent the corresponding query, key, and value matrices in the attention encoding of the infrared modality; U(-r, r) is a uniform distribution with parameter r; Conv 1×1 (·) is a convolution operation with a kernel size of 1×1.

6. The image fusion detection method based on dual-stream attention guidance according to claim 5 is characterized in that: In the modal cross attention guidance module, in the steps of query Q, key K, value V after different combinations, the query Q(X rgb ) and Q(X ir ) are applied to infrared and visible light keys K(X ir ) and K(X rgb ), expressed as: Where MLP(·) is a multi-layer perceptron; is the visible light feature after fusion; It is the fused infrared signature.

7. The image fusion detection method based on dual-stream attention guidance according to claim 1 is characterized in that: During the training process of the deep object detection network, the loss function used is expressed as: In the formula, is the difference loss between the predicted anchor box and the true anchor box; The loss of whether the target category is correct or not; Whether it is the loss of the target; is the dynamic convolution kernel loss; λ is a hyperparameter used for balancing; in, S rgbir =sign(K rgb -K ir ); S ir|rgb =sign(K ir -K rgb ); Where μ is a preset hyperparameter used to balance the two losses; D js (·) represents the JS divergence between two features; K rgb Represents the kernel of the visible light image; K ir represents the kernel of the infrared image; S rgb|ir and S ir|rgb They represent the unique feature information in the visible light image and infrared image respectively; sign(·) is the sign function.

8. An image fusion detection system based on dual-stream attention guidance, characterized in that: include: An image acquisition module is used to acquire visible light images and infrared images of the target detection area; The target detection module is used to perform target detection based on the acquired visible light image and infrared image of the target detection area using a pre-trained deep target detection network to obtain a target detection result; Wherein, the deep object detection network includes: A feature extraction module, used for inputting a visible light image and an infrared image and performing feature extraction, and outputting a feature map of a visible light modality and a feature map of an infrared modality; The dual-stream attention guidance module is used to input the feature map of the visible light modality and the feature map of the infrared modality, perform feature fusion, and output the fused feature map; The detection module is used to input the fused feature map and perform target detection, and output the target detection result after modal fusion; wherein the target detection result includes the target anchor box position and the target classification result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the image fusion detection method based on dual-stream attention guidance as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image fusion detection method based on dual-stream attention guidance as described in any one of claims 1 to 7 is implemented.

Images