Visible light and infrared image fusion method based on double-domain cooperative detection driving

Abstract

The invention discloses a visible light and infrared image fusion method based on dual-domain cooperative detection driving, and the method comprises the steps: constructing a dual-domain cooperative detection driving network (DC-DDNet), and carrying out the detection of the dual-domain cooperative detection driving network (DC-DDNet) through the dual-domain cooperative detection driving network (DC-DDNet), and carrying out the detection of the dual-domain cooperative detection driving network (DC-DDNet) through the dual-domain cooperative detection driving network (DC-DDNet). The method specifically comprises the following steps: inputting initial visible light and infrared images into a target perception double-domain attention guiding module to generate a target mask; inputting the initial visible light and infrared images into a spatial feature extraction module, and introducing a target mask to obtain a mask enhanced visible light and infrared feature map; inputting the initial visible light and infrared images into a frequency domain feature extraction module to obtain a mask enhanced frequency domain feature map; and finally, a multi-task joint loss function is constructed, real-time performance feedback is provided by embedding a lightweight detection sub-network, end-to-end optimization of fusion quality and detection precision is realized, and a fusion image is obtained. The device is reasonable in structural design and suitable for the technical field of image fusion.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to image fusion, particularly to a visible light and infrared image fusion method based on dual-domain collaborative detection. Background Technology

[0002] In the fields of computer vision and signal processing, multimodal data fusion technology is driving a leapfrog development in environmental perception capabilities. This technology holds significant value in practical applications such as medical diagnosis, geographic information surveying, and intelligent security. Especially under complex lighting and extreme weather conditions, acquiring complementary information from visible light and infrared radiation images of the same scene has become a mainstream technique. Visible light images possess rich texture details and color fidelity, while infrared images overcome lighting limitations through thermal radiation characteristics, enabling all-weather target recognition. The effective fusion of these two technologies not only overcomes the physical limitations of a single sensor but also significantly enhances the environmental adaptability of the perception system through cross-modal information complementarity.

[0003] However, most current fusion algorithms focus on pixel-level contrast enhancement and texture detail preservation, such as general metrics like structural similarity and edge strength, while neglecting the fundamental purpose of image fusion: serving real-world applications. For object detection tasks, existing fusion methods have certain limitations: on the one hand, excessive pursuit of pixel-level detail may lead to redundant preservation of task-irrelevant information, such as complex background noise; on the other hand, key features related to the target, such as the thermal target contour in infrared mode and texture details in visible light mode, may be weakened due to the need for universality in the fusion strategy. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a visible light and infrared image fusion method based on dual-domain collaborative detection, so that the final fused image can simultaneously meet the following two requirements: (1) Fusion quality assurance: retain the texture details of the visible light image and the thermal target features of the infrared image; (2) Task performance drive: suppress background interference unrelated to target detection and enhance the semantic consistency of the target region.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A visible light and infrared image fusion method based on dual-domain collaborative detection includes the following steps: Step 1: Construct a dual-domain collaborative detection-driven network (DC-DDNet). DC-DDNet consists of a spatial domain feature extraction module, a frequency domain feature extraction module, and a task-aware dual-domain attention (TA-DA) module. Each module takes infrared and visible light images as input. Step 2: The target perception dual-domain attention guidance module adopts a dual-branch mask generation strategy, generating independent masks for the infrared branch and the visible light branch respectively, and then fusing them through channel-cascaded convolution to obtain the target mask. This guides the subsequent feature optimization process; Step 3: The target perception dual-domain attention guidance module and the spatial domain feature extraction module constitute a target mask-guided spatial domain attention network. The spatial domain feature extraction module first extracts infrared feature maps with complementary information. and visible light feature map Then, a target mask is introduced, which is respectively compared with the target mask. Perform bit-by-bit multiplication to obtain the masked enhanced infrared feature map. With mask-enhanced visible light feature map ; Step 4: The target perception dual-domain attention guidance module and the frequency domain feature extraction module constitute a target mask-guided spectral decomposition optimization network. The frequency domain feature extraction module obtains the modally complementary amplitude spectrum through fast Fourier transform and encoding / decoding. and phase spectrum Using the target mask Guided amplitude spectrum Frequency band selection and phase spectrum The target region is enhanced, and then subjected to inverse Fourier transform to obtain a mask-enhanced frequency domain feature map; Step 5, enhance the infrared feature map with the mask. Mask-enhanced visible light feature map The mask-enhanced frequency domain feature map is stitched together along the channel to perform cross-domain fusion and obtain a fused image. .

[0006] In this invention, the target perception dual-domain attention guidance module uses the intermediate layer feature map of the target detector as the attention guidance signal, and generates the target mask through multi-scale feature fusion and lightweight convolution. In a spatial domain attention network guided by the target mask, the target mask is... With infrared feature map and visible light feature map By performing bitwise multiplication, the network becomes more focused on the channel characteristics of the target region; in the target mask-guided spectral decomposition optimization network, the target mask is utilized. Guided amplitude spectrum Frequency band selection, enhancement of target correlation frequency band, suppression of high-frequency background noise, and processing of phase spectrum. A target-oriented phase-preserving strategy was proposed, which prioritizes preserving the phase information of the target region during inverse Fourier transform.

[0007] In one embodiment of the present invention, in step 2, a target mask is obtained. The steps are as follows: Step 21: Introduce the pre-trained YOLOv5 Backbone main architecture network and corresponding parameters through transfer learning.

[0008] Step 22: A target mask generation method based on multi-scale feature fusion is adopted to extract feature maps from different layers such as C1_1, C1_2, and C1_3 of the Backbone to obtain multi-scale features.

[0009] Step 23: A lightweight feature pyramid network is used for multi-scale fusion, and the size is adjusted to match the original image size through bilinear interpolation upsampling and other methods to obtain the infrared fused features. Features of fusion with visible light ,in Its dimensions H×W are the same as the original Figure 1 Therefore, channel dimension C contains target semantic information. One specific method is as follows: The high-level and low-level features are fused layer by layer using a top-down approach. First, the highest-level feature (C1_3 layer in this embodiment) is compressed by a 1×1 convolution to reduce the number of channels. Then, it is upsampled using bilinear interpolation to the same size as the feature map of the previous layer (C1_2 layer in this embodiment), and then element-wise added to the feature map of the previous layer (C1_2 layer in this embodiment). Next, the feature map of the previous layer (C1_2 layer in this embodiment) is compressed by a 1×1 convolution to reduce the number of channels, and then upsampled using bilinear interpolation to the same size as the feature map of the layer above it (C1_1 layer in this embodiment), and then element-wise added to the feature map of the layer above it (C1_1 layer in this embodiment), and so on, to obtain the preliminary fused feature. Finally, the preliminary fused feature is upsampled using bilinear interpolation to the same size as the original image to obtain the infrared fused feature. Features of fusion with visible light Its channel dimension contains target semantic information.

[0010] Step 24: Use a lightweight convolutional module to obtain single-channel probability masks for infrared and visible light. and , Specifically, this lightweight convolution can be implemented using two layers of 3×3 convolutions, a ReLU activation function, and a 1×1 convolution, which can be expressed by the formula: In the formula, The sigmoid function ensures that the mask value is between 0 and 1. To refer to parameters, that is or .

[0011] Step 25, and By cascading along the channels and then processing with a 1×1 convolutional layer and a sigmoid function, a single-mode target mask is obtained. , is represented as: Step 26: During the training phase, mask the target using the real target bounding boxes output by the target detector. Supervision is applied. The loss function is chosen as the binary cross-entropy, with the specific formula as follows: in, Indicates the target area. Indicates the background area. For the total number of training rounds, For the first The target mask obtained from the training.

[0012] In one embodiment of the present invention, the spatial domain feature extraction module outputs an infrared feature map with complementary information through the combined action of an AFCB (Attention Fully Connected Block) network, an edge attention mechanism, and convolution. and visible light feature map The AFCB network performs the following steps: Step 31: First, a 3×3 convolutional kernel combined with a parameterized ReLU activation function is used to perform feature encoding on the original bimodal input image. The output is then subjected to global average pooling. Finally, the feature map is compressed into a 1×1×C vector, where C is the number of channels. Step 32: Perform FFN1 operation on the output 1×1×C vector. FFN1 consists of a fully connected layer and a ReLU activation function. In order to reduce computational complexity and the number of parameters, a fully connected layer is used to reduce the number of channels from C to C / r, where the dimensionality reduction factor r is set to 4 by default in this invention.

[0013] Step 33: Perform FFN2 operation on the dimensionality-reduced vector. FFN2 consists of a fully connected layer and a sigmoid activation function. The fully connected layer restores the number of feature channels from C / r to the original dimension C to ensure compatibility of subsequent operations. Then, apply the sigmoid activation function to the restored feature vector to generate channel attention weights. Step 34: Perform a dot product operation between the generated channel attention weights and the original input image to weight the feature map, thereby enhancing important features and suppressing irrelevant features.

[0014] In one embodiment of the present invention, a target mask is introduced, and the channel weights are dynamically adjusted to adjust the target mask. Copy along channel dimension Next (in this embodiment) ), to obtain the adjusted mask Then to and Spatial attention weighting is performed separately, and the formula is as follows: in, This represents element-wise multiplication; in the visible light branch, the target region... Enhance visible light texture features; in the infrared branch, background areas This enhances the infrared thermal target characteristics.

[0015] Furthermore, the present invention can dynamically adjust channel weights through a target-aware dual-domain attention (TA-DA) module.

[0016] In one embodiment of the present invention, in step 4, the target mask-guided spectral decomposition optimization network utilizes the target mask. Guided amplitude spectrum The frequency band selection is implemented as follows: target mask The frequency domain mask is obtained by performing a fast Fourier transform. Then, the frequency domain mask is applied through a 1×1 convolutional layer and a sigmoid activation function. The sum of the real and imaginary parts is processed to generate the frequency band selection matrix. The formula is: Using the generated frequency band selection matrix Amplitude spectrum Element-wise multiplication yields the optimized amplitude spectrum. .

[0017] In one embodiment of the present invention, in step 4, the target mask-guided spectral decomposition optimization network utilizes the target mask. phase spectrum The method for target region enhancement is as follows: Introduce a learnable parameter It is initialized (chosen as 0.5 in this embodiment), and its value is dynamically adjusted through training to optimize the phase spectrum. Expressed as a formula: In one embodiment of the present invention, step 5 introduces a multi-task joint optimization loss function, the total loss function. Represented as: In the formula It is a dynamic weighting coefficient. For fusion losses, including content losses Structural similarity loss Frequency domain fusion loss , To detect loss, including classification loss. Location loss and confidence loss ,Right now ; The content loss The content loss under dual constraints mainly includes pixel intensity loss and gradient loss, as shown in the following formula: in For pixel intensity loss, For gradient loss, and For learnable parameters, This is the fusion result of real infrared and visible light images, compared with the final fused image of infrared and visible light feature maps. They have different concepts. For the Sobel gradient operator, It is an L1 norm; The structural similarity loss The formula is as follows: in The original infrared image, This is the original visible light image; The frequency domain fusion loss The formula is as follows: in This is the result obtained by converting the fused amplitude spectrum and phase spectrum back to the spatial domain using inverse Fourier transform. , The amplitude spectrum after fusion. The phase spectrum after fusion. This represents the inverse fast Fourier transform. , Representing variables I The correlation coefficient between infrared and visible light images. It is an indicator that measures the degree of linear correlation between two variables, and its value generally ranges from [...]. Between 1 and 1]. This represents the regularization term used to reduce noise. To control the hyperparameters of regularization strength, yes and Covariance between and yes and Standard deviation; and To refer to parameters, Reference , Refers to the reference or .

[0018] For a batch One sample, and The classification loss has several categories. The classification loss is the average of the binary cross-entropy loss for all samples across all classes. The formula is expressed as Here, we assume that for a given sample, the true class label is... This indicates whether the sample belongs to the first... The model predicts that the sample belongs to the class. The probability of class is ,but For a single sample in the first The binary cross-entropy loss on the class is , The model represents the first i The predicted probability values ​​for each category, Indicates the firstn The sample, the first i Real labels for each category Indicates the first n The sample, the first i The predicted probability values ​​for each category; For a batch A predicted bounding box, the localization loss CIoU loss for all predicted boxes The average value is calculated using the following formula: in It is the center point of the prediction box. and the center point of the true frame The square of the Euclidean distance between them It is intersection, union, and comparison. c It is the diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted bounding box and the ground truth bounding box. It is a balancing factor. v This is a parameter used to measure the consistency of the aspect ratio between the predicted bounding box and the ground truth bounding box. Specifically, it refers to the Complete Intersection over Union (CIoU) loss in YOLOv5.

[0019] For a batch A prediction box, the confidence loss Confidence loss for all predicted boxes The average value is given by the formula: in For the confidence loss of a single predicted bounding box, The true confidence level label indicates whether the predicted bounding box contains the target. The predicted bounding box for the model includes the confidence level of the target. , , The values ​​of are preferably the same.

[0020] In one embodiment of the present invention, step 5 involves introducing a lightweight detection subnetwork, MobileNetV3-YOLOv5s, during the training phase. The input to this network is the fused image. ,in H and W These represent the height and width of the image, respectively, and the output is the detection loss. This loss is used for backpropagation to optimize the parameters of the fused network.

[0021] The training strategy employed in this invention sets dynamic weight coefficients. , The adjustment is automatically based on the ratio of detection loss to fusion loss, using the following formula: in This is a hyperparameter (default value is 0.5). Used to prevent division by zero errors.

[0022] To address the problem that existing fusion algorithms, in their excessive pursuit of pixel-level detail preservation, introduce a large amount of background noise irrelevant to the task, resulting in insufficient accuracy in subsequent detection tasks, this invention is based on a visible light and infrared image fusion algorithm driven by dual-domain collaborative detection. Guided by the needs of target detection, it first proposes a task-aware mechanism, which dynamically allocates cross-modal attention weights through the response map of multi-scale features in the intermediate layer of the target detector, thereby enhancing the saliency of the target region and the visible light texture resolution, achieving feature optimization and noise suppression.

[0023] Furthermore, this invention constructs a multi-task joint loss function and provides real-time performance feedback by embedding a lightweight detection sub-network, thereby achieving end-to-end optimization of fusion quality and detection accuracy.

[0024] Experimental verification on public datasets shows that the fused images generated by this invention effectively preserve visible light texture and infrared thermal features, significantly improving the average accuracy of target detection tasks. Attached Figure Description

[0025] Figure 1 This is a diagram of a dual-branch mask generation network architecture.

[0026] Figure 2 This is a diagram of a single-modal mask generation network architecture.

[0027] Figure 3 This is a diagram of a spatial domain attention network architecture guided by a target mask.

[0028] Figure 4 This is a diagram of the target mask-guided spectral decomposition optimization network architecture.

[0029] Figure 5 This is a diagram of the visible light and infrared image fusion network architecture based on DC-DDNet. Detailed Implementation

[0030] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0031] There are often two major problems in deep learning-based image fusion algorithms: (1) Over-pursuing pixel-level details may lead to redundant retention of task-irrelevant information, such as complex background noise; (2) Key features related to the target, such as the thermal target contour in infrared mode and the texture details in visible light mode, may be weakened due to the universality requirements of the fusion strategy.

[0032] Based on this, this invention proposes a Dual-domain Collaborative Detection-Driven Network (DC-DDNet), which introduces a dual-domain attention-guided module for target perception to learn prior information about targets in infrared and visible light images, thereby enriching the hierarchy of features and effectively optimizing the network's ability to extract salient targets. Furthermore, a multi-level task joint loss function is introduced into the original network loss function, enabling the model to achieve a better balance between fusion quality and detection performance.

[0033] To verify the effectiveness of the method of this invention, the M3FD public dataset was used as experimental data to provide rich and representative data samples for the experiment. This dataset contains 4200 pairs of registered infrared and visible light images with corresponding target detection labels. The images are 1024×768 pixels in size, and most scenes were taken on university campuses and urban roads, characterized by frequent pedestrian traffic, diverse types of people, relatively complex environments, and various lighting and background interferences. Since the dataset was not pre-segmented during the image fusion experiment, it was necessary to divide the dataset into training and testing sets to scientifically evaluate the algorithm's performance. 840 pairs of infrared and visible light images were randomly selected for testing, with a segmentation rate of 0.2.

[0034] The algorithm in this invention was written in Python 3.10 under Windows, and the DC-DDNet network model was trained and tested using the PyTorch framework. An NVIDIA GeForce RTX 2080 Ti GPU was used for acceleration during the experiments. During model training, the batch size for each iteration was set to 4, and a total of 100 epochs were trained. The Adam optimizer was used to optimize the network parameters, with an initial learning rate of 0.001. The hyperparameters in the loss function... Set to 0.5. The evaluation criteria mainly include: Information Entropy (EN), Standard Deviation (SD), Structural Similarity (SSIM), Visual Information Fidelity (VIF), Mutual Information (MI), and Edge Preservation Factor (Q). AB / F The higher the value of the above evaluation index, the better the image fusion effect.

[0035] (1) Information entropy (EN) is an important concept in information theory, and it is a quantitative estimate of the amount of information in a fused image. In the result of image fusion, the higher the information entropy value, the more information the image contains, the more source image information is preserved, and the better the algorithm performance. Its calculation expression is as follows: In the formula, It is the total number of gray levels in the image. This is the normalized histogram of gray levels in the fused image.

[0036] (2) Standard Deviation (SD) is a measure of the degree of variation or dispersion of pixel values ​​in an image, reflecting the richness of texture and contrast. A larger standard deviation in a fused image indicates a wider distribution of grayscale values, resulting in higher contrast and detail. Its calculation expression is: In the formula, It is to merge images at the pixel level The grayscale value of the location, It is the average value at that pixel.

[0037] (3) Structural Similarity (SSIM) is a comprehensive image quality assessment metric that considers the human eye's sensitivity to image structural information. Therefore, directly comparing the structural features of images is more effective in reflecting image quality than simply comparing pixel values. Specifically, it measures the similarity between two images by comparing the differences in three variables: structure, contrast, and brightness. Its calculation expression is as follows: In the formula, the input image is X The result is shown in the figure. F , for X and F covariance; and for A and F The standard deviation; and for A and F The average value. C 1. C 2. C 3 represents a parameter used to improve the stability of the algorithm. The overall structural similarity of images is defined as... In the formula, and These represent the input images respectively. A With input image B The structural similarity of the input and the fusion result. A higher structural similarity value indicates a greater proportion of the input is passed to the fusion result, suggesting a better fusion effect.

[0038] (4) Visual Information Fidelity (VIF) is a measure of the degree of distortion in the image being measured. Based on human visual characteristics, it considers the fidelity of information in the image and its importance to the observer's vision. A higher VIF value means that the fused image retains a higher degree of information, resulting in better image quality and effectively measuring the visual quality of the image. Its calculation expression is as follows: In the formula, VIF(•) is the image information fidelity function.

[0039] (5) Mutual Information (MI) is a measure of the statistical correlation between multiple random variables, effectively measuring the dependence and sharing of information. A higher MI value indicates a stronger dependence between the two variables and better fusion quality. Its calculation expression is: In the formula, and It is the input image A and B The degree to which information in a random variable can be represented in a fused image. Mutual information between any two random variables is defined as... In the formula, and Representing the input images respectively X With fused images F The normalized histogram, The joint histogram representing the two.

[0040] (6) Edge preservation factor ( ( ) is a measure of how well edge information is preserved in the fusion result during multimodal image fusion. Its value is typically between 0 and 1. A higher value indicates that the fusion result preserves the edges in the input image better, resulting in better fusion quality. Its calculation expression is: In the formula, Indicates the position of pixels in an image. Indicates the location Edge strength at the location, This represents the relative importance weight of each input source when generating the fused image. This indicates the direction value at that location.

[0041] The process of fusing visible light and infrared images using the method of this invention is as follows: Step 1 involves selecting visible light and infrared image pairs from the dataset.

[0042] Step 2: Input the initial visible light and infrared images into the target perception dual-domain attention guidance module, use the intermediate layer feature map of the target detector as the attention guidance signal, and generate the target mask through multi-scale feature fusion and lightweight convolution. .

[0043] In this step, the intermediate layer feature map of the target detector contains semantic priors for the target region, such as the activation response of a thermal target or the contour features of a visible light target. By extracting these features, a probabilistic mask for the target region can be generated, guiding the fusion network to focus on key regions. To more effectively fuse target prior information from infrared and visible light images, this invention employs a dual-branch mask generation strategy, such as... Figure 1 As shown, separate masks are generated for the infrared and visible light branches, and then the two are fused through channel-level convolution to guide the subsequent feature optimization process.

[0044] Target mask generation network architecture for visible light and infrared images, such as Figure 2 As shown, the specific steps include: Step 21: Introduce the pre-trained YOLOv5 Backbone main architecture network and corresponding parameters through transfer learning.

[0045] Step 22: Design a target mask generation method based on multi-scale feature fusion, extract feature maps from different layers such as C1_1, C1_2, and C1_3 of the Backbone to obtain multi-scale features.

[0046] Step 23: Multi-scale fusion is performed by designing a lightweight feature pyramid network. The specific process is as follows: High-level features are fused layer by layer with low-level features through a top-down path. First, the high-level features of layer C1_3 are compressed by a 1×1 convolution, then upsampled using bilinear interpolation to the same size as the feature map of layer C1_2, and added element-wise to the features of layer C1_2. Next, the features of layer C1_2 are compressed by a 1×1 convolution, then upsampled using bilinear interpolation to the same size as the feature map of layer C1_1, and added element-wise to the features of layer C1_1 to obtain the fused features. Finally, the fused features are upsampled using bilinear interpolation to the same size as the original image to obtain the final fused features. Its dimensions H×W are the same as the original Figure 1 Therefore, channel dimension C contains target semantic information.

[0047] Step 24: Design a lightweight convolutional module to obtain a single-channel probability mask. The lightweight convolution operation is implemented using two 3×3 convolutions, a ReLU activation function, and a 1×1 convolution, which can be expressed by the formula: In the formula, Use the Sigmoid function to ensure that the mask value is between 0 and 1.

[0048] Step 25: Concatenate the visible light and infrared image masks along the channels, and then process them using a 1×1 convolutional layer and a sigmoid function to obtain a single-mode target mask. , is represented as: Step 26: During the training phase, use the ground truth bounding boxes output by the detector. Supervision is applied, and the loss function is binary cross-entropy, with the specific formula as follows: in, Indicates the target area. Indicates the background area. For the total number of training rounds, For the first The target mask obtained from the training.

[0049] Step 3: Input the initial visible light and infrared images into the spatial domain feature extraction module, introduce the target mask, and multiply the mask bit by bit with the visible light and infrared image features output by the original spatial feature extraction module to obtain the mask-enhanced visible light and infrared feature map.

[0050] In this step, the target mask-guided spatial domain attention network architecture is as follows: Figure 3 As shown, the spatial domain feature extraction module outputs infrared feature maps with complementary information through the combined action of the AFCB network, edge attention mechanism, and convolution. and visible light feature map Introducing a target mask and dynamically adjusting channel weights, the single-channel mask... Copy 8 times along the channel dimension to obtain Next, spatial attention weighting is applied to the visible light and infrared feature maps respectively, using the following formula: In the formula, This represents element-wise multiplication; in the visible light branch, the target region... Enhance visible light texture features; in the infrared branch, background areas This enhances the characteristics of infrared thermal targets. Through differentiated weighting, a balance is achieved between detection requirements and scenario consistency.

[0051] Step 4: Input the initial visible light and infrared images into the frequency domain feature extraction module, use the target mask to guide the frequency band selection of the amplitude spectrum, enhance the target related frequency band, and obtain the mask-enhanced frequency domain feature map.

[0052] In this step, the target mask-guided spectral decomposition optimization network architecture is as follows: Figure 4 As shown.

[0053] Step 41: Input a single-channel visible light image and an infrared image into the frequency domain module, and perform a fast Fourier transform on these two images to convert them from the spatial domain to the frequency domain, thereby obtaining the corresponding complex spectrum.

[0054] Step 42: Feature extraction and fusion of the dual-mode amplitude spectrum and phase spectrum are achieved through a dual-branch encoding and decoding structure, resulting in modally complementary amplitude spectra. and phase spectrum .

[0055] Step 43, introduce the target mask First, perform a Fast Fourier Transform on the target mask to obtain the frequency domain mask. Then, the sum of the real and imaginary parts of the frequency domain mask is processed by a 1×1 convolutional layer and a sigmoid activation function to generate a frequency band selection matrix. Its calculation formula is This matrix assigns different weights to different frequency bands of the amplitude spectrum, making the weights of low-frequency regions close to 1, while the weights of high-frequency noise regions close to 0. Finally, the generated frequency band selection matrix is ​​used... Amplitude spectrum Element-wise multiplication yields the optimized amplitude spectrum. In this way, target-related low-frequency information can be enhanced while suppressing background noise.

[0056] To further highlight the phase information of the target region, the phase spectrum... Target region augmentation is performed. Specifically, a learnable parameter is introduced. It is initialized to 0.5 and its value is dynamically adjusted through training. Optimized phase spectrum. This can be expressed by the formula as follows: By employing a frequency band selection strategy, the low-frequency components of the target region (i.e., the thermal radiation weight of the target) are significantly enhanced, thereby increasing the proportion of infrared information in the channel features. Simultaneously, the texture details of the visible light image are also preserved to some extent. This characteristic helps to more accurately highlight the thermal radiation features of the target in the fused image, while also taking into account the detailed information of the visible light image. In high-frequency background noise regions, the frequency band selection strategy reduces the channel weights, weakening the background response in the feature map and effectively reducing the false alarm rate of the detector. By suppressing background noise interference, the accuracy of target detection and recognition can be improved, further enhancing the overall system performance.

[0057] Step 5: Input the mask-enhanced visible light and infrared feature maps and the mask-enhanced frequency domain feature map into the fusion module, introduce the multi-task joint optimization loss function, add a loss term for the target detection task, realize end-to-end joint optimization, and obtain the fused image.

[0058] The specific steps in this process are as follows: A lightweight object detection sub-network is embedded during the training phase to provide real-time detection performance feedback and guide parameter updates for the fusion network. This sub-network is only used during training and can be removed during testing to avoid increasing computational burden. Detection feedback adjusts the attention weights of the fusion network through gradient backpropagation, achieving task-driven feature optimization.

[0059] Following the above process, this invention proposes a visible light and infrared image fusion method driven by dual-domain collaborative detection. This algorithm introduces a target perception dual-domain attention-guided module to learn prior target information from both infrared and visible light images, thereby enriching the feature hierarchy and effectively optimizing the network's ability to extract salient targets. A multi-level task joint loss function is introduced into the original network loss function, enabling the model to achieve a better balance between fusion quality and detection performance. Finally, a series of experiments verify that the proposed method significantly improves the accuracy of subsequent target detection tasks while ensuring the quality of the fused image. The complete architecture and process of this invention can be found in [reference needed]. Figure 5 As shown.

[0060] To demonstrate the effectiveness of this method, it was compared and analyzed with traditional algorithms and deep learning algorithms. Furthermore, it was compared and verified with five classic algorithms: the wavelet transform-based fusion method (WTIF); and four deep learning-based algorithms: the unified unsupervised image fusion method (U2Fusion), the densely connected network-based fusion method (DenseFuse), the depth image decomposition-based fusion method (DIDFuse), and the Y-shaped dynamic transformer-based fusion method (YDTR).

[0061] To demonstrate the effectiveness of the algorithm, a comparative experiment was conducted on the M3FD public dataset to perform an objective metric comparison analysis.

[0062] Table 1 shows the average values ​​of objective evaluation metrics for each method in ten random sets of visible light and infrared images fused together. The best results in the table are marked in bold red. It can be seen that the algorithm of this invention has the highest values ​​in the evaluation metrics of information entropy (EN), standard deviation (SD), structural similarity (SSIM), visual information fidelity (VIF), mutual information (MI), and edge preservation factor (QAB / F), which means that the image fusion effect is the best.

[0063] Table 1. Average evaluation metrics for fusion results of ten sets of images in the M3FD dataset using different methods. In summary, this invention employs a visible light and infrared image fusion method driven by dual-domain collaborative detection. Addressing the problem that existing algorithms, by over-focusing on pixel-level details and thus retaining a large amount of task-irrelevant information, result in insufficient detection rates of the fused image in subsequent object detection tasks, this invention introduces a target-aware dual-domain attention-guided module. This module learns prior target information from both infrared and visible light images, enriching the feature hierarchy and effectively optimizing the network's ability to extract salient targets. Furthermore, a multi-level task joint loss function is introduced into the original network loss function, enabling the model to achieve a better balance between fusion quality and detection performance. Experimental results show that DC-DDNet outperforms several mainstream algorithms on the M3FD public dataset, demonstrating good generalization ability. This invention has a reasonable structural design and is suitable for the field of image fusion technology.

Claims

1. A visible light and infrared image fusion method based on dual-domain collaborative detection, characterized in that, Includes the following steps: Step 1: Construct the dual-domain collaborative detection driving network DC-DDNet. DC-DDNet consists of a spatial domain feature extraction module, a frequency domain feature extraction module, and a target perception dual-domain attention guidance module. Each module takes infrared images and visible light images as input. Step 2: The target perception dual-domain attention guidance module adopts a dual-branch mask generation strategy, generating independent masks for the infrared branch and the visible light branch respectively, and then fusing them through channel-cascaded convolution to obtain the target mask. ; Step 3: The spatial domain feature extraction module first extracts infrared feature maps with complementary information. and visible light feature map Then, respectively with the target mask Perform bit-by-bit multiplication to obtain the masked enhanced infrared feature map. With mask-enhanced visible light feature map ; Step 4: The frequency domain feature extraction module obtains the modal complementarity amplitude spectrum through fast Fourier transform and encoding / decoding. and phase spectrum Using the target mask Guided amplitude spectrum Frequency band selection and phase spectrum The target region is enhanced, and then subjected to inverse Fourier transform to obtain a mask-enhanced frequency domain feature map; Step 5: Enhance the infrared feature map using the mask. Mask-enhanced visible light feature map The mask-enhanced frequency domain feature map is stitched together along the channel to perform cross-domain fusion and obtain a fused image. .

2. The visible light and infrared image fusion method based on dual-domain collaborative detection as described in claim 1, characterized in that, In step 2, the target mask is obtained. The steps are as follows: Step 21: Introduce the pre-trained YOLOv5 Backbone main architecture network and corresponding parameters through transfer learning; Step 22: A target mask generation method based on multi-scale feature fusion is adopted to extract feature maps from different layers of the backbone to obtain multi-scale features; Step 23: Use a lightweight feature pyramid network for multi-scale fusion and adjust it to match the size of the original image to obtain the infrared fused features. Features of fusion with visible light ; Step 24: Use a lightweight convolutional module to obtain single-channel probability masks for infrared and visible light. and ; Step 25, and By cascading along the channels and then processing with a 1×1 convolutional layer and a sigmoid function, a single-modal target mask is obtained. , is represented as: Step 26: During the training phase, mask the target using the real target bounding boxes output by the target detector. To exert oversight.

3. The visible light and infrared image fusion method based on dual-domain collaborative detection as described in claim 2, characterized in that, In step 23, the infrared fusion feature is obtained. Features of fusion with visible light The method is as follows: The high-level and low-level features are fused layer by layer using a top-down approach. First, the highest-level features are compressed by a 1×1 convolution, then upsampled to the same size as the feature map of the previous layer using bilinear interpolation, and added element-wise to the previous layer's features. Next, the features of the previous layer are compressed by a 1×1 convolution, upsampled to the same size as the feature map of the layer above that, and added element-wise to the layer above that, and so on, to obtain the preliminary fused features. Finally, the preliminary fused features are upsampled to the same size as the original image using bilinear interpolation to obtain the infrared fused features. Features of fusion with visible light Its channel dimension contains target semantic information.

4. The visible light and infrared image fusion method based on dual-domain collaborative detection driving according to claim 2, characterized in that, In step 24, single-channel probability masks for infrared and visible light are obtained through two layers of 3×3 convolution, ReLU activation function, and 1×1 convolution. and The formula is as follows: In the formula, The sigmoid function ensures that the mask value is between 0 and 1. To refer to parameters, that is or ; In step 26, the loss function is the binary cross-entropy, and the formula is: in, Indicates the target area. Indicates the background area. For the total number of training rounds, For the first The target mask obtained from the training.

5. The visible light and infrared image fusion method based on dual-domain collaborative detection driving according to claim 1, characterized in that, The spatial domain feature extraction module outputs infrared feature maps with complementary information through the combined action of the AFCB network, edge attention mechanism, and convolution. and visible light feature map The AFCB network performs the following steps: Step 31: First, a 3×3 convolutional kernel combined with a parameterized ReLU activation function is used to perform feature encoding on the original bimodal input image. The output is then subjected to global average pooling. Finally, the feature map is compressed into a 1×1×C vector, where C is the number of channels. Step 32: Perform FFN1 operation on the output 1×1×C vector. FFN1 consists of a fully connected layer and a ReLU activation function, which reduces the number of channels from C to C / r, where r is the dimensionality reduction factor. Step 33: Perform FFN2 operation on the dimensionality-reduced vector. FFN2 consists of a fully connected layer and a sigmoid activation function. The fully connected layer restores the number of feature channels from C / r to the original dimension C, and then applies the sigmoid activation function to generate channel attention weights. Step 34: Perform a dot product operation between the generated channel attention weights and the original input image to weight the feature map, thereby enhancing important features and suppressing irrelevant features.

6. The visible light and infrared image fusion method based on dual-domain collaborative detection driven according to claim 1 or 5, characterized in that, In step 3, the target mask is... Copy along channel dimension Next, obtain the adjusted mask. ,right and Spatial attention weighting is performed separately, and the formula is as follows: in, This represents element-wise multiplication; in the visible light branch, the target region... Enhance visible light texture features; in the infrared branch, background areas This enhances the infrared thermal target characteristics.

7. The visible light and infrared image fusion method based on dual-domain collaborative detection as described in claim 1, characterized in that, Step 4 utilizes the target mask. Guided amplitude spectrum The frequency band selection is implemented as follows: target mask The frequency domain mask is obtained by performing a fast Fourier transform. Then, the frequency domain mask is applied through a 1×1 convolutional layer and a sigmoid activation function. The sum of the real and imaginary parts is processed to generate the frequency band selection matrix. The formula is: Using the generated frequency band selection matrix Amplitude spectrum Element-wise multiplication yields the optimized amplitude spectrum. .

8. The visible light and infrared image fusion method based on dual-domain collaborative detection as described in claim 1, characterized in that, Step 4 utilizes the target mask. phase spectrum The method for target region enhancement is as follows: Introduce a learnable parameter It is initialized and its value is dynamically adjusted through training to optimize the phase spectrum. Expressed as a formula: 。 9. The visible light and infrared image fusion method based on dual-domain collaborative detection driving according to claim 1, characterized in that, Step 5 introduces a multi-task joint optimization loss function, the total loss function. Represented as: In the formula It is a dynamic weighting coefficient. For fusion losses, including content losses Structural similarity loss Frequency domain fusion loss , To detect loss, including classification loss. Location loss and confidence loss ; The content loss The formula for the double-constraint content loss is as follows: in For pixel intensity loss, For gradient loss, and For learnable parameters, The result of fusing real infrared and visible light images. For the Sobel gradient operator, It is an L1 norm; The structural similarity loss The formula is as follows: in The original infrared image, This is the original visible light image; The frequency domain fusion loss The formula is as follows: in This is the result obtained by converting the fused amplitude spectrum and phase spectrum back to the spatial domain using inverse Fourier transform. , The amplitude spectrum after fusion. The phase spectrum after fusion. This represents the inverse fast Fourier transform. , Representing variables I The correlation coefficient between infrared and visible light images This represents the regularization term used to reduce noise. To control the hyperparameters of regularization strength, yes and Covariance between and yes and Standard deviation; For a batch One sample, and The classification loss has several categories. The classification loss is the average of the binary cross-entropy loss for all samples across all classes. The formula is expressed as Here, we assume that for a given sample, the true class label is... This indicates whether the sample belongs to the first... The model predicts that the sample belongs to the class. The probability of class is ,but For a single sample in the first The binary cross-entropy loss on the class is , The model represents the first i The predicted probability values ​​for each category, Indicates the first n The sample, the first i Real labels for each category Indicates the first n The sample, the first i The predicted probability values ​​for each category; For a batch A predicted bounding box, the localization loss CIoU loss for all predicted boxes The average value is calculated using the following formula: in Center point of the prediction box and the center point of the true frame The square of the Euclidean distance between them For intersection, union, and comparison, c The diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted and ground truth boxes. As a balance factor, v This is a parameter used to measure the aspect ratio consistency between the predicted bounding box and the ground truth bounding box; For a batch A prediction box, the confidence loss Confidence loss for all predicted boxes The average value is given by the formula: in For the confidence loss of a single predicted bounding box, The true confidence level label indicates whether the predicted bounding box contains the target. The predicted bounding box for the model includes the confidence level of the target.

10. The visible light and infrared image fusion method based on dual-domain collaborative detection driving according to claim 9, characterized in that, In step 5, a lightweight detection sub-network, MobileNetV3-YOLOv5s, is introduced during the training phase. The input to this network is the fused image. The output is the detection loss. ; The training strategy employed sets dynamic weight coefficients. , The adjustment is automatically based on the ratio of detection loss to fusion loss, using the following formula: in For hyperparameters, Used to prevent division by zero errors.

Images