Image fusion method based on texture and edge information enhancement of degraded image

Abstract

The application discloses an image fusion method based on texture and edge information enhancement of degraded images, and relates to the technical field of image processing and computer vision. In the application, Y, Cb and Cr channel images of a to-be-processed degraded image are acquired; the Y channel image and the corresponding infrared image are input into an image fusion network model, and are forward propagated once to obtain a Y channel fusion image containing high-level semantic information; the Cb and Cr channel images and the Y channel fusion image are spliced and converted into an RGB format to obtain a color fusion image; the model is obtained by training an image fusion network; the network comprises a texture and edge information enhancement branch; the branch enhances edge information and texture information of the Y channel image to obtain a fine-grained structure feature map containing rich texture information and edge information. The color fusion image obtained by the method has richer texture detail information and clearer edge contours.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and specifically relates to an image fusion method based on texture and edge information enhancement of degraded images. Background Technology

[0002] Currently, methods for fusing visible light and infrared images mainly fall into two categories: traditional image fusion methods and deep learning-based methods. However, both traditional and existing deep learning-based image fusion methods perform poorly when fusing visible light and infrared images in degraded scenes. This is primarily because the texture and edge information of visible light images in degraded scenes is severely weakened by interference from lighting, fog, rain, etc.; while infrared images, although providing clear target thermal radiation contours, lack texture and edge information, resulting in generally poor-quality fused images. Therefore, this application proposes an image fusion method based on texture and edge information enhancement for degraded images. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides an image fusion method based on texture and edge information enhancement of degraded images.

[0004] The technical solution of this invention is as follows: An image fusion method based on texture and edge information enhancement of degraded images includes the following steps: The degraded image to be processed is converted into a YCbCr image. Then, the Y-channel, Cb-channel, and Cr-channel images of the visible light image are separated to obtain the Y-channel, Cb-channel, and Cr-channel images of the degraded image to be processed. The Y-channel image of the degraded image to be processed and its corresponding infrared image are input into the image fusion network model and propagated forward once to obtain the Y-channel fused image containing high-level semantic information. Then, the Y-channel fused image, the Cb-channel image of the degraded image to be processed, and the Cr-channel image of the degraded image to be processed are concatenated along the channel dimension. After concatenation, the image is converted to RGB format to obtain the color fused image. The image fusion network model is obtained by training an image fusion network; The image fusion network includes texture and edge information enhancement branches. These branches consist of a convolutional layer, a patch embedding layer, a Swing Transformer block, a first normalization layer, a patch inverse embedding layer, a first residual connection layer, an upsampling layer, and a Sigmoid layer, all connected in sequence. The Swing Transformer block comprises a second normalization layer, a window-based multi-head self-attention layer, a second residual connection layer, a third normalization layer, a multilayer perceptron layer, and a third residual connection layer, all connected in sequence. The output of the patch embedding layer is connected to the inputs of the first and second residual connection layers, respectively, and the output of the third normalization layer is connected to the input of the third residual connection layer. The texture and edge enhancement branch enhances the Y-channel image of the degraded image with edge and texture information, resulting in a fine-grained structural feature map containing rich texture and edge information. Specifically, in the texture and edge enhancement branch, convolutional layers perform preliminary feature extraction on the Y-channel image of the degraded image, obtaining a spatial feature map with basic structural features. A patch embedding layer converts the spatial feature map into a patch sequence feature. The patch sequence feature is then input into the Swing Transformer block. In the Transformer block, the second normalization layer normalizes the tile sequence features; the window-based multi-head self-attention layer divides the feature map output by the normalization layer into local windows of fixed size (8×8 in this embodiment), performs multi-head self-attention calculation within each local window, and introduces a relative position bias matrix to enhance the positional encoding of the attention weights, thereby learning and enhancing the edge information of the image and outputting a feature map with enhanced local attention; the second residual connection layer performs residual connection on the feature map output by the window-based multi-head self-attention layer and the tile sequence features output by the tile embedding layer to obtain a feature map with rich detail information; the third normalization layer normalizes the feature map output by the second residual connection layer, and the multilayer perceptron performs nonlinear feature mapping and high-order feature extraction operations on the feature map output by the third normalization layer to obtain a feature map with global semantic association information; the third residual connection layer is used to normalize the feature map output by the second residual connection layer. The feature maps output from the three normalization layers and the feature map output from the multilayer perceptron are residually connected to obtain a feature map with global semantic association information and rich detail information. The feature map output from the third residual connection layer is input to the first normalization layer in the texture and edge information enhancement branch. The first normalization layer normalizes the feature map output from the third residual connection layer. The patch inversion layer performs sequence feature reshaping on the feature map output from the first normalization layer to obtain a two-dimensional spatial feature map with rich local spatial information. The first residual connection layer performs residual connection on the feature maps output from the patch embedding layer and the patch inversion layer to obtain a feature map with key features (such as texture features, edge features, and contour features). The upsampling layer is used to restore the feature map output from the first residual connection layer to its original size. The sigmoid layer is used to activate the feature map output from the upsampling layer to obtain a fine-grained structural feature map containing rich texture information and rich edge information.

[0005] Preferably, the image fusion network further includes: The visible light information extraction branch is used to extract features from the Y channel image of the degraded image to obtain the initial visible light feature map; The enhanced feature perception module is used to perform channel attention weighting on the initial visible light feature map input to obtain a clean feature map without degradation information; The degradation layer perception module is used to perform channel attention weighting on the initial visible light feature map input to obtain a degradation feature map containing degradation information; The degradation-aware decoding network is used to reconstruct features from clean feature maps and degraded feature maps respectively, resulting in clean feature reconstructed images and degraded feature reconstructed images. The infrared information extraction branch is used to extract thermal radiation information from infrared images to obtain feature maps with thermal radiation information; A cross-modal complementary feature fusion network is used to perform multimodal feature interaction, text semantic modulation, and attention weighted fusion on clean feature reconstructed images, feature maps with thermal radiation information, fine-grained structural feature maps containing rich texture and edge information, and semantic description text to obtain a Y-channel fused image containing high-level semantic information; wherein, the semantic description text is obtained by processing degraded images in the training set using a text generation model.

[0006] Preferably, the degradation-aware decoding network includes an enhancement decoding branch and a degradation feature decoding branch. The input of the enhancement decoding branch is connected to the output of the enhancement feature perception module, and the input of the degradation feature decoding branch is connected to the output of the degradation layer perception module.

[0007] Preferably, in the enhanced feature perception module, two convolutional units A are used to extract clean features from the initial visible light feature map. The global average pooling layer is used to globally average compress the height and width dimensions of the output feature map of the second convolutional unit A in the enhanced feature perception module to obtain a channel-level global feature vector. The Softmax layer is used to normalize the weights of the channel-level global feature vector to obtain clean feature channel attention weights. The element-wise multiplication unit is used to multiply the clean feature channel attention weights with the initial visible light feature map element-wise to achieve channel attention weighting, thereby strengthening clean image features and obtaining a clean feature map without degradation information.

[0008] Preferably, in the degradation layer perception module, two convolutional units A are used to extract degradation interference features from the initial visible light feature map; the global average pooling layer performs global average pooling on the feature map output by the second convolutional unit A in the degradation layer perception module to obtain a channel-level global feature vector; the Softmax layer is used to normalize the weights of the channel-level global feature vector to obtain the degradation feature channel attention weights; and the element-wise multiplication unit is used to multiply the degradation feature channel attention weights with the initial visible light feature map element-wise to achieve channel attention weighting, so as to suppress degradation interference features in the image and obtain a degradation feature map containing degradation information.

[0009] Preferably, the enhanced decoding branch includes three convolutional units A and one convolutional unit B connected in sequence; both convolutional units A and B in the enhanced decoding branch are used for feature extraction; wherein, the LReLU layer in convolutional unit A is used to perform nonlinear feature transformation; the Sigmoid layer in convolutional unit B is used to map feature values ​​to the (0,1) interval to achieve feature reconstruction and obtain a clean feature reconstruction image.

[0010] Preferably, in the degradation feature decoding branch, the first convolutional unit A is used to perform preliminary feature extraction on the degradation feature map containing degradation information output by the degradation layer perception module; the first Concat layer is used to concatenate the feature map output by the first convolutional unit A in the degradation feature decoding branch with the feature map output by the first convolutional unit A in the enhancement decoding branch to obtain a fused complementary structure feature map; in the degradation feature decoding branch, the second convolutional unit A is used to perform deep feature extraction on the feature map output by the first convolutional unit A, and the third convolutional unit A is used to perform refined feature extraction on the feature map output by the second convolutional unit A; in the degradation feature decoding branch, the second Concat layer... The first layer is used to concatenate the feature maps output by the third convolutional unit A in the degradation feature decoding branch and the third convolutional unit A in the enhancement decoding branch, thereby strengthening the representational ability of degradation features and obtaining the final concatenated features. In the degradation feature decoding branch, the convolutional layer in convolutional unit B performs fine reconstruction of the final concatenated features, the batch normalization layer in convolutional unit B performs batch normalization on the feature maps output by the convolutional layers, and the Sigmoid layer performs pixel value normalization on the feature maps output by the batch normalization layer, thereby realizing feature reconstruction and obtaining a degradation feature reconstruction image containing degradation information.

[0011] Preferably, the cross-modal complementary feature fusion network includes an adjusted convolutional layer (the adjusted convolutional layer is a convolutional layer with a kernel size of 1×1) and a lightweight dense connection unit; wherein, the input of the adjusted convolutional layer is connected to the output of the degenerate feature decoding branch, and the output of the adjusted convolutional layer and the output of the infrared information extraction branch are both connected to the input of the lightweight dense connection unit. The lightweight dense connection unit is sequentially connected to the shallow text injection unit, the element-wise multiplication unit, the deep text injection unit, the attention unit, and the fusion decoding unit. The output of the element-wise multiplication unit is also connected to the input of the attention unit. The output of the edge information enhancement branch is connected to the input of the shallow text injection unit, the input of the element-wise multiplication unit, and the input of the deep text injection unit, respectively. In addition, the shallow text injection unit and the deep text injection unit also input semantic description text.

[0012] Preferably, training the image fusion network includes the following steps: training the image fusion network based on the training set and the total loss of the image fusion network to obtain the image fusion network model.

[0013] Preferably, the total loss of the image fusion network includes degradation decoupling loss, overexposure suppression loss, and fusion loss; the overexposure suppression loss uses the pixel-level localized overexposure mask image as a spatial weight constraint, and adaptively divides the overexposure region by combining the dark channel image of the degraded image, and realizes differentiated loss constraint between non-overexposure region and overexposure region through dual thresholds; the dual thresholds include 4 times the dark channel mean and 7 times the dark channel mean.

[0014] Preferably, the overexposure mask image with pixel-level localization is obtained by segmenting the overexposure regions in the degraded images in the training set using a semantic segmentation model.

[0015] Preferably, training the image fusion network includes the following steps: S3-1. Convert all degraded images in the training set into YCbCr images, and then separate the Y channel image, Cb channel image and Cr channel image to obtain the Y channel image, Cb channel image and Cr channel image of the degraded image to be processed. S3-2. Input the infrared image into the infrared information extraction branch of the image fusion network, and input the Y channel image of the degraded image into the visible light information extraction branch and the texture and edge information enhancement branch of the image fusion network respectively. Propagate forward to calculate the total loss of the image fusion network, and backpropagate under the guidance of the total loss of the image fusion network to update the weight parameters of the image fusion network. S3-3. Repeat steps S3-1 and S3-2 until the number of iterations (epochs) reaches the set threshold to obtain the image fusion network model. The process of completing one step S3-1 and S3-2 constitutes one epoch of iteration.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides an image fusion method based on texture and edge information enhancement of degraded images. The texture and edge information enhancement branch constructed in this application can enhance the edge and texture information of the Y channel image of the degraded image, resulting in a fine-grained structural feature map containing rich texture and edge information. The cross-modal complementary feature fusion network reconstructs the image based on clean features, a feature map with thermal radiation information, a fine-grained structural feature map containing rich texture and edge information, and semantic description text. It performs multimodal feature interaction, text semantic modulation, and attention weighted fusion to obtain a Y channel fused image containing high-level semantic information. In addition, the overexposure suppression loss L set in this application over overexposure mask image M located at the pixel level oe As a spatial weight constraint, combined with the dark channel image I of the degraded image. dc The overexposed region is adaptively segmented, and differential loss constraints between non-overexposed and overexposed regions are achieved through dual thresholds (i.e., 4 times the dark channel mean and 7 times the dark channel mean). With the above settings, the degraded image to be processed and its corresponding infrared image can effectively avoid the effects of overexposure when using the image fusion network model obtained in this application to obtain a color fusion image.

[0017] Tests revealed that the color fusion image obtained by the fusion method described in this application contains richer texture details, has clearer edge contours, and is more consistent with human visual perception. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the network structure of the image fusion network in this application; Figure 2 for Figure 1 A schematic diagram showing the connection relationship between the enhanced feature perception module, the degradation layer perception module, and the degradation perception decoding network; Figure 3 for Figure 1 A schematic diagram of the network structure for the mid-texture and edge information enhancement branch; Figure 4 for Figure 3 A schematic diagram of the structure of the Swing Transformer block; Figure 5 for Figure 1 A schematic diagram of the network structure of a mid-to-span modal complementary feature fusion network; Figure 6 for Figure 5 A schematic diagram of the network structure of a shallow to medium-level text injection unit; Figure 7 for Figure 5 A schematic diagram of the network structure of the attention unit; Figure 8 The image is a fused image obtained by the method described in this application and six existing image fusion methods in a low-light scene; wherein, Figure 8 In the image, (a) represents the original infrared image used as input, (b) represents the original visible light image used as input, and (c) to (i) represent the color fusion images obtained by testing visible light and infrared images in low-light scenes using the Tar-DAL method, DIVFusion method, Text-IF method, OmniFuse method, URFusion method, IASSF method, and the method described in this application, respectively. Figure 9 The image fusion method described in the application and six existing image fusion methods were tested on visible light and infrared images in a low-light flare scene to obtain a color fused image; among them, Figure 9 In the image, (a) represents the original infrared image used as input, (b) represents the original visible light image used as input, and (c) to (i) represent the color fusion images obtained by testing visible light and infrared images in low-light scenes using the Tar-DAL method, DIVFusion method, Text-IF method, OmniFuse method, URFusion method, IASSF method, and the method described in this application, respectively. Figure 10 The image fusion method described in this application and six existing image fusion methods are color fused images obtained from tests of visible light and infrared images in foggy scenes; among them, Figure 10 In the image, (a) represents the original infrared image used as input, (b) represents the original visible light image used as input, and (c) to (i) represent the color fusion images obtained by testing visible light and infrared images in low-light scenes using the Tar-DAL method, DIVFusion method, Text-IF method, OmniFuse method, URFusion method, IASSF method, and the method described in this application, respectively. Figure 11 The image fusion method described in this application and six existing image fusion methods are color fused images obtained from tests of visible light and infrared images in a rainy scene; among them, Figure 11 In the image, (a) represents the original infrared image used as input, (b) represents the original visible light image used as input, and (c) to (i) represent the color fusion images obtained by testing visible light and infrared images in low-light scenes using the Tar-DAL method, DIVFusion method, Text-IF method, OmniFuse method, URFusion method, IASSF method, and the method described in this application, respectively. Figure 12 The image fusion method described in this application and six existing image fusion methods are color fused images obtained from tests of visible light and infrared images in blurred scenes; wherein, Figure 12 In the diagram, (a) represents the original infrared image used as input, (b) represents the original visible light image used as input, and (c) to (i) represent the color fusion images obtained by testing visible light and infrared images in low-light scenes using the Tar-DAL method, DIVFusion method, Text-IF method, OmniFuse method, URFusion method, IASSF method, and the method described in this application, respectively. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0020] An image fusion method based on texture and edge information enhancement of degraded images specifically includes the following steps: S1. Obtain the training and test sets; S1-1, Constructing the training set; including the following steps: This application randomly selects 676 pairs of paired infrared images and visible light images under degraded scenes from the EMS dataset, and selects visible light images without degraded scenes corresponding to the visible light images under the degraded scenes; wherein, the degraded scenes include any one of low light scenes, rain scenes, fog scenes, and blurred scenes; the EMS dataset in this application is an existing dataset; Then, from the remaining images in the EMS dataset, 129 pairs of paired infrared images and their corresponding visible light images under degraded scenes are randomly selected, along with visible light images without degraded scenes corresponding to the visible light images under the degraded scenes. The visible light images from the 129 pairs of infrared images and their corresponding visible light images under degraded scenes are used as the original images. For any original image, one flare image can be randomly selected and superimposed on the original image, or multiple flare images can be randomly selected and superimposed on the original image to obtain the visible light images under low-light flare scenes used for training. The flare images superimposed on different original images are different. The infrared images from the 129 pairs of infrared images and their corresponding visible light images under degraded scenes are not processed in any way. Taking the overlay of a flare image onto the original image as an example, the steps include: 1) Flare image preprocessing and flare center calibration: Randomly select one flare image from the flare image dataset and scale it to a set size (set size is 1000×1000); use the interactive mouse to calibrate the coordinates of the flare center in the flare image as the spatial center reference point for flare superposition; 2) Target position calibration of the original image to be superimposed: Import the original images, and for each original image, use the interactive mouse to calibrate the coordinates of the target position (such as headlights, light source area) of the original image to be superimposed, and record the coordinates of the target position; 3) Coordinate conversion and flare region cropping: Combine the target position of the flare to be superimposed on the original image and the coordinates of the flare center, calculate the coordinate offset, and crop the area outside the actual pasting area of ​​the flare on the original image according to the coordinate offset. The actual pasting area of ​​the flare on the original image is referred to as the flare region. The flare region on the original image is cropped to obtain the cropped flare image. 4) Linear brightness processing and overlay: The cropped flare image is filtered by a brightness threshold (threshold is 10), and the cropped flare images with brightness values ​​greater than the brightness threshold are retained to preserve the flare emitting area and suppress the dark area; the cropped flare images with brightness values ​​greater than the brightness threshold are directly linearly added to the corresponding areas of the original image, that is, the goal of overlaying a flare image onto the original image is achieved. In this application, the flare image dataset used is an existing dataset; In this application, when multiple flare images are superimposed onto an original image, the multiple flare images are superimposed onto the same original image one by one, following the steps of superimposing one flare image onto the original image.

[0021] In this application, the selected 676 pairs of paired infrared images, together with visible light images in degraded scenes and visible light images in low-light flare scenes and their corresponding infrared images, as well as the selected visible light images without degraded scenes corresponding to the visible light images in degraded scenes, constitute a training set. In this application, both visible light images in degraded scenes and visible light images in low-light flare scenes are referred to as degraded images; S1-2, Obtain the test set: From the remaining images in the EMS dataset, excluding the visible light images and their corresponding infrared images of the degraded scenes selected in step S1-1, 265 pairs of paired infrared images and their corresponding visible light images of the degraded scenes are randomly selected to form sub-dataset A, and 12 pairs of paired infrared images and their corresponding visible light images of the degraded scenes are selected to form sub-dataset B. Then, the visible light images in sub-dataset B are used as the original images. For any original image, one or more flare images are randomly selected and superimposed onto the original image using the steps described in step S1-1 to obtain the visible light images of the low-light flare scenes used for testing. The flare images superimposed on different original images are different. The infrared images in sub-dataset B are not processed in any way. The test set is composed of all images in subset A, as well as the visible light images and their corresponding infrared images in the low-light flare scene used for testing. S1-2. Using existing text generation models, generate semantic description texts for degraded images in the training set (i.e., visible light images in degraded scenes and visible light images in low-light flare scenes), and add these semantic description texts to the training set. S1-3. Using existing text generation models, generate semantic description text for degraded images (i.e., visible light images in degraded scenes and visible light images in low-light flare scenes) in the test set, and add these semantic description texts to the test set. In this application, the existing text generation model is one of CLIP, BLIP, and BLIP-2. The CLIP model used in this embodiment has the same structure as the CLIP disclosed in the paper "Learning Transferable Visual Models From Natural Language Supervision". The BLIP model used in this embodiment has the same structure as the BLIP disclosed in the paper "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation". The BLIP-2 model used in this embodiment has the same structure as the BLIP-2 disclosed in the paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models". S1-4. Using existing semantic segmentation models, the overexposed regions in the degraded images (i.e., visible light images in degraded scenes and visible light images in low-light flare scenes) in the training set are segmented to obtain pixel-level localized overexposed mask images, and these pixel-level localized overexposed mask images are added to the training set. S2. Construct an image fusion network; In this application, the image fusion network structure is as follows: Figure 1 As shown, it includes an infrared information extraction branch, a texture and edge information enhancement branch, a visible light information extraction branch, an enhanced feature perception module and a degradation layer perception module connected to the visible light information extraction branch, and a degradation perception decoding network connected to the enhanced feature perception module and the degradation layer perception module; the degradation perception decoding network, the texture and edge information enhancement branch and the infrared information extraction branch are all connected to a cross-modal complementary feature fusion network; The visible light information extraction branch includes four convolutional units A connected in sequence, i.e., the first to the fourth convolutional units A are connected sequentially. All convolutional units A in this application have the same structure. Each convolutional unit A includes a convolutional layer (with a kernel size of 3×3), a batch normalization layer, and an LReLU layer connected in sequence. In this application, the visible light information extraction branch is used to extract features from the Y channel image of the degraded image to obtain an initial visible light feature map. In this application, the output of the LReLU layer of the last convolutional unit A (i.e., the fourth convolutional unit A) in the visible light information extraction branch is connected to the input of the enhancement feature perception module and the degradation layer perception module, respectively. The enhanced feature perception module includes two convolutional units A, a global average pooling layer, a softmax layer, and an element-wise multiplication unit connected in sequence. The input of the first convolutional unit A in the enhanced feature perception module is also connected to the input of the element-wise multiplication unit. In this application, the enhanced feature perception module is used to perform channel attention weighting on the initial visible light feature map as input to obtain a clean feature map without degradation information. Specifically, the two convolutional units A in the enhanced feature perception module are used to extract clean features from the initial visible light feature map. The global average pooling layer is used to perform global average compression on the height and width dimensions of the output feature map of the second convolutional unit A in the enhanced feature perception module to obtain a channel-level global feature vector. The Softmax layer is used to normalize the weights of the channel-level global feature vector to obtain clean feature channel attention weights. The element-wise multiplication unit is used to multiply the clean feature channel attention weights with the initial visible light feature map element-wise to achieve channel attention weighting, thereby enhancing clean image features and obtaining a clean feature map without degradation information. In this application, the network structures of the enhanced feature perception module and the degenerate layer perception module are the same; during the training of the image fusion network using the training set, the weight parameters of the enhanced feature perception module and the degenerate layer perception module are not shared, and the enhanced feature perception module and the degenerate layer perception module are constrained by different loss functions, so that the enhanced feature perception module and the degenerate layer perception module can learn different weight parameters autonomously. The degradation layer perception module performs channel attention weighting on the initial visible light feature map as input to obtain a degradation feature map containing degradation information. Specifically, the two convolutional units A of the degradation layer perception module are used to extract degradation interference features from the initial visible light feature map. The global average pooling layer performs global average pooling on the feature map output by the second convolutional unit A in the degradation layer perception module to obtain a channel-level global feature vector. The Softmax layer is used to normalize the weights of the channel-level global feature vector to obtain the degradation feature channel attention weights. The element-wise multiplication unit is used to multiply the degradation feature channel attention weights with the initial visible light feature map element-wise to achieve channel attention weighting, thereby suppressing degradation interference features in the image and obtaining a degradation feature map containing degradation information. Furthermore, during the training of the image fusion network using the training set, the weights of the enhancement decoding branch and the degradation feature decoding branch are not shared; the weight parameters of the visible light information extraction branch, infrared information extraction branch, texture and edge information enhancement branch in the image fusion network, as well as the text semantic modulation module and attention unit in the cross-modal complementary feature fusion network, are also not shared. Among them, the visible light information extraction branch, infrared information extraction branch, texture and edge information enhancement branch, as well as the text semantic modulation module and attention unit in the cross-modal complementary feature fusion network, can independently learn the weight parameters of their respective functions.

[0022] In this application, the structure of the degradation-aware decoding network is as follows: Figure 2 As shown; the degradation-aware decoding network includes an enhancement decoding branch and a degradation feature decoding branch. The input of the enhancement decoding branch is connected to the output of the enhancement feature perception module, and the input of the degradation feature decoding branch is connected to the output of the degradation layer perception module. The degradation-aware decoding network is used to perform feature reconstruction on the clean feature map and the degradation feature map respectively, to obtain a clean feature reconstruction image and a degradation feature reconstruction image. In this application, during the training of the image fusion network using the training set, the degradation decoupling loss L is obtained based on the clean feature reconstruction image and the degradation feature reconstruction image. decom And utilize decoupling loss L decom To constrain the degradation separation effect of the image fusion network; The enhanced decoding branch includes three convolutional units A and one convolutional unit B connected in sequence. The clean feature map with degraded information is input into the enhanced decoding branch. Both convolutional units A and B in the enhanced decoding branch are used for feature extraction. The LReLU layer in convolutional unit A is used to perform nonlinear feature transformation. The Sigmoid layer in convolutional unit B is used to map feature values ​​to the (0,1) interval to achieve feature reconstruction and obtain a clean feature reconstruction image. The degenerate feature decoding branch includes three convolutional units A, two concatenated layers, and one convolutional unit B. The first convolutional unit A in the degenerate feature decoding branch and the first convolutional unit A in the enhancement decoding branch are both connected to the first concatenated layer in the degenerate feature decoding branch. The first concatenated layer is connected sequentially to the second and third convolutional units A in the degenerate feature decoding branch. The third convolutional unit A in the degenerate feature decoding branch and the third convolutional unit A in the enhancement decoding branch are both connected to the second concatenated layer in the degenerate feature decoding branch. The second concatenated layer is connected to convolutional unit B in the degenerate feature decoding branch. Convolutional unit B includes a convolutional layer (with a 3×3 kernel size), a batch normalization layer, and a sigmoid layer connected sequentially. The input to the degradation feature decoding branch is a degradation feature map containing degradation information. In this branch, the first convolutional unit A performs preliminary feature extraction on the input degradation feature map containing degradation information. The first Concat layer in the degradation feature decoding branch concatenates the feature map output by the first convolutional unit A in the degradation feature decoding branch with the feature map output by the first convolutional unit A in the enhancement decoding branch, obtaining a fused complementary structure feature map. In the degradation feature decoding branch, the second convolutional unit A performs depth feature extraction on the feature map output by the first convolutional unit A. In the degradation feature decoding branch, the third convolutional unit A is used to refine the feature map output by the second convolutional unit A. The second Concat layer in the degradation feature decoding branch... The first layer is used to concatenate the feature maps output by the third convolutional unit A in the degradation feature decoding branch and the third convolutional unit A in the enhancement decoding branch, thereby strengthening the representational ability of degradation features and obtaining the final concatenated features. In the degradation feature decoding branch, the convolutional layer in convolutional unit B performs fine reconstruction of the final concatenated features, the batch normalization layer in convolutional unit B performs batch normalization on the feature maps output by the convolutional layers, and the Sigmoid layer performs pixel value normalization on the feature maps output by the batch normalization layer, thereby realizing feature reconstruction and obtaining a degradation feature reconstruction image containing degradation information.

[0023] In this application, the structure of the texture and edge information enhancement branches is as follows: Figure 3 As shown, the texture and edge information enhancement branch includes a convolutional layer, a tile embedding layer, a Swing Transformer block, a first normalization layer, a tile inverse embedding layer, a first residual connection layer, an upsampling layer, and a Sigmoid layer connected in sequence. The structure of the Swin Transformer block is as follows: Figure 4As shown, the Swing Transformer block includes a second normalization layer, a window-based multi-head self-attention layer, a second residual connection layer, a third normalization layer, a multilayer perceptron layer, and a third residual connection layer connected in sequence. The output of the tile embedding layer is connected to the inputs of the first and second residual connection layers, respectively, and the output of the third normalization layer is connected to the input of the third residual connection layer. The window-based multi-head self-attention layer used in this application has the same structure as the multi-head self-attention modules disclosed in the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". The residual connection layer used in this application is used to implement residual connections, and the residual connection method is the same as the residual connection method disclosed in the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". The multilayer perceptron used in this application has the same structure as the MLP disclosed in the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". In this application, the texture and edge information enhancement branch is used to enhance the edge and texture information of the Y channel image of the degraded image, so as to obtain a fine-grained structural feature map containing rich texture and edge information; In the texture and edge information enhancement branch, the convolutional layer is used to perform preliminary feature extraction on the Y channel image of the degraded image to obtain a spatial feature map with basic structural features; the patch embedding layer is used to convert the spatial feature map into patch sequence features; The tile sequence features are input into the Swin Transformer block. Within the Swin Transformer block, the second normalization layer normalizes the tile sequence features. A window-based multi-head self-attention layer divides the feature map output from the normalization layer into fixed-size (8×8 in this embodiment) local windows. Multi-head self-attention calculation is performed within each local window, and a relative position bias matrix is ​​introduced to enhance the positional encoding of the attention weights, thereby learning and enhancing the image's edge information and outputting a locally enhanced attention feature map. The second residual connection layer performs residual connections between the feature map output from the window-based multi-head self-attention layer and the tile sequence features output from the tile embedding layer, resulting in a feature map with rich detail information. The third normalization layer normalizes the feature map output from the second residual connection layer. The multilayer perceptron performs nonlinear feature mapping and high-order feature extraction operations on the feature map output from the third normalization layer, resulting in a feature map with global semantic association information. The third residual connection layer performs residual connections between the feature map output from the third normalization layer and the feature map output from the multilayer perceptron, resulting in a feature map with both global semantic association information and rich detail information. The feature map output from the third residual connection layer is input to the first normalization layer in the texture and edge information enhancement branch; the first normalization layer normalizes the feature map output from the third residual connection layer; the patch inversion layer performs sequence feature reshaping on the feature map output from the first normalization layer to obtain a two-dimensional spatial feature map with rich local spatial information; the first residual connection layer performs residual connection on the feature maps output from the patch embedding layer and the patch inversion layer to obtain a feature map with key features (such as texture features, edge features, and contour features); the upsampling layer restores the feature map output from the first residual connection layer to its original size; the sigmoid layer performs feature activation on the feature map output from the upsampling layer to obtain a fine-grained structural feature map containing rich texture and edge information.

[0024] In this application, the infrared feature extraction branch includes a convolutional layer (with a kernel size of 3×3) and an LReLU layer connected in sequence; the infrared information extraction branch is used to extract thermal radiation information from the infrared image to obtain a feature map with thermal radiation information.

[0025] In this application, the structure of the cross-modal complementary feature fusion network is as follows: Figure 5As shown, the cross-modal complementary feature fusion network includes an adjusted convolutional layer (using a 1×1 kernel) and lightweight dense connection units. The input of the adjusted convolutional layer is connected to the output of the degenerate feature decoding branch. The outputs of the adjusted convolutional layer and the infrared information extraction branch are both connected to the input of the lightweight dense connection units. These units sequentially connect to the shallow text injection unit, the element-wise multiplication unit, the deep text injection unit, the attention unit, and the fusion decoding unit. The output of the element-wise multiplication unit is also connected to the input of the attention unit. The output of the edge information enhancement branch is connected to the inputs of the shallow text injection unit, the element-wise multiplication unit, and the deep text injection unit, respectively. Furthermore, the shallow and deep text injection units also receive semantic description text as input. The lightweight densely connected unit comprises four convolutional units C and three concat layers; wherein the first concat layer, the first convolutional unit C, the second convolutional unit C, the second concat layer, the third convolutional unit C, the third concat layer, and the fourth convolutional unit C are connected sequentially; the input of the first concat layer is connected to the output of the adjustment convolutional layer and the output of the infrared information extraction branch, the output of the first convolutional unit C is connected to the input of the second concat layer, the outputs of the first and second convolutional units C are both connected to the input of the third concat layer, and the output of the fourth convolutional unit C is connected to the input of the shallow text injection unit; in this application, each convolutional unit C includes a convolutional layer (with a kernel size of 3×3) and an LReLU layer connected sequentially; In this application, the structure of the shallow text injection unit is as follows: Figure 6 As shown, the shallow text injection unit includes two serialization layers, one CLIP encoding layer, one convolutional layer, two linear layers, and one LReLU layer. The first serialization layer, CLIP encoding layer, convolutional layer (with a 3×3 kernel), first linear layer, second serialization layer, LReLU layer, and second linear layer are sequentially connected. The output of the second linear layer is the output of the shallow text injection unit. In this application, the deep text injection unit has the same structure as the shallow text injection unit. The CLIP encoding layer used in this application has the same structure as the text encoder disclosed in the paper "Learning Transferable Visual Models From Natural Language Supervision". In this application, the structure of the attention unit is as follows: Figure 7As shown, the attention unit includes a query convolutional layer (using a 1×1 kernel), a key convolutional layer (using a 1×1 kernel), a value convolutional layer (using a 1×1 kernel), a scaling and summing layer, a linear layer, a concat layer, and two element-wise multiplication units. The input of the query convolutional layer is connected to the output of the element-wise multiplication unit in the cross-modal complementary feature fusion network. The inputs of the key and value convolutional layers are both connected to the output of the deep text injection unit. The outputs of the query and key convolutional layers are connected to the output of the first element-wise multiplication unit in the attention unit. The inputs are connected. The features output by the first element-wise multiplication unit are first summed along the channel dimension, and then scaled by a scaling factor to obtain the attention weights for each spatial location. The attention weights for each spatial location and the value features output by the value convolutional layer are input into the second element-wise multiplication unit for element-wise multiplication. The feature map output by the second element-wise multiplication unit is summed along the head dimension to complete the attention weighted aggregation and obtain the aggregated features. The aggregated features are input into the linear layer. The output of the linear layer and the output of the deep text injection unit are both connected to the input of the Concat layer in the attention unit. The output of the Concat layer is connected to the fusion convolutional layer. The fusion convolutional layer uses a convolutional layer with a kernel size of 1×1. In this application, the structure of the fusion decoding unit is as follows: Figure 5 As shown, it includes a convolutional unit C, a convolutional layer, and a sigmoid layer, which are connected sequentially; the input of the convolutional unit C in the fusion decoding unit is connected to the output of the attention unit. In this application, the function of the cross-modal complementary feature fusion network is as follows: A cross-modal complementary feature fusion network is used to perform multimodal feature interaction, text semantic modulation and attention weighted fusion on clean feature reconstruction images, feature maps with thermal radiation information, fine-grained structural feature maps containing rich texture information and rich edge information, and semantic description text to obtain a Y-channel fused image containing high-level semantic information. Among them, the adjustment convolutional layer is used to adjust the channel dimension of the image reconstructed from the clean features, so as to obtain a clean feature map that matches the feature map channel with thermal radiation information; In the lightweight dense connection unit, the first Concat layer is used to concatenate the feature map at the output of the adjusted convolutional layer and the feature map with thermal radiation information in the channel dimension to obtain a concatenated feature map; the first to fourth convolutional units C are all used for feature extraction and non-linear activation; the second and third Concat layers are both used for feature concatenation; the setting of the lightweight dense connection unit in this application enables the reconstruction of images based on clean features and the feature map with thermal radiation information to achieve multi-scale interaction and fusion of multimodal features; The shallow text injection unit performs text semantic modulation based on the semantic description text, the feature map output by the lightweight dense connection unit, and the fine-grained structural feature map containing rich texture and edge information output by the edge information enhancement branch, obtaining modulation parameters γ and β. Specifically: In the shallow text injection unit, the first serialization layer converts the semantic description text into a text sequence. The CLIP encoding layer maps the text sequence to fixed-dimensional (512-dimensional) text semantic guidance features. The convolutional layer expands the dimensions of the text semantic guidance features, outputting a feature map with global semantic information. The first linear layer performs feature upscaling on the feature map output by the convolutional layer. The second serialization layer reshapes the dimensions of the feature map output by the first linear layer. The LReLU layer performs non-linear activation on the feature map output by the second serialization layer, outputting a feature map with enhanced global semantic information. The second linear layer performs parameter mapping on the feature map output by the LReLU layer, outputting modulation parameters γ and β, where γ is a scaling factor and β is an offset factor. Then, the modulation parameters γ and β are used to perform text semantic modulation on the fine-grained structural feature map containing rich texture and edge information, outputting a feature map with enhanced text semantic information. The method of using modulation parameters γ and β to perform text semantic modulation on the fine-grained structural feature map containing rich texture and edge information is similar to that in the paper "Text-IF: Leveraging Semantic Text". The computation method is the same as that of the semantic interaction guidance module (i.e., the Semantic Interaction Guidance Module) disclosed in "Guidance for Degradation-Aware and Interactive Image Fusion"; The element-wise multiplication unit in the cross-modal complementary feature fusion network is used to perform element-wise multiplication on the feature map with enhanced text semantic information output by the shallow text injection unit and the fine-grained structural feature map with rich texture and edge information output by the texture and edge information enhancement branch, so as to obtain a fine-grained structural feature map with further enhanced edge and texture semantic information. The deep text injection unit performs text semantic modulation based on the semantic description text, the feature map output by the element-wise multiplication unit in the cross-modal complementary feature fusion network, and the fine-grained structural feature map containing rich texture and edge information output by the edge information enhancement branch; the deep text injection unit performs semantic modulation in the same way as the shallow text injection unit. In the attention unit, the query convolutional layer extracts features from the feature map output by the element-wise multiplication unit, outputting query features; the key convolutional layer extracts features from the feature map output by the deep text injection unit, outputting key features; the value convolutional layer extracts features from the feature map output by the deep text injection unit, outputting value features; the first element-wise multiplication unit performs element-wise multiplication of the query and key features. The features output by the first element-wise multiplication unit are first summed along the channel dimension, and then scaled by a scaling factor to obtain the attention weights at each spatial location; the attention weights at each spatial location and the value features output by the value convolutional layer are input into the second element-wise multiplication unit for element-wise multiplication. The first element-wise multiplication unit performs element-wise multiplication of the value features with the attention weights at each spatial location. The feature maps output by the second element-wise multiplication unit are summed along the head dimension to complete the attention weighted aggregation and obtain aggregated features. The aggregated features are input to a linear layer, which performs feature space mapping on the aggregated features. The Concat layer concatenates the feature maps output by the linear layer and the feature maps output by the deep text injection unit to further enhance the global semantic information. The fusion convolutional layer is used to perform channel compression and linear integration on the feature maps output by the Concat layer, outputting a Y-channel fused image containing high-level semantic information. In the fusion decoding unit, convolutional unit C performs channel dimensionality reduction and feature extraction on the Y-channel fused image output by the fusion convolutional layer, which contains high-level semantic information, to obtain a feature map with enhanced details; the convolutional layer performs convolution operation on the feature map output by convolutional unit C to obtain a feature map with rich edge information and clear texture; the sigmoid layer is used to perform numerical normalization on the feature map output by the convolutional layer to obtain the Y-channel fused image containing high-level semantic information.

[0026] S3, the total loss L based on the training set and the image fusion network total Training an image fusion network to obtain an image fusion network model includes the following steps: S3-1. Convert the visible light images in the degraded scene and the visible light images in the low light flare scene in the training set into YCbCr images. Then, separate the Y channel image, Cb channel image and Cr channel image of the visible light image to obtain the Y channel image, Cb channel image and Cr channel image of the degraded image to be processed. S3-2. Input the infrared image into the infrared information extraction branch of the image fusion network. Input the Y channel image of the degraded image into the visible light information extraction branch and the texture and edge information enhancement branch of the image fusion network, respectively. Then perform forward propagation and calculate the total loss L of the image fusion network. total In the total loss L of the image fusion network total Guided by the algorithm, backpropagation is performed to update the weight parameters of the image fusion network; S3-3. Repeat steps S3-1 and S3-2 until the number of iterations (epochs) reaches the set threshold to obtain the image fusion network model. One iteration of steps S3-1 and S3-2 constitutes one epoch. In this embodiment, the set threshold for the number of iterations during the training of the image fusion network is 950.

[0027] In this embodiment, the image fusion network is trained using an NVIDIA A100 GPU chip, and the Adam optimizer is used to optimize the loss gradient and perform backpropagation. During training, the batch size is set to 16, and the initial learning rate is set to 1×10⁻⁶. -4 .

[0028] In this application, the total loss L of the image fusion network total Including decoupling loss L decom Overexposure suppression loss L over and fusion loss L fusion The total loss L of the image fusion network total The calculation formula is shown in equation (1): L total =L decom +L over +L fusion (1) In equation (1), L decom To decouple the loss, L over To suppress overexposure loss, L fusion For fusion loss; In equation (1), the decoupling loss L decom Including de-degradation reconstruction loss L recon Degradation consistency loss L cons Perceived loss L per Degradation feature smoothing loss L deg_smooth and structural loss L stu The calculation formula is shown in equation (2): L decom =λ1L recon +λ2L cons +λ3Lper +λ4L deg_smooth +λ5L stu (2) In equation (2), L recon To remove degradation and reconstruct loss, L cons For degradation consistency loss, L per For sensing loss, L deg_smooth For degenerate feature smoothing loss, L stu For structural loss; λ1, λ2, λ3, λ4 and λ5 are all hyperparameters; in this embodiment, the hyperparameters λ1, λ2, λ3, λ4 and λ5 are set to 1000, 7, 40, 4 and 30 respectively; In equation (2), the dedegeneration reconstruction loss L recon The calculation formula is shown in equation (3): (3) In equation (3), H and W represent the height and width of the Y channel image of the degraded image, respectively, and I clean I represents a clean feature-reconstructed image. GT The Y-channel image representing the visible light image that does not exhibit the degraded scene, corresponding to the degraded image. L2 norm calculation, This represents the squared value of the L2 norm calculation result; In equation (2), the degradation consistency loss L cons It is completely consistent with the illumination smoothness loss calculation formula disclosed in the paper "DIVFusion: Darkness-free infrared and visible image fusion"; In equation (2), the perceived loss L per The calculation formula is shown in equation (4): (4) In equation (4), H and W represent the height and width of the Y channel image of the degraded image, respectively, and F vgg I represents the feature extraction function of the conv5_1 layer of the VGG16 network. dg The Y channel image representing the degraded image, I GT F represents the Y channel image of the visible light image that does not have the degraded scene, corresponding to the degraded image. vgg (I dg This indicates that the Y-channel image of the degraded image is extracted using the conv5_1 layer of the VGG16 network. vgg (I GTThis indicates that the conv5_1 layer of the VGG16 network is used to perform feature extraction on the ground truth image of the Y channel of the degraded image. L1 norm calculation; In equation (2), the degradation feature smoothing loss L deg_smooth The calculation formula is shown in equation (5): (5) In equation (5), H and W represent the height and width of the Y channel image of the degraded image, respectively. For horizontal and vertical gradient operators, Reconstruct the image for degraded features, where e represents the base; In equation (2), the structural loss L stu The calculation formula for edge loss is completely consistent with that published in the paper "SLFusion: A Structure-aware Infrared and Visible Image Fusion Network for Low-light Scenes"; The total loss L of the image fusion network constructed in this application is... total Including overexposure suppression loss L over Overexposure suppression loss L over overexposure mask image M located at the pixel level oe As a spatial weight constraint, combined with the dark channel image I of the degraded image. dc Adaptively segmenting overexposed areas and using dual thresholds (i.e., 4 times the dark channel mean and 7 times the dark channel mean) to achieve differentiated loss constraints between non-overexposed and overexposed areas, this application enables the degradation image to be processed and its corresponding infrared image to obtain a color fusion image using the image fusion network model obtained in this application, effectively avoiding the effects of overexposure. In this application, the overexposure suppression loss L over Including mask area loss Loss in non-masked areas Overexposure suppression loss L over The calculation formula is shown in equation (6): (6) In equation (6), For mask area loss, Loss in non-masked areas; In equation (6), the mask region loss The calculation formula is shown in equation (7): (7) In equation (7), Lf L represents the mask region fusion constraint loss. sobel M represents the Sobel gradient loss over the masked region. oe This represents the overexposure mask image with pixel-level localization, and ε represents the smoothing factor to prevent the denominator from being zero; the value of ε is 10. -6 E() represents the mean operation; λ6, λ7, and λ8 are hyperparameters used to adjust the mask region fusion constraint loss L. f Sobel gradient loss L in the mask region sobel In this embodiment, the hyperparameter λ6 is 0.1 and the product of the number of training epochs (i.e., the number of eopch times) is used as a dynamic weighting term, while λ7 and λ8 are 0.5 and 0.8 respectively. In equation (7), the mask region fusion constraint loss L f The calculation formula is shown in equation (8): (8) In equation (8), , For adaptive fusion weights, L MSE ( ) indicates the calculation of mean square error, I fu This represents a Y-channel fused image containing advanced semantic information. For regional reference, Iir is the visible light image; Iir is the infrared image. In equation (8), the regional reference visible light image is used. The method for obtaining is shown in equation (9): (9) In equation (9), I ir For infrared images, I vi To reconstruct the image with clean features, I dc For the dark channel image of the degraded image, μ dc The global mean of the dark channel in the degraded image; This indicates that the pixel value at each pixel location in the dark channel image is greater than the threshold 7μ. dc The area, 7μ dc This represents 7 times the dark channel mean. In equation (9), the global mean of the dark channel is μ dc The calculation formula is shown in equation (10): (10) In equation (10), H and W represent the height and width of the dark channel image of the degraded image, respectively, and I dc (i,j) represents the position of a single pixel in the dark channel image with horizontal and vertical coordinates of (i,j); In equation (7), the Sobel gradient loss L in the masked regionsobel The calculation formula is shown in equation (11): (11) In equation (11), L MSE ( ) indicates the mean square error calculation, G fu G represents the fused feature gradient map. max Represents the target reference gradient map; In equation (11), the fused feature gradient map G fu Target reference gradient map G max The methods for obtaining are shown in equations (12) and (13) respectively: (12) (13) In equation (12), For the Sobel gradient operator, I fu This represents a Y-channel fused image containing advanced semantic information; In equation (13), max{} represents the maximum element selection, G vi To provide a regional reference visible light feature gradient map, G ir Represents the infrared feature gradient map; In equation (13), the regional reference visible light characteristic gradient map G vi Infrared feature gradient map G ir The methods for obtaining are shown in equations (14) and (15): (14) (15) In equation (14), This represents the Sobel gradient operator. This represents a regional reference visible light image; In equation (15), I represents the Sobel gradient operator. ir Represents an infrared image; In equation (6), the loss of the non-masked region The calculation formula is shown in equation (16): (16) In equation (16), For non-masked region fusion constraint loss, For the Sobel gradient loss in the non-masked region, M oe This represents the overexposure mask image with pixel-level localization, and ε represents the smoothing factor to prevent the denominator from being zero; the value of ε is 10. -6E() represents the mean operation; λ8 and λ9 are both hyperparameters. λ8 and λ9 are used to adjust the weights of the non-masked region fusion constraint loss and the non-masked region Sobel gradient loss. In this embodiment, λ8 in formula (7) has the same meaning and value as λ8 in formula (16). λ9 represents the weighting coefficient of the non-masked region Sobel gradient loss. In this application, the loss of the non-masked region is calculated. When, the dynamic weighting term shown in λ6 in formula (7) is not used, but only the weights of the non-masked region fusion constraint loss and the non-masked region Sobel gradient loss are adjusted by λ8 and λ9 to adapt to the suppression requirements of overexposed regions during training. In this application, the non-mask region fusion constraint loss The calculation method and the constraint loss L of the mask region fusion f The only difference is that the visible light characteristics are referenced in different regions. In the acquisition method, the pixel value at each pixel position in the dark channel image is greater than the threshold 4μ. dc The area (i.e.) Infrared image I ir Replace, remaining areas I vi Replacement is performed to reconstruct the image with clean features; 4μ dc This indicates that the non-masked areas use 4 times the dark channel mean. In this application, the non-masked region Sobel gradient loss described in equation (16) is... The calculation method is the same as the Sobel gradient loss L in the mask region in equation (11). sobel The calculation method is the same.

[0029] In equation (1), the fusion loss L fusion Including strength loss L max Structural similarity loss L ssim Color consistency loss L color and gradient fidelity loss L grad Fusion loss L fusion The calculation formula is shown in equation (17): (17) In equation (17), L max For strength loss, L ssim For structural similarity loss, L color For color consistency loss, L grad For gradient-fidelity loss; λ 10 , λ 11 , λ 12 and λ 13All are hyperparameters; during the training of the image fusion network, the fusion loss can adaptively adjust the hyperparameters for different degradation scenarios. In this embodiment, the parameters for typical degradation scenarios are set as follows: low-light scene and low-light flare scene λ. 10 , λ 11 , λ 12 and λ 13 The values ​​are 8, 1, 12, and 10 respectively; λ represents foggy scenes, rainy scenes, and blurred scenes. 10 , λ 11 , λ 12 and λ 13 The values ​​are 6, 1, 12 and 10 respectively.

[0030] In this application, the calculation formulas for intensity loss, structural similarity loss, color consistency loss, and gradient fidelity loss are consistent with the calculation formulas for intensity loss, structural similarity loss, color consistency loss, and maximum gradient loss disclosed in the paper "Text-IF: Leveraging Semantic Text Guidance for Degradation-Awareand Interactive Image Fusion". S4. Convert the degraded image to be processed into a YCbCr image. Then, separate the Y-channel image, Cb-channel image, and Cr-channel image of the visible light image to obtain the Y-channel image, Cb-channel image, and Cr-channel image of the degraded image to be processed. Input the Y-channel image of the degraded image and its corresponding infrared image into the image fusion network model obtained in step S4. Propagate forward once to obtain the Y-channel fused image containing high-level semantic information. Then, stitch the Y-channel fused image, the Cb-channel image, and the Cr-channel image of the degraded image to be processed along the channel dimension. After stitching, convert the image to RGB format to obtain the color fused image.

[0031] test: To verify the fusion effect of the image fusion method described in this application, this application first utilizes the image fusion method described in this application, as well as the Tar-DAL method (from "Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection"), the DIVFusion method (from "DIVFusion: Darkness-free infrared and visible image fusion"), the Text-IF method (from "Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion"), the Omnifuse method (from "OmniFuse: Composite degradation-robust image fusion with language-driven semantics"), the URFusion method (from "URFusion: Unsupervised Unified Degradation-Robust Image Fusion Network"), and the IASSF method (from "Infrared-assisted single-stage framework for joint restoration and fusion of visible and infrared images under hazy conditions"). These six existing image fusion methods (see "conditions") generate color fused images based on a test set, and then test the color fused images using existing Matlab test modules. Since low-light flare scenes are a special subclass of low-light scenes, both belong to the category of low-light degradation and generally suffer from the common problems of texture detail degradation and insufficient image contrast. Therefore, this application will conduct statistical analysis of test index results for low-light scenes and low-light flare scenes together. Specifically, the method described in this application and the above six existing image fusion methods generate color fused images based on visible light images and their corresponding infrared images in low-light scenes and visible light images and their corresponding infrared images in low-light flare scenes in the test set. The color fused images are tested using the existing Matlab test module, and the test index results are shown in Table 1. Furthermore, to verify that the method described in this application can effectively suppress localized extreme overexposure degradation caused by strong light flares, this application displays the fused image obtained from the visible light image and its corresponding infrared image in the low-light scene, as well as the fused image obtained from the visible light image and its corresponding infrared image in the low-light flare scene, in the visualization result display stage. Specifically, the method described in this application and the above six existing image fusion methods generate color fused images based on the visible light image and its corresponding infrared image in the low-light scene of the test set. The color fused images are then used to perform visual effect testing using the existing Matlab testing module. The test results are as follows: Figure 8 As shown; when testing visible light images and infrared images in low-light scenes, the semantic description text generated by the text generation model is "Two people are crossing the street at a crosswalk.", which means "Two people are crossing the road at a crosswalk." The method described in this application, along with the six existing image fusion methods mentioned above, generates color fused images based on visible light images and their corresponding infrared images in a low-light flare scene test set. The color fused images are then used to perform visual effect testing using existing Matlab testing modules. The test results are as follows: Figure 9 As shown; when testing visible light images and infrared images in low-light flare scenarios, the semantic description text generated by the text generation model is "A car is approaching a group of people crossing the street at a crosswalk.", which means "A car is driving towards a group of people crossing the street at a crosswalk." The method described in this application, along with the six existing image fusion methods mentioned above, generates color fused images based on visible light images and their corresponding infrared images in a foggy scene within the test set. These color fused images are then tested using existing Matlab testing modules. The test results are shown in Table 2, and the visual effects are as follows: Figure 10 As shown; when testing visible light and infrared images in foggy scenes, the semantic description text generated by the text generation model is "A small white building with a brown roof and a bicycle parked in front of it.", which means "A small white building with a brown roof has a bicycle parked in front of it." The method described in this application, along with the six existing image fusion methods mentioned above, generates color fused images based on visible light images and their corresponding infrared images in a rainy scene from the test set. The color fused images are then tested using existing Matlab testing modules. The test results are shown in Table 3, and the visual effects are as follows: Figure 11 As shown; when testing visible light and infrared images in rainy weather scenarios, the semantic description text generated by the text generation model is "A car is driving down a highway.", which means "A car is driving along the highway." The method described in this application, along with the six existing image fusion methods mentioned above, generates color fused images based on visible light images and their corresponding infrared images in blurred scenes within the test set. These color fused images are then tested using existing Matlab testing modules. The test results are shown in Table 3, and the visual effects are as follows: Figure 12 As shown; when testing visible light images and infrared images in blurred scenes, the semantic description text generated by the text generation model is "A red car is stopped at a red light in the city.", which means "A red car in the city stopped at a red light." Taking the method described in this application as an example of generating a color fused image based on a visible light image and its corresponding infrared image in a blurred scene in the test set, the testing strategy adopted for testing the method described in this application includes the following steps: The visible light images of blurred scenes in the test set are converted into YCbCr images. Then, the Y-channel, Cb-channel, and Cr-channel images of the visible light images are separated to obtain the Y-channel, Cb-channel, and Cr-channel images of the degraded image to be processed. The Y-channel image of the degraded image and its corresponding infrared image (which is also the infrared image corresponding to the visible light image in the blurred scene) are input into the image fusion network model obtained by the method described in this application. After one forward propagation, the Y-channel fused image containing high-level semantic information can be obtained. Then, the Y-channel fused image, the Cb-channel image of the degraded image to be processed, and the Cr-channel image of the degraded image to be processed are concatenated along the channel dimension. After concatenation, the image is converted to RGB format to obtain a color fused image.

[0032] Table 1 shows the test index results obtained based on visible light images and corresponding infrared images of low-light and low-light flare scenes in the test set.

[0033] Table 2 shows the test index results obtained based on visible light images and their corresponding infrared images in foggy scenarios in the test set.

[0034] Table 3 shows the test index results obtained based on visible light images and their corresponding infrared images under rainy weather scenarios in the test set.

[0035] Table 4 shows the test index results obtained based on visible light images and their corresponding infrared images in blurred scenes in the test set.

[0036] In Tables 1 to 4, SF measures the detail and texture richness of the fused image; a higher value indicates richer texture details and better fusion effect. AG characterizes the edge and detail clarity of the fused image; a higher value indicates clearer edge contours and stronger detail representation. CC quantifies the correlation between the fused image and the source image; a higher value indicates more complete retention of source image information and better information transmission. PSNR evaluates the distortion between the fused image and the source image; a higher value indicates less distortion and better image quality. Qabf measures the edge information transferred from the source image to the fused image; a higher value indicates better edge information transmission to the fused image. MS_SSIM, based on SSIM combined with multi-scale structural similarity, comprehensively evaluates structural similarity at multiple scales to comprehensively assess the distortion of the fused image; a higher value indicates higher image quality. Furthermore, the image fusion method described in this application is shown in Tables 1 to 4. Figures 8 to 12 All are represented using the Ours method.

[0037] As shown in Table 1: The Tar-DAL method outperforms the six existing image fusion methods mentioned above. Therefore, this application focuses on comparing the test results of the method described in this application and the Tar-DAL method based on visible light images and their corresponding infrared images in low-light scenes and visible light images and their corresponding infrared images in low-light flare scenes; the details are as follows: 1) The SF value obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and low-light flare scenes is 13.49% higher than that of the Tar-DAL method. This indicates that the color fusion image obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and visible light images and corresponding infrared images in low-light flare scenes contains richer texture detail information and has a more prominent detail representation capability. 2) The AG value obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and low-light flare scenes is 11.12% higher than that of the Tar-DAL method. This indicates that the color fusion image obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and visible light images and corresponding infrared images in low-light flare scenes has a clearer edge contour. 3) The CC value obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and low-light flare scenes is 5.87% higher than that of the Tar-DAL method. This indicates that the color fusion image obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and visible light images and corresponding infrared images in low-light flare scenes has a stronger correlation with the source image and can more completely retain the key information of the source image. 4) The PSNR value obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and low-light flare scenes is 2.48% higher than that of the Tar-DAL method. This indicates that the color fusion image obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and visible light images and corresponding infrared images in low-light flare scenes has lower distortion and better image quality. 5) The Qabf value obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and low-light flare scenes is 35.45% higher than that of the Tar-DAL method. This indicates that the color fusion image obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and visible light images and corresponding infrared images in low-light flare scenes can transfer more edge information of the source image to the fusion image, so that the color fusion image contains more detail information and edge information from the source image. 6) The MS_SSIM value obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and low-light flare scenes is 8.79% higher than that of the Tar-DAL method. This indicates that the color fusion image obtained by the method described in this application based on visible light images and corresponding infrared images in low-light scenes and visible light images and corresponding infrared images in low-light flare scenes has a higher multi-scale structural similarity with the source image and is more in line with human visual perception.

[0038] As shown in Table 2: The URFusion method outperforms the six existing image fusion methods mentioned above. Therefore, this application focuses on comparing the test results of the method described in this application and the URFusion method based on visible light images and their corresponding infrared images in foggy scenes; the details are as follows: 1) The SF value obtained by the method described in this application based on visible light images and corresponding infrared images in foggy scenes is improved by 5.41% compared with the URFusion method; 2) The AG value obtained by the method described in this application based on visible light images and corresponding infrared images in foggy scenes is 2.00% lower than that obtained by the URFusion method; 3) The CC value obtained by the method described in this application based on visible light images and corresponding infrared images in foggy scenes is 1.26% higher than that obtained by the URFusion method; 4) The PSNR value obtained by the method described in this application based on visible light images and corresponding infrared images in foggy scenes is 3.86% higher than that obtained by the URFusion method; 5) The Qabf value obtained by the method described in this application based on visible light images and corresponding infrared images in foggy scenes is improved by 27.93% compared with the URFusion method; 6) The MS_SSIM value obtained by the method described in this application based on visible light images and corresponding infrared images in foggy scenes is improved by 7.28% compared with the URFusion method.

[0039] As shown in Table 3: The URFusion method outperforms the six existing image fusion methods mentioned above. Therefore, this application focuses on comparing the test results of the method described in this application and the URFusion method based on visible light images and corresponding infrared images in rainy scenes; the details are as follows: 1) The SF value obtained by the method described in this application based on visible light images and corresponding infrared images under rainy weather conditions is 16.55% higher than that obtained by the URFusion method; 2) The AG value obtained by the method described in this application based on visible light images and corresponding infrared images in rainy scenes is 10.68% higher than that obtained by the URFusion method; 3) The CC value obtained by the method described in this application based on visible light images and corresponding infrared images in rainy scenes is 3.75% higher than that obtained by the URFusion method; 4) The PSNR value obtained by the method described in this application based on visible light images and corresponding infrared images in rainy weather scenarios is 2.28% higher than that of the URFusion method; 5) The Qabf value obtained by the method described in this application based on visible light images and corresponding infrared images under rainy weather conditions is improved by 28.18% compared to the URFusion method; 6) The MS_SSIM value obtained by the method described in this application based on visible light images and corresponding infrared images in rainy scenes is improved by 12.30% compared with the URFusion method.

[0040] As shown in Table 4: The Tar-DAL method outperforms the six existing image fusion methods mentioned above. Therefore, this application focuses on comparing the test results of the method described in this application and the URFusion method based on visible light images and their corresponding infrared images in blurred scenes; the details are as follows: 1) The SF value obtained by the method described in this application based on the visible light image and its corresponding infrared image in a blurred scene is improved by 13.50% compared with the URFusion method; 2) The AG value obtained by the method described in this application based on the visible light image and its corresponding infrared image in a blurred scene is improved by 7.22% compared with the URFusion method; 3) The CC value obtained by the method described in this application based on the visible light image and its corresponding infrared image in a blurred scene is improved by 4.31% compared with the URFusion method; 4) The PSNR value obtained by the method described in this application based on the visible light image and its corresponding infrared image in a blurred scene is 2.99% higher than that of the URFusion method; 5) The Qabf value obtained by the method described in this application based on the visible light image and its corresponding infrared image in a blurred scene is improved by 27.17% compared with the URFusion method; 6) The MS_SSIM value obtained by the method described in this application based on the visible light image and its corresponding infrared image in a blurred scene is improved by 12.53% compared with the URFusion method.

[0041] In addition, from Figures 8 to 12 It can also be seen that the color fusion images obtained by the method described in this application, based on visible light images and their corresponding infrared images under low-light flare scenes, foggy scenes, rainy scenes, and blurred scenes, all have more natural contrast and overall brightness, and are more in line with human visual perception. Moreover, through comparison... Figure 8 and Figure 9It can be seen that the method described in this application can effectively suppress the local extreme overexposure degradation caused by strong light flares.

Claims

1. An image fusion method based on texture and edge information enhancement of degraded images, characterized in that: Includes the following steps: Acquire the Y, Cb, and Cr channel images of the degraded image to be processed; input the Y channel image and its corresponding infrared image into the image fusion network model and propagate forward once to obtain the Y channel fusion image containing high-level semantic information; stitch the Cb, Cr channel images and the Y channel fusion image together and convert them to RGB format to obtain a color fusion image; The model is obtained by training an image fusion network; The network includes texture and edge information enhancement branches; the branches include a convolutional layer, a tile embedding layer, a Swing Transformer block, a first normalization layer, a tile inverse embedding layer, a first residual connection layer, an upsampling layer, and a Sigmoid layer connected in sequence. The Swin Transformer block consists of a second normalized layer, a window-based multi-head self-attention layer, a second residual connection layer, a third normalized layer, a multilayer perceptron layer, and a third residual connection layer connected in sequence; the tile embedding layer is connected to the first and second residual connection layers respectively, and the third normalized layer is connected to the third residual connection layer; The branch enhances the edge and texture information of the Y channel image to obtain a fine-grained structural feature map containing rich texture and edge information. Image fusion networks also include: The visible light information extraction branch is used to extract features from the Y channel image to obtain the initial visible light feature map. The enhanced feature perception module is used to perform channel attention weighting on the initial visible light feature map to obtain a clean feature map without degradation information. The degradation layer perception module is used to perform channel attention weighting on the initial visible light feature map to obtain a degradation feature map containing degradation information; A degradation-aware decoding network is used to reconstruct features from clean feature maps and degraded feature maps respectively, resulting in clean feature reconstructed images and degraded feature reconstructed images. The infrared information extraction branch is used to extract thermal radiation information from infrared images to obtain feature maps with thermal radiation information. A cross-modal complementary feature fusion network is used to perform multimodal feature interaction, text semantic modulation, and attention weighted fusion on clean feature reconstructed images, feature maps with thermal radiation information, fine-grained structural feature maps, and semantic description text to obtain a Y-channel fused image containing high-level semantic information; wherein, the semantic description text is obtained by processing degraded images in the training set using a text generation model.

2. The image fusion method for enhancing texture and edge information of degraded images according to claim 1, characterized in that: The degradation-aware decoding network includes an enhancement decoding branch and a degradation feature decoding branch. The enhancement feature perception module is connected to the enhancement decoding branch, and the degradation layer perception module is connected to the degradation feature decoding branch.

3. The image fusion method for enhancing texture and edge information of degraded images according to claim 1, characterized in that: In the enhanced feature perception module, two convolutional units A extract clean features from the initial visible light feature map. The global average pooling layer performs global average pooling on the feature map output by the second convolutional unit A to obtain a channel-level global feature vector. The Softmax layer normalizes the weights of the feature vector to obtain clean feature channel attention weights. The element-wise multiplication unit is used to multiply the weights with the initial visible light feature map element-wise to achieve channel attention weighting and obtain a clean feature map without degradation information.

4. The image fusion method for enhancing texture and edge information of degraded images according to claim 1, characterized in that: In the degradation layer perception module, two convolutional units A extract degradation interference features from the initial visible light feature map; the global average pooling layer performs global average pooling on the feature map output by the second convolutional unit A in the degradation layer perception module to obtain the channel-level global feature vector. The Softmax layer normalizes the weights of the feature vector to obtain the attention weights for the degradation feature channels. The element-wise multiplication unit multiplies the attention weights with the initial visible light feature map element-wise to achieve channel attention weighting and obtain a degradation feature map containing degradation information.

5. The image fusion method for enhancing texture and edge information of degraded images according to claim 2, characterized in that: The enhanced decoding branch includes three convolutional units A and one convolutional unit B connected in sequence; both convolutional units A and B in the enhanced decoding branch are used for feature extraction; the LReLU layer in convolutional unit A performs nonlinear feature transformation; the Sigmoid layer in convolutional unit B maps the feature values ​​to the (0,1) interval to achieve feature reconstruction and obtain a clean feature reconstruction image.

6. The image fusion method for enhancing texture and edge information of degraded images according to claim 2, characterized in that: In the degradation feature decoding branch, the first convolutional unit A performs preliminary feature extraction on the degradation feature map output by the degradation layer perception module; the first Concat layer concatenates the feature map output by the first convolutional unit A in the degradation feature decoding branch with the feature map output by the first convolutional unit A in the enhancement decoding branch; in the degradation feature decoding branch, the second convolutional unit A is used to extract deep features from the feature map output by the first convolutional unit A, and the third convolutional unit A is used to extract refined features from the feature map output by the second convolutional unit A; in the degradation feature decoding branch, the second Concat layer concatenates the feature maps output by the third convolutional unit A in the degradation feature decoding branch and the feature maps output by the third convolutional unit A in the enhancement decoding branch, obtaining the final concatenated feature; in the degradation feature decoding branch, the convolutional layer in convolutional unit B performs refined reconstruction of the final concatenated feature, the batch normalization layer in convolutional unit B performs batch normalization, and the Sigmoid layer performs pixel value normalization, realizing feature reconstruction and obtaining a degradation feature reconstruction image containing degradation information.

7. The image fusion method for enhancing texture and edge information of degraded images according to claim 2, characterized in that: The cross-modal complementary feature fusion network includes an adjusted convolutional layer and lightweight dense connection units. The degenerate feature decoding branch is connected to the adjusted convolutional layer. The adjusted convolutional layer and the infrared information extraction branch are both connected to the lightweight dense connection units. The lightweight dense connection units are connected in sequence to the shallow text injection unit, the element-wise multiplication unit, the deep text injection unit, the attention unit, and the fusion decoding unit. The element-wise multiplication unit is also connected to the attention unit. The texture and edge information enhancement branches are also connected to the shallow text injection unit, the element-wise multiplication unit, and the deep text injection unit, respectively. The shallow text injection unit and the deep text injection unit also take semantic description text as input.

8. The image fusion method for enhancing texture and edge information of degraded images according to claim 1, characterized in that: Training an image fusion network involves the following steps: training the image fusion network based on the training set and the total loss of the image fusion network to obtain the image fusion network model.

9. The image fusion method for enhancing texture and edge information of degraded images according to claim 8, characterized in that: The total loss of the image fusion network includes degradation decoupling loss, overexposure suppression loss and fusion loss; the overexposure suppression loss uses the pixel-level localized overexposure mask image as a spatial weight constraint, and combines the dark channel image of the degraded image to adaptively divide the overexposure region, and uses dual thresholds to achieve differential loss constraints between non-overexposure regions and overexposure regions; the dual thresholds include 4 times the dark channel mean and 7 times the dark channel mean.

Images