Infrared and visible light image fusion method, electronic equipment and storage medium

Abstract

The invention discloses an infrared and visible light image fusion method, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a weak visible light image and an infrared image corresponding to the weak visible light image; an image fusion network is constructed, the image fusion network is supervised by using a supervised label generated by a pre-trained brightness adjustment network, and the quality of a fused image is constrained by using a pre-trained cue word generation network; and inputting the weak visible light image and the infrared image corresponding to the weak visible light image into the image fusion network for fusion to obtain a fused image. According to the technical scheme, the problem of daytime overexposure existing in a traditional night image fusion algorithm is effectively restrained, and meanwhile the problem of night point light source overexposure is effectively restrained.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to an infrared and visible light image fusion method, electronic equipment and storage medium. Background Art

[0002] With the rapid development of computer vision technology, multi-sensor image fusion technology has gradually emerged in various application scenarios. A single sensor can often only capture local information of a scene and it is difficult to fully describe the target, which may not only lead to insufficient information, but also cause misjudgment in the decision-making process. In order to solve this problem, image fusion technology came into being. By fusing the image information captured by different sensors, more complex and detailed images are generated, which significantly improves the accuracy of subsequent processing and decision-making.

[0003] Among the many fusion technologies, the fusion of infrared and visible light images is particularly prominent. Visible light sensors rely on reflected light imaging, and the images they generate have high spatial resolution and rich details, which meet the needs of the human visual system. However, the imaging effect of visible light images will be greatly reduced under adverse weather conditions, such as haze, rain and snow. In contrast, infrared sensors capture the thermal radiation of objects for imaging, and can work all day in various climate conditions, making them particularly suitable for the identification and tracking of thermal targets. Despite this, infrared images usually lack details and texture information, which limits their performance in certain high-precision applications. The complementarity of information between infrared and visible light images makes their fusion an effective means to improve the comprehensiveness and accuracy of scene description. The fused image not only has high spatial resolution, but also retains the thermal information of the object, so it has been widely used in road monitoring, target recognition, field reconnaissance, and hidden weapon detection. In addition, the relative simplicity and low equipment cost of infrared and visible light imaging technologies have further promoted the popularization of these technologies in various scenarios.

[0004] Although infrared and visible light image fusion technology has shown great potential in a variety of applications, there are still some urgent problems to be solved when processing images in low-light environments. First, due to insufficient light, local details of the fused image are often blurred, resulting in loss of image information in key areas. Second, the low contrast of the target reduces the distinction between the target and the background, affecting the accuracy of target recognition. Although existing fusion methods have solved these problems to a certain extent, it is difficult to achieve consistent high-quality fusion under different lighting conditions. For example, in a general fusion network with sufficient light during the day, excessive enhancement may lead to loss of details in the highlight area and overexposure, thereby affecting the overall quality of the image. At the same time, in night scenes, especially when facing point light sources such as street lights and car lights, local overexposure is also common. This excessive local brightness not only obscures the surrounding details, but also may prevent the network from correctly capturing other important information in the scene. These overexposure problems make the existing fusion methods insufficiently adaptable and robust under dynamic lighting conditions, limiting their effectiveness in practical applications.

[0005] Therefore, in order to solve these problems, an image fusion method that can be adaptively adjusted under different lighting conditions is urgently needed to improve image detail retention, target contrast, and adaptability to lighting changes. This adaptive fusion method needs to provide high-quality fused images both during the day and at night to ensure the accuracy of subsequent processing. Summary of the invention

[0006] In order to overcome the technical problems existing in the above prior art, the present invention provides an infrared and visible light image fusion method, an electronic device and a storage medium.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0008] A method for fusing infrared and visible light images comprises the following steps:

[0009] Acquire a weak visible light image and an infrared image corresponding to the weak visible light image;

[0010] Constructing an image fusion network, wherein the image fusion network is supervised by supervised labels generated by a pre-trained brightness adjustment network, and a pre-trained prompt word generation network is used to constrain the quality of the fused image;

[0011] The weak visible light image and the infrared image corresponding to the weak visible light image are input into the image fusion network for fusion to obtain a fused image.

[0012] Furthermore, the brightness adjustment network is trained based on a brightness enhancement dataset. The brightness adjustment network uses text guidance to perform brightness enhancement on weak visible light images at night to improve their imaging quality under low light conditions; and only performs image reconstruction tasks on daytime images.

[0013] Furthermore, CLIP is used to encode the image and text simultaneously and calculate the similarity between the two to determine whether the scene matches the text description;

[0014] The given text is embedded into the training of the brightness adjustment network, and the text encoded by CLIP is presented as a vector; in the process of embedding the text vector into the brightness adjustment network, the guide text is encoded by CLIP to generate a fixed vector with a length of 512; since the number of feature channels of each layer in the network is inconsistent, the KAN network is used to adjust the length of the text vector to match the feature dimension of the current layer;

[0015] The KAN network expands and adjusts the input text vector through two linear layers, and implements nonlinear transformation through the LeakyReLU activation function, so that the text vector can adapt to the different channel number requirements of each layer of the network.

[0016] Furthermore, an affine transformation is used to more effectively combine text information; the affine transformation performs linear and bias operations on the input features by calculating scaling coefficients and offset coefficients, thereby ensuring that the text vector can have an impact on the input features.

[0017] Furthermore, the brightness adjustment network includes an encoder, a spatial attention module, an affine transformation module and a decoder, the encoder includes a DFN module and a downsampling module, and the decoder includes a DFN module and an upsampling module;

[0018] The working process of the brightness adjustment network includes:

[0019] Flatten the input weak visible light image so that the Mamba module in the DFN module can perform serialization operations;

[0020] The features of the input weak visible light image are extracted through the DFN module in the encoder, and the local texture and global semantic information are extracted respectively through the parallel CNN and Mamba architectures in the DFN module;

[0021] The multi-level structure of the DFN module gradually deepens the depth of feature extraction, helping the network capture image details and semantic information at different levels, thereby improving the fusion effect;

[0022] The downsampling module is used to act on the output of each DFN module to gradually reduce the spatial resolution;

[0023] The spatial attention module is used to enhance key area features, supplement the deficiencies of global modeling, and highlight significant spatial information.

[0024] The text features are embedded into each level of the decoder through the affine transformation module, so as to realize text-guided image enhancement. The affine transformation module uses the KANLinear layer and the LeakyReLU activation function to process the text features layer by layer, and generates transformation parameters adapted to the current decoder level. The transformation parameters include scaling factors and offset values, and the generated scaling factors and offset values ​​are combined with the image features in the decoder to realize text-based guidance.

[0025] Each layer of DFN module in the decoder can receive the text feature guidance after the affine transformation module to ensure that the decoder outputs the fused text information layer by layer. After decoding and fusion of all levels, the final output image is generated through the convolution layer.

[0026] Each upsampling module in the decoder is used to double the spatial resolution of the feature map, thereby gradually restoring it to the same resolution as the input image. After each upsampling module, the feature map in the decoder enters the next DFN module for further integration and processing of features;

[0027] The convolution layer in the decoder is used to convert the output feature map of the decoder into the final fused image; the convolution layer uses a 3×3 convolution kernel.

[0028] Furthermore, the DFN module includes two branches, namely a Mamba Block module and a Multi-CNN module, and the two branches are weightedly fused through learnable weights to achieve adaptive feature extraction of global semantic information and local texture information;

[0029] The Mamba Block module is used to extract semantic information of the image from a global perspective. The Mamba Block module includes a Norm module, three MLP modules, two Conv modules, a Forward SSM module, a Backward SSM module and an Activation module. The three MLP modules are respectively a first MLP module, a second MLP module and a third MLP module.

[0030] In the Mamba Block module, the input data is standardized through the Norm module to ensure stable feature distribution and improve numerical stability during training;

[0031] The first MLP module is used to perform a preliminary nonlinear transformation on the input features; the features processed by the first MLP module are sent to the subsequent Conv module and state space model to enrich the feature expression; the Conv module extracts local information from the input features and extracts local temporal patterns or spatial features;

[0032] The convolved features are sent to the Forward SSM module and the Backward SSM module; the Forward SSM module and the Backward SSM module are used to capture the forward and backward relations of sequence features respectively; the Forward SSM module extracts features through the state equation in the forward direction, while the Backward SSM module captures the reverse relation through the state equation in the backward direction;

[0033] Next, the output is passed to the Activation module through the second MLP module, allowing the model to perform nonlinear transformations; the activation features output from the Activation module are multiplied element-by-element with the outputs of the Forward SSM module and the Backward SSM module respectively;

[0034] After element-wise multiplication, the forward and backward features are added together to form a more robust intermediate feature that combines information from both directions;

[0035] The combined features are further processed by the third MLP module to further abstract the features and help the model extract more useful information;

[0036] The initial input features are added to the output of the third MLP module, and the original feature details are preserved through residual connections, so that low-level and high-level features can be integrated in the final output;

[0037] The Multi-CNN module is used to extract local texture information, and it adopts a parallel structure of convolutional layers with different convolution kernel sizes to obtain multi-scale image features;

[0038] The Multi-CNN module includes 3x3 convolution kernels, 5x5 convolution kernels, and 7x7 convolution kernels; the 3x3 convolution kernel is used to capture the detailed features of the image; the 5x5 convolution kernel has a larger receptive field than the 3x3 convolution kernel, can capture texture and edge information in a larger range, and can extract medium-scale image features; the 7x7 convolution kernel has a larger receptive field than the 5x5 convolution kernel, can capture global texture and structural information, and can extract larger-scale features; the Multi-CNN module can capture rich texture detail information at different scales by extracting features of different convolution kernel sizes in parallel;

[0039] The multi-scale features are spliced ​​to form a fused feature representation with multi-level texture information;

[0040] The DFN module also includes a weight fusion module; in the DFN module, the features extracted by the two branches of the Mamba Block module and the Multi-CNN module are fused through learnable weights; the weight fusion module is used to adaptively adjust the feature contributions of the two branches of the Mamba Block module and the Multi-CNN module; through training, the network can learn the role proportions of the Mamba Block module and the Multi-CNN module in different channels and spatial positions, thereby achieving better feature extraction; the features of the two branches of the Mamba Block module and the Multi-CNN module are weightedly added according to the weights to obtain the final fused feature representation.

[0041] Furthermore, the input of the image fusion network is a weak visible light image and an infrared image corresponding to the weak visible light image;

[0042] The input weak visible light image and the infrared image corresponding to the weak visible light image are converted into a form suitable for encoder processing through the Flatten operation;

[0043] Encoding the image features of the two modalities through an encoder, extracting their respective feature representations, and distilling them with the features extracted from the brightness adjustment network;

[0044] The Aix module is used to initially fuse the features of different modalities, and the fused features of the Aix module are input into the spatial attention module to weight the features of the key areas, making the features of the key areas more prominent, thereby improving the fusion effect;

[0045] Based on the text guidance, the Aix module supports different scene processing; the day scene text is used for processing under daytime conditions, emphasizing the image features of high brightness and high contrast; the night scene text is suitable for low brightness and cold environment conditions; by embedding affine transformation into the fusion process, the image fusion network's response to different scenes is controlled;

[0046] The decoder receives the fused features and restores the image spatial resolution through multiple upsampling modules;

[0047] After two 3x3 convolutional layers, the output is adjusted to the expected output image format to obtain a fused image.

[0048] Furthermore, the input of the Aix module is infrared features and visible light features, and the Aix module is a parallel branch design of Mamba and CNN;

[0049] In the Mamba branch, the input features are flattened; layer normalization is performed to ensure the stability of the input; the Mamba block module is entered to capture the sequence features through the Mamba block module for global modeling; after serialization, the two modal features are concatenated in different orders to capture the impact of the sequence order on feature modeling;

[0050] The splicing operation forms two different sequence combinations, namely a first combination and a second combination; the first combination is that the visible light is in front and the infrared is in the back; the second combination is that the infrared is in front and the visible light is in the back;

[0051] The two sequences are deserialized back to the original two-dimensional feature form, and the serialized global features are remapped back to the spatial domain to restore the spatial information of H and W dimensions. Learnable weights are applied to the two split features to balance the fusion proportion of the two, and the contribution of different modalities to the final features is adaptively adjusted according to the specific scenario.

[0052] In the CNN branch, the input features are pooled in the H and W dimensions to compress the spatial information into the channel dimension so that local information can be extracted in a smaller spatial range later. After pooling along the H dimension, features of shape B×C×1×W are obtained, and after pooling along the W dimension, features of shape B×C×H×1 are obtained.

[0053] The four vectors are concatenated to form a tensor with a shape of B×C×(2(W+H))×1; the concatenated features are sent to MLP for channel attention learning, specifically: squeeze to reduce the number of channels, reduce the feature dimension from 2(W+H), and then excite to restore the dimension; learn the dependency between different pixels through the squeeze-excitation mechanism; split the MLP output into two parts, each used to enhance the input features; and use the Sigmoid activation function to generate channel attention weights, add a new dimension to the two enhanced features, and convert it to B×1×C×H×W through the unsqueeze operation, and then concatenate along the newly added dimensions to obtain a tensor with a shape of B×2×C×H×W; use 3D convolution to process the concatenated features, learn and fuse the information of the two modalities, and generate the fused features; remove the extra dimensions through the pooling operation to obtain the output;

[0054] The two enhanced branch features are put together and fused through learnable weights; the weights are used to dynamically adjust the fusion ratio of the two modalities to adapt to the needs of different scenarios and features, and obtain the final fused output features.

[0055] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned infrared and visible light image fusion method when executing the computer program.

[0056] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the infrared and visible light image fusion method is implemented.

[0057] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0058] The present invention provides an infrared and visible light image fusion method, an electronic device and a storage medium. The method is based on distillation and text-guided technology and can solve the problem of image processing and enhancement in low-light environments. The method also designs a set of parallel feature extraction and fusion modules, combines CNN and Mamba structures, and can adaptively adjust weights between global semantic information and local texture information to ensure that different features can play an effective role in the fusion process. Through this adaptive approach, the network can dynamically handle fusion requirements under complex lighting conditions. The technical solution of the present invention not only effectively suppresses the daytime overexposure problem existing in traditional night image fusion algorithms, but also effectively suppresses the nighttime point light source overexposure problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a schematic diagram of the process of the infrared and visible light image fusion method of the present application;

[0061] Figure 2 It is a schematic diagram of the overall network structure of this application;

[0062] Figure 3 is a schematic diagram of a brightness adjustment network in this application;

[0063] Figure 4 is a schematic diagram of the DFN module in this application;

[0064] Figure 5 is a schematic diagram of the Aix module in this application;

[0065] Figure 6 is a schematic diagram of a text-guided image fusion network in this application;

[0066] Figure 7 It is a rendering of the present application for demonstrating the beneficial effect of suppressing the daytime overexposure problem. The a1 and a2 on the left are renderings of the traditional fusion algorithm, and the b1 and b2 on the right are renderings of the present application;

[0067] Figure 8 These are the renderings of the present application used to demonstrate the beneficial effects of suppressing the overexposure problem of point light sources at night, c is an infrared image, d is a visible light image; e1 is a rendering of a traditional fusion algorithm, and e2 is a rendering of the present application. DETAILED DESCRIPTION

[0068] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0069] Embodiment 1:

[0070] like Figure 1 As shown, the present invention provides a technical solution:

[0071] A method for fusing infrared and visible light images comprises the following steps:

[0072] Acquire a weak visible light image and an infrared image corresponding to the weak visible light image;

[0073] Constructing an image fusion network, wherein the image fusion network is supervised by supervised labels generated by a pre-trained brightness adjustment network, and a pre-trained prompt word generation network is used to constrain the quality of the fused image;

[0074] The weak visible light image and the infrared image corresponding to the weak visible light image are input into the image fusion network for fusion to obtain a fused image.

[0075] Continue reading Figure 1, the traditional brightness adjustment fusion network lacks high-quality brightness enhanced images for supervision, while for the fusion network proposed in the present invention, a pre-trained brightness adjustment network is used to generate supervised labels, and the knowledge distillation technology is combined to extract feature maps in each layer of the brightness adjustment network, and each feature map is averaged and pooled along the B and C dimensions respectively. In this way, each layer of feature maps is converted into a vector representation, ensuring that the features between different network layers can be directly compared. The specific implementation is: in order to make the features extracted from the fusion network as similar as possible to the features of the brightness adjustment network, the mean square error method is used to calculate the difference between the feature vectors of the two networks. The prompt word generation network learns to generate texts describing unimodal images. These texts provide additional supervision signals during the training process, helping the fusion network understand the features of unimodal images, and then approach the features of the fusion image. Through the constraints of the two parts, combined with text guidance, different functions are displayed according to different texts. In the daytime scene, the guidance text is set to "Good brightness and high contrast images", and only the image fusion task is performed. In the night environment, the guidance text is set to "Dark, poorly lit, cold environments", and the image fusion and brightness adjustment tasks are coupled. Finally, a high-quality fused image is generated.

[0076] The main application scenarios of this embodiment are: image processing tasks in complex and low-light environments, including road monitoring, field reconnaissance, target identification and tracking, hidden weapon detection, disaster relief, industrial inspection, and night monitoring. By fusing infrared and visible light images, the present invention can provide high-quality fused images under low-light conditions, significantly improve the accuracy of target identification and tracking, enhance image details and contrast, and is suitable for various applications that require high-precision image processing.

[0077] Embodiment 2:

[0078] Based on Example 1, Figure 1 The infrared and visible light image fusion method provided in Example 1 specifically includes:

[0079] 1. Enhanced training of weak visible light images. Through the brightness adjustment network trained on the brightness enhancement dataset, with text guidance, the brightness of weak visible light images (night images) is enhanced to improve their imaging quality under low light conditions. Only image reconstruction tasks are performed on daytime images.

[0080] 1.1: 494 pairs of normal brightness I were selected from the LOL dataset norm and low brightness I low For training, the image resolution is 400*600, and the loss function used is defined as:

[0081] L total =50×L grad +10×L1+10×L ssim +0.001×L tv

[0082] 1.2: The loss function is guided by the text and distinguishes between daytime and nighttime tasks. During the day, only reconstruction tasks are performed, while image enhancement tasks are performed at night. I en As the updated image is used as the generated output of the model, I ref As a benchmark to calculate various losses, during the daytime I norm As I ref , and at night I low As I ref .

[0083] In order to ensure that the gradients between the generated image and the reference image match, thereby maintaining the edge sharpness and details during the fusion process, a gradient loss function L is used. grad , where L grad Defined as:

[0084]

[0085] The L1 loss function is used to ensure that the generated image is as consistent as possible with the reference image at the pixel level, so that the enhanced image has similar brightness and detail characteristics as the target output;

[0086] Among them, y i is the pixel value of the real image, that is, the image I after the target brightness is enhanced ref ,and is the pixel value of the enhanced image output by the training step, which is specifically defined as:

[0087]

[0088] SSIM loss is used to ensure that the enhanced image has similar structural details to the reference image visually to avoid blurring or over-enhancement; x is the generated image I en , y is the reference image I ref .μ x and μ y They represent the mean of image x and y, respectively, reflecting the brightness of the image. and is the variance of images x and y, indicating the contrast of the image. xy is the covariance of images x and y, indicating the structural similarity of the two images. C1 and C2 are constants, which are used to stabilize the calculation of brightness and contrast, respectively, to avoid the situation where the denominator is 0. The specific definition is:

[0089]

[0090] Finally, for the generated image I en The total variation loss is added to reduce the noise in the image and maintain the smoothness of the image. I(i,j) represents the pixel value of the image at position (i,j), and the horizontal and vertical neighboring pixel differences of each pixel are summed. The specific formula is expressed as:

[0091]

[0092] 1.3: To ensure that the enhancement network can perform low-light image enhancement in night scenes without image enhancement in daytime scenes, the present invention introduces a text-guided mechanism based on the pre-trained large model CLIP; the CLIP model encodes the image and text at the same time and calculates the similarity between the two to determine whether the scene matches the text description.

[0093] When a text input related to daytime scenes is provided (for example, "Image processing tasks in daytime scenes"), the CLIP model recognizes that this is a daytime scene with normal brightness by calculating the similarity between the text and the image. At this time, the network only performs the image restoration task without brightness enhancement. When the input text is related to night scenes (for example, "Image processing tasks in night scenes"), the CLIP model also recognizes that the current image belongs to a low-brightness night scene through similarity calculation, triggering the network to perform low-light image enhancement tasks. In this way, the network can dynamically adjust its functions according to the guiding text to achieve adaptive image processing in different scenarios.

[0094] Through this text-guided approach, the present invention ensures that the network can select a suitable image processing path according to scene requirements, thereby improving the intelligence and flexibility of image processing.

[0095] 1.4: The given text needs to be embedded in the training of the brightness adjustment network. The text encoded by CLIP is presented as a vector in the present invention, and the number of channels of the vector is adjusted using the KAN network.

[0096] In the process of embedding the text vector into the brightness adjustment network, the guide text is first encoded through CLIP to generate a fixed vector of length 512. Since the number of feature channels of each layer in the network is inconsistent, the KAN network is used to adjust the length of the text vector to match the feature dimension of the current layer. The KAN network expands and adjusts the input text vector through two linear layers, and implements nonlinear transformation through the LeakyReLU activation function, so that the text vector can adapt to the different channel number requirements of each layer of the network.

[0097] Simply adjusting the vector length through the KAN network is not enough to fully embed the text information into the network. To this end, the present invention further adopts an affine transformation similar to the channel attention mechanism to more effectively combine the text information. The affine transformation performs linear and bias operations on the input feature x by calculating the scaling factor γ and the offset factor β to ensure that the text vector can affect the input feature.

[0098] Specifically, after the text vector is processed by the KAN network, the input feature x is adjusted in the form of an affine transformation:

[0099] x=(1+γ)·x+β

[0100]

[0101] The feature map is generated by the KAN network and then adjusted pixel by pixel by channel. This processing mechanism ensures that the network can flexibly adjust its behavior according to the input text information, so that the text affects the output of the network as an additional input parameter. In this way, the network can dynamically adjust the brightness enhancement task in different scenarios and make full use of the text information to improve the effect of image processing tasks.

[0102] 1.5: The training of the enhancement network adopts a two-stage parameter update strategy to ensure that the network can be adaptively optimized for different scenes. Stage 1: Daytime scene parameter update The network receives low-brightness images in dark scenes as input, and combines the feature vectors of daytime scene texts. The network generates a restored image and reconstructs the input image through the restoration network. Stage 2: The network receives input text related to night scenes. The input image is still a low-brightness image, but in this stage, the network not only restores the image, but also performs brightness enhancement tasks. The network generates an enhanced image, and the loss function is used to evaluate the difference between the generated image and the reference image under normal lighting conditions (bright light images in the LOL dataset).

[0103] 1.6: Reference Figure 3 The enhanced network and fusion network in the present invention both adopt the feature extraction module DFN with parallel branching idea. For the specific structure, please refer to Figure 4. The DFN (Dual Feature Network) feature extraction module combines the advantages of Mamba and CNN, focusing on global modeling and local feature extraction. Since Mamba is better at processing global features, it may ignore these details when processing local areas containing detailed texture information, resulting in poor performance on these channels. The CNN branch introduces a multi-scale convolution structure and uses convolution kernels of different sizes to capture features under different receptive fields. This design can better capture local texture information and spatial details, making up for the shortcomings of Mamba in local feature extraction. In order to address the respective advantages and disadvantages of Mamba and CNN branches, DFN designs a learnable weight mechanism W. This weight mechanism automatically adjusts the output weights of Mamba and CNN through the training process, so that the network learns to balance the contributions of the two on different channels. This design greatly enhances the ability to capture image features, and can achieve a balance between global and local information in both the enhanced network and the fusion network, improving the expressiveness and clarity of the image.

[0104] 2. Supervised training plays a key role in image fusion. Through supervised training, the fusion network can understand what kind of image can be called a high-quality fusion image. With the help of the text guidance idea proposed in the present invention, the network can be guided and trained by providing prompt words (such as "good fusion image"), which can be used as an effective supervision method. However, directly using the CLIP model for training may face some challenges, such as: insufficient image and text data pairs during training, limitations of transfer learning and other problems. Therefore, the present invention proposes a new solution, that is, generating text for describing unimodal images by training a network that generates text. When training the fusion network, the pre-trained prompt word generation network is used as a further fusion quality supervision to improve the final quality of the fusion image.

[0105] 2.1: In order to train the prompt word generation network, the present invention selects 100 low-quality single-modal images, including 50 visible light images and 50 infrared images from the LLVIP dataset, and 50 high-quality fusion images (selected from the LENFusion method). During the training process, the prompt word generation network learns to generate text describing the single-modal images. These texts provide additional supervision signals during the training process to help the fusion network understand the characteristics of high-quality fusion images.

[0106] 2.2: Each image is assigned a binary label to mark the type of image. Specifically, 1 represents a single-modal image (visible light or infrared image) and 0 represents a high-quality fusion image. With this label, the model can distinguish the category of images during training and integrate the binary classification task into the learning process of the prompt word generation network.

[0107] During training, the cue word generation network receives image features and corresponding labels as input. The model aligns unimodal images with image features labeled "1" and fused images with image features labeled "0". This supervisory signal enables the model to learn how to generate appropriate text descriptions to distinguish between unimodal and fused images.

[0108] In order to further improve the training effect, the present invention also introduces data enhancement technology. For single-modal images and fused images, random rotation, flipping, scaling and other data enhancement methods are applied respectively to increase the diversity of data and enhance the generalization ability of the model in different scenarios. In this way, the prompt word generation network can not only understand the characteristics of different images, but also adapt to various types of transformed image data, thereby improving its robustness.

[0109] 2.3: During the training of the prompt word generation network, the system uses cosine similarity as a metric to evaluate the match between the generated text prompt and the image features. In each training iteration, the model evaluates the accuracy of the text prompt by calculating the cosine similarity between the image features and the generated text embedding. Specifically, the system expects that the text description of a unimodal image has a high similarity (close to 1) with its image features, while the text description of a fused image should also have a high similarity with its features. This similarity calculation method can effectively guide the model to adjust the representation of the text embedding to generate more accurate prompt words.

[0110] In order to further improve the quality of generated text, this paper proposes a new text embedding optimization strategy. During the training process, by comparing different batches of images and their corresponding text embeddings, the model will continuously optimize the generated text prompts so that they can more accurately describe single modality and fused images in the feature space. This strategy can ensure that in the process of generating high-quality images, text prompts play a key supervisory role and help the model identify what is a "high-quality" fused image.

[0111] 2.4: The training result of the present invention is not to directly generate natural language text describing unimodal images, but to optimize a text prompt embedding that matches the unimodal image features. This embedding can be regarded as an effective description of the unimodal image, but it is expressed in a numerical form for alignment with the image features.

[0112] 3. For the proposed fusion network, the present invention uses a pre-trained brightness adjustment network to generate supervised labels, and uses a pre-trained boosting word generation network to further constrain the quality of the fused image. We selected several pairs of images from the MSRS dataset, expanded the data by cropping, and obtained 26112 pairs of single-modal images. During the training process, the random seed is fixed to 3407, the small batch random descent is set to 32, the learning rate is set to 0.0005, the training round is set to 10, and the hyperparameters of the distillation loss function during the training process are set to 1.0, 0.5, 0.25, and 0.125. The proposed fusion network can show different functions according to different texts. In the daytime scene, the guide text is set to "Good brightness and high contrast images" and only performs image fusion tasks. In the night environment, the guide text is set to "Dark, poorly lit, cold environments" to perform the task of image fusion and brightness enhancement. The Adam optimizer is used for parameter update in the network training. In order to accelerate the training, mixed precision is used for calculation.

[0113] 3.1: The fusion network uses the idea of ​​knowledge distillation to extract the features of each layer in the pre-trained enhancement network. When training the fusion network, the features of the brightness adjustment network and the image fusion network are made as close as possible, thereby achieving the effect of enhancing and fusing at the same time.

[0114] 3.2: The distillation operation of the present invention is: extract feature maps in each layer of the brightness adjustment network, perform average pooling operations on each feature map along the B and C dimensions respectively, and convert each layer of feature maps into vector representation in this way to ensure that the features between different network layers can be directly compared. In order to make the features extracted from the fusion network as similar as possible to the features of the brightness adjustment network, the mean square error method is used to calculate the difference between the feature vectors of the two networks:

[0115]

[0116] Among them, N is the number of layers in the network, n is the dimension of the feature vector, and f fuse and f en They are fusion network features and enhanced network features; f en,i and f fuse,i Represents the element at the i-th position of the enhanced network vector and the fused network vector.

[0117] 3.3: The loss function used by the fusion network is the same L total However, we additionally use distillation guidance and text supervision to assist in generating fused images that are more suitable for dark environments.

[0118] 3.4: In order to further enhance the deeper information fusion of the fusion network, the present invention additionally designs the Aix feature fusion module according to the parallel idea. Figure 5 , still using the Mamba branch and the CNN branch to work in parallel, and fuse the information of different modalities through adaptive weighting. For the fusion task, we improved the Mamba branch by bidirectional sequence fusion. Due to the different principles of visible light and infrared imaging, the features they capture are different. Through the arrangement and combination of bidirectional sequences, the model can fully capture the relationship and complementarity between the two modalities and enhance the expressiveness of fusion. The core value is that it provides a rich source of information by changing the arrangement order of different modal features, and improves the model's ability to capture multimodal information. On the CNN branch, we added a coordinate attention mechanism to compress and excite the information in the channel, optimize the dependency between pixels, and use 3D convolution to further ensure that the information interaction and fusion between different modalities are no longer limited to the plane, but better model the dependency between the two through the spatial dimension. This module achieves a deeper level of information fusion between different modalities, greatly improving the effect of image fusion, and providing clearer and more detailed image performance in low-light and multimodal scenes.

[0119] like Figure 2 As shown, Figure 2 Schematic diagram of the overall network structure of the present application; wherein the input of the brightness adjustment network is a visible light image, and the multi-level features F of the visible light image are distilled out. dis1 , F dis2 , F dis3 and F dis4 The features extracted by the encoder of the mean square error and text-guided fusion network are used for training approximation. The input of the text prompt network is a high-quality fused image and a unimodal image. A simple neural network is used to generate a unimodal text description. The pre-trained unimodal text description is used as a Clip loss to supervise the training of the text fusion network to generate a fusion image that is more in line with multimodal features. The input of the fusion network is a low-light image and an infrared image. Two single encoders are used to extract the corresponding image features. The extracted image uses the Aix feature fusion module proposed by the present invention and the corresponding text guidance to complete the comprehensive feature reorganization, and the decoder is used to reconstruct a high-quality fused image as output.

[0120] Furthermore, an encoder and decoder scheme is used in the network architecture of the brightness adjustment network. The input image first passes through the encoder to extract multi-level features, and then the image is restored through a series of decoding operations. During the process, text features are used to adjust the features of each layer, so that the image enhancement process can be dynamically adjusted in combination with text information.

[0121] Specifically, Figure 3As shown in the figure, this figure introduces the structure of the brightness adjustment network. The example in the figure is an input night visible light image. First, the Mamba module in DFN is serialized by flattening. In the encoder, DFN is responsible for extracting the features of the input image, and extracting local texture and global semantic information respectively through parallel CNN and Mamba architectures. The multi-level structure of DFN (such as 2×DFN, 3×DFN, and 4×DFN in the figure) deepens the depth of feature extraction layer by layer, helping the network capture the details and semantic information of the image at different levels, thereby improving the fusion effect. Downsampling acts on the output of each DFN module to gradually reduce the spatial resolution. This layer-by-layer downsampling not only reduces computational overhead, but also allows the network to focus on a larger receptive field and improve its perception of global contextual information. Spatial attention plays the role of enhancing key regional features, supplementing the global modeling deficiencies, and highlighting significant spatial information. The AffineTransformation module is then used to embed text features into each level of the decoder, thereby achieving text-guided image enhancement. The Transformation module uses the KANLinear layer and the LeakyReLU activation function to process the text features layer by layer, generating transformation parameters that are adapted to the current decoder level. These parameters include scaling factors (gamma) and offset values ​​(beta). The generated gamma and beta values ​​are combined with the image features in the decoder to achieve text-based guidance. Each layer of the DFN module in the decoder can receive text feature guidance after Affine Transformation to ensure that the decoder outputs layer-by-layer fusion of text information. Finally, after decoding and fusion at all levels, the final output image is generated through the convolutional layer. Each upsampling module is responsible for doubling the spatial resolution of the feature map, thereby gradually restoring it to the same resolution as the input image. After each upsampling module, the feature map in the decoder will enter the next DFN module to further integrate and process the features. The combination of upsampling and DFN modules enables the feature map to gradually enhance feature information during the process of layer-by-layer enlargement, ensuring that important texture and semantic information is retained while restoring resolution. The last convolution layer is used to convert the decoder's output feature map into the final fused image. This layer uses a 3×3 convolution kernel, which can better capture local features while reducing the number of channels to match the requirements of the final output image.

[0122] Furthermore, the DFN feature extraction module is a parallel structure of Mamba and CNN. The present invention introduces a learnable weight parameter W to adaptively fuse the two parts, so that the network can perform weighted fusion of Mamba and CNN according to actual conditions. The network automatically adjusts the weights through learning during the training process, so that the features extracted by Mamba or CNN can be flexibly selected at different channels and different levels.

[0123] Specifically, Figure 4 As shown in the figure, the internal structure of DFN (Dual-feature Fusion Network) consists of two main branches: Mamba Block and Multi-CNN, which are weighted fused through learnable weights to achieve adaptive feature extraction of global semantic information and local texture information. Mamba Block is a key module in DFN, responsible for extracting semantic information of images from a global perspective. (Norm) First, the input data is standardized to ensure stable feature distribution and improve numerical stability during training. Then a multi-layer perceptron (MLP) is used in the module to perform preliminary nonlinear transformation on the input features. The features processed by the MLP module are sent to the subsequent convolutional layers and state-space models to enrich the feature expression. The convolutional layer extracts local information from the input features. This operation is particularly important in the processing of sequence data, and can extract local temporal patterns or spatial features. Here, the convolved features are distributed to the forward and backward SSM modules. Forward SSM and Backward SSM are the core parts of this module, which are used to capture the forward and backward relationships of sequence features, respectively. The forward SSM extracts features through the state equation in the forward direction, while the reverse SSM captures the reverse relationship through the state equation in the backward direction. Through SSM modeling in these two directions, Mamba Block can capture timing information more comprehensively.

[0124] The design of forward and backward SSM allows the model to exploit bidirectional information when processing complex sequences, thereby improving the ability to capture long-term dependencies. The lower MLP passes the output to the Activation module, allowing the model to perform nonlinear transformations, which is very important for capturing complex patterns. The activation features output from the Activation module are multiplied element-by-element with the outputs of the Forward SSM and Backward SSM, respectively. After element-by-element multiplication, the forward and backward features are added to form a more robust intermediate feature that combines information from both directions. The combined features are further processed by the final MLP layer, which further abstracts the features and helps the model extract more useful information. Finally, the initial input features are added to the output of the final MLP. This residual connection ensures that the original feature details are preserved, allowing low-level and high-level features to be integrated in the final output. The Multi-CNN module is mainly used to extract local texture information. It uses a parallel structure of convolutional layers with different convolution kernel sizes to obtain multi-scale image features. The following are the components of this module: The 3x3 convolution kernel is used to capture the detailed features of the image, which is suitable for extracting smaller texture information and providing a higher resolution response. The 5x5 convolution kernel can capture texture and edge information in a larger range, has a wider receptive field than the 3x3 convolution, and can extract medium-scale image features. The 7x7 convolution kernel has a larger receptive field, can capture global texture and structural information, and is suitable for extracting larger-scale features. The Multi-CNN module can capture rich texture detail information at different scales by extracting features of different convolution kernel sizes in parallel. These multi-scale features are then concatenated to form a fused feature representation with multi-level texture information. In DFN, the features extracted by the Mamba Block and Multi-CNN branches are fused through learnable weights. The role of the weight fusion module is as follows: Weight: This weight is a learnable parameter that can adaptively adjust the feature contribution of the Mamba Block and Multi-CNN branches. Through training, the network can learn the role of Mamba and CNN in different channels and spatial positions, thereby achieving better feature extraction. The features of the Mamba Block and Multi-CNN branches are weighted and added according to the weights to obtain the final fused feature representation. This fused feature contains global semantic information and local texture information, and can provide a more complete input for subsequent feature processing.

[0125] Among them, the Norm module is the normalization module, which uses Batch Normalization (BN) or LayerNormalization (LN). The function of the normalization module is to standardize the input data, reduce the internal covariate shift, accelerate the convergence speed of the network, and improve the generalization ability of the model. Through normalization, it can ensure that the distribution of input features between each layer is more stable, thereby improving the performance of the network.

[0126] MLP module (Multi-Layer Perceptron): The multi-layer perception module consists of multiple fully connected layers and is used to learn and extract high-level features of images. In Mamba Block, the MLP module gradually captures and abstracts the complex semantic information in the image through forward propagation and back propagation. Each MLP module can be regarded as a feature transformer that converts input features into higher-dimensional representations, thereby enhancing the model's ability to understand global information.

[0127] Conv module (Convolutional Layer): The convolution module is the most basic image feature extraction unit. The convolution layer performs convolution operations on the input image through a sliding window to extract local features, and gradually builds a more complex global feature representation by stacking multiple convolution layers.

[0128] Forward SSM module (Spatial Semantic Module): The forward spatial semantic module is used to enhance the spatial structure information of features. This module captures the long-distance dependencies between pixels in the image through the multi-head self-attention mechanism and spatial position encoding. The forward SSM module helps the model better understand the semantic structure of the image and enhances the perception of global information by focusing on the global context.

[0129] Backward SSM module: The backward spatial semantic module is similar to the forward SSM module, but its role is to further optimize and refine the global features through back propagation. Through the forward and backward spatial semantic modules, the global features of the image can be more comprehensively captured and preserved.

[0130] Activation module: It is an activation module, in which the activation function introduces nonlinearity so that the model can learn more complex feature representations. It can ensure that the network maintains nonlinear transformation capabilities during the layer-by-layer feature extraction process, thereby improving the model's expressiveness and generalization performance.

[0131] Furthermore, the Aix feature fusion module follows the parallel idea, but needs to be more in line with the requirements of the fusion task. For the Mamba branch, the present invention proposes a dual-modal serialized splicing fusion strategy to ensure that the differences caused by the sequence order can be balanced through learning, and adaptive global information fusion is achieved. For the CNN branch, the fusion rule based on the improved coordinate attention mechanism is first adopted, and the preliminary fusion results are achieved by using coordinate decomposition, channel compression and expansion. In order to further enhance the fusion effect, the present invention proposes a three-dimensional convolution fusion scheme to ensure the deep interaction of visible light and infrared features. During the entire fusion process, the outputs of the CNN and Mamba branches are still weighted fused through the learnable weights W to ensure that the fusion module can adaptively select more important features according to specific tasks, thereby improving the quality of the fusion results.

[0132] Specifically, Figure 5 As shown in Figure 1, the input of the Aix module is infrared features and visible light features, which is also a parallel branch design of Mamba and CNN.

[0133] Mamba Branch:

[0134] Flatten and LayerNorm: First, the input features are flattened (Flatten), and then layer normalization (LayerNorm) is performed to ensure the stability of the input and prepare to enter the Mamba module. The sequence features are captured through the Mamba block for global modeling. After serialization, the two modal features are spliced ​​in different orders to capture the impact of the sequence order on feature modeling. This splicing operation forms two different sequence combinations:

[0135] Combination 1: Visible light in front, infrared in the back (expressed as xy).

[0136] Combination 2: infrared in front, visible light in the back (expressed as yx).

[0137] Unflatten: The two sequences are deserialized back to the original two-dimensional feature form. This step remaps the serialized global features back to the spatial domain and restores the spatial information of H and W dimensions. Learnable weights are applied to the two split features to balance the fusion ratio of the two. In this way, the contribution of different modalities to the final features can be adaptively adjusted according to the specific scenario.

[0138] CNN branches:

[0139] The input features are pooled in the H and W dimensions to compress the spatial information into the channel dimension so that local information can be extracted in a smaller spatial range later. After pooling along the H dimension, features of shape B×C×1×W are obtained, and after pooling along the W dimension, features of shape B×C×H×1 are obtained. The four vectors obtained (two modes, H and W directions) are concatenated to form a tensor of shape B×C×(2(W+H))×1. The concatenation here adjusts the dimension of the tensor through the permute transposition operation to align the data in all directions. The concatenated features are sent to a small MLP for channel attention learning: first "squeeze", that is, reduce the number of channels and reduce the feature dimension from 2(W+H), and then "excite" to restore the dimension. In this way, the dependency between different pixels can be learned through the squeeze-excitation mechanism. The MLP output is split into two parts, each used to enhance the input features. The channel attention weight F is generated using the Sigmoid activation function. vis en F ir en , add a new dimension to the two enhanced features, convert them into B×1×C×H×W through the unsqueeze operation, and then concatenate along the newly added dimension to obtain a tensor with a shape of B×2×C×H×W. Use 3D convolution (convolution kernel is 1×1×1) to process the concatenated features, learn and fuse the information of the two modalities, and generate the fused features. Through the pooling operation, remove the extra dimension to get the output.

[0140] Finally, the two enhanced branch features are put together and fused through learnable weights. The weights are used to dynamically adjust the fusion ratio of the two modalities to meet the needs of different scenarios and features, and obtain the final fusion output features.

[0141] Furthermore, the image fusion network contains two encoders, which process two input image modalities respectively. Each encoder contains four encoding levels, each of which extracts image features through the DFN module and performs downsampling after each level to halve the feature resolution while increasing the number of channels. At each encoding level, the features of the two modalities are fused separately, and the fusion effect is optimized using the Aix feature fusion module and text embedding guidance. The fused features are gradually upsampled and decoded to restore the image to the original resolution, and finally the fused image is generated through the output layer.

[0142] Specifically, Figure 6As shown in the figure, the visible light image and the infrared image are the input of the image fusion network. The input image is converted into a form suitable for encoder processing through the Flatten operation. The encoder encodes the image features of the two modalities and extracts their respective feature representations (such as F1, 2, 3, 4). These feature representations retain the key information of the input image and are distilled with the features extracted from the brightness adjustment network. The Aix module is used for the preliminary fusion of features of different modalities. The fused features of the Aix module are input into the spatial attention module to weight the features of the key areas, making the features of the important areas more prominent, thereby improving the fusion effect. According to the text guide, this module supports different scene processing: the daytime scene text is used for processing under daytime conditions, emphasizing the image features of high brightness and high contrast. The night scene text is suitable for conditions such as low brightness and cold environments. It is embedded in the fusion process through affine transform to control the network's response to different scenes. The decoder receives the fused features and restores the image spatial resolution through multiple upsampling modules. Finally, after two 3x3 convolutional layers, the output is adjusted to the desired output image format. The final fused image contains the complementary information of visible light and infrared images, and performs well in both daytime and nighttime scenes.

[0143] In summary, the invention proposes an image enhancement and fusion method for low-light environments, and makes multi-faceted improvements to the shortcomings of traditional fusion methods in dark light environments. By designing an innovative network architecture and combining text guidance with distillation technology, the problem of overexposure of daytime images and overexposure of point light sources at night during the fusion process is effectively solved. The feature extraction module adopts the CNN and Mamba parallel structure to fully capture and fuse features from both local texture and global semantic information, and realizes adaptive feature selection and optimization through a dynamic weight adjustment mechanism. The fusion module introduces innovative sequence hybrid splicing and coordinate attention mechanisms to further improve the fusion effect of multimodal features.

[0144] Compared with other fusion methods, this invention has higher image quality and detail retention. Through text guidance and distillation technology, it avoids the loss of image information caused by traditional enhancement methods, especially in low-light environments, it significantly improves the brightness, detail expression and target recognition accuracy of the image. For specific scenes (daytime and nighttime), the enhancement strategy is adaptively adjusted to avoid overexposure caused by excessive enhancement, making the final fused image more natural and clear visually.

[0145] In addition, by utilizing innovative dual-modal feature extraction and fusion, a parallel feature extraction module of CNN and Mamba is introduced, which makes full use of the advantages of both in capturing local and global information, and effectively fuses the two features through a dynamic weight allocation mechanism, ensuring the fusion effect of infrared and visible light images. In the feature fusion module, Mamba solves the information loss problem caused by single-order modeling by sequence mixing and splicing of different orders, while CNN models the dependency between different pixels by introducing a coordinate attention mechanism, achieving more refined feature fusion. The present invention also has effective multi-scene processing capabilities. With the help of the text guidance function of the CLIP model, the network can intelligently choose whether to perform image enhancement according to different scenes (daytime and nighttime), so that the network shows stronger adaptability and flexibility when dealing with complex environments.

[0146] Proven beneficial effects:

[0147] The present invention discloses an image enhancement and fusion method for low-light environments. The method proposes a fusion network architecture based on text guidance and distillation, and a feature extraction and fusion module based on deep learning CNN and Mamba parallelism, aiming to solve the overexposure problem of daytime pictures and the overexposure problem of nighttime point light sources that existed in the previous fusion network in low-light environments. Specifically, the traditional brightness adjustment network lacks highlight images as supervision, so we built a brightness adjustment network trained on a brightness-enhanced dataset (a dataset with pairs of dark and bright image data), but directly embedding the enhancement network in front of the fusion network will inevitably lead to information loss. Therefore, knowledge distillation technology is used to achieve a novel edge-enhancement and edge-fusion strategy through effective information transmission and feature approximation. In order to achieve different processing of different scenes in the fusion network, text-guided processing of different tasks is introduced (no enhancement task is performed for daytime scenes, and enhancement and fusion are performed for nighttime scenes). CLIP is used to encode daytime and nighttime texts respectively, and the text vectors are embedded in the training of the enhancement and fusion networks respectively, generating different enhanced images and performing back propagation optimization through loss calculation; in addition, a new text loss function is designed in the fusion stage. Using the pre-trained single-modal image description text, the fusion network needs to generate image features that are different or unique from the content described by the single-modal image to achieve more effective information fusion. In order to make the fusion model have better feature extraction capabilities, a new DFN feature extraction module with parallel weight allocation is designed in the feature extraction module to fully capture the value of the two extraction blocks and explore the contribution of shallow texture and deep semantic features of the image to infrared image fusion. For the feature fusion part, the Aix fusion module was designed using the idea of ​​parallel weight distribution. However, the infrared features and visible light features extracted by Mamba need to be mixed and spliced ​​and then fused to make up for the insufficient fusion caused by serialization. The CNN block adds a coordinate attention mechanism for preliminary fusion. With the help of the two-dimensional to three-dimensional idea, the added dimension is further adaptively fused and then pooled back to a more comprehensive fusion result. The two modules obtain the final fused image result based on their contribution.

[0148] The technical solution of this application effectively suppresses the daytime overexposure problem existing in the traditional nighttime image fusion algorithm. The pictures in the TNO dataset are taken as reference. Figure 7 As shown, a1 and a2 on the left are the effect diagrams of the traditional fusion algorithm, and b1 and b2 on the right are the effect diagrams of the present application;

[0149] The technical solution of this application also effectively suppresses the problem of overexposure of point light sources at night. The pictures in the LLVIP dataset are taken as follows: Figure 8 As shown, c is an infrared image, d is a visible light image; e1 is a rendering of a traditional fusion algorithm, and e2 is a rendering of the present application.

[0150] The three indicators of the fused image of this application on the TNO dataset are optimal, as shown in Table 1 below:

[0151] Table 1

[0152]

[0153] The four indicators of the fused image of this application on the MSRS dataset are optimal, as shown in Table 2 below:

[0154] Table 2

[0155]

[0156] The four indicators of the fused image of this application on the LLVIP dataset are optimal, as shown in Table 3 below:

[0157] Table 3

[0158]

[0159] Embodiment 3:

[0160] The present invention provides a technical solution:

[0161] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned infrared and visible light image fusion method when executing the computer program.

[0162] In this embodiment, the computer system suitable for implementing the electronic device of the embodiment of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM), such as executing the method described in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other through a bus. Input / output (I / O) interface is also connected to the bus.

[0163] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage part as needed.

[0164] Embodiment 4:

[0165] The present invention provides a technical solution:

[0166] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer-readable storage medium implements the aforementioned infrared and visible light image fusion method. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0170] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0171] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the application point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0172] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0173] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be construed as a limitation to the claims. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A method for fusing infrared and visible light images, characterized in that: The following steps are involved: Acquire a weak visible light image and an infrared image corresponding to the weak visible light image; Constructing an image fusion network, wherein the image fusion network is supervised by supervised labels generated by a pre-trained brightness adjustment network, and a pre-trained prompt word generation network is used to constrain the quality of the fused image; The weak visible light image and the infrared image corresponding to the weak visible light image are input into the image fusion network for fusion to obtain a fused image.

2. The infrared and visible light image fusion method according to claim 1, characterized in that: The brightness adjustment network is trained based on a brightness enhancement dataset. The brightness adjustment network uses text guidance to perform brightness enhancement on weak visible light images at night to improve their imaging quality under low light conditions; and only performs image reconstruction tasks on daytime images.

3. The infrared and visible light image fusion method according to claim 2, characterized in that: Use CLIP to encode images and text simultaneously and calculate the similarity between the two to determine whether the scene matches the text description; The given text is embedded into the training of the brightness adjustment network, and the text encoded by CLIP is presented as a vector; in the process of embedding the text vector into the brightness adjustment network, the guide text is encoded by CLIP to generate a fixed vector with a length of 512; since the number of feature channels of each layer in the network is inconsistent, the KAN network is used to adjust the length of the text vector to match the feature dimension of the current layer; The KAN network expands and adjusts the input text vector through two linear layers, and implements nonlinear transformation through the LeakyReLU activation function, so that the text vector can adapt to the different channel number requirements of each layer of the network.

4. The infrared and visible light image fusion method according to claim 3, characterized in that: An affine transformation is used to more effectively combine text information; the affine transformation performs linear and bias operations on input features by calculating scaling factors and offset factors, thereby ensuring that the text vector can have an impact on the input features.

5. The infrared and visible light image fusion method according to claim 1, characterized in that: The brightness adjustment network includes an encoder, a spatial attention module, an affine transformation module and a decoder, the encoder includes a DFN module and a downsampling module, and the decoder includes a DFN module and an upsampling module; The working process of the brightness adjustment network includes: Flatten the input weak visible light image so that the Mamba module in the DFN module can perform serialization operations; The features of the input weak visible light image are extracted through the DFN module in the encoder, and the local texture and global semantic information are extracted respectively through the parallel CNN and Mamba architectures in the DFN module; The multi-level structure of the DFN module gradually deepens the depth of feature extraction, helping the network capture image details and semantic information at different levels, thereby improving the fusion effect; The downsampling module is used to act on the output of each DFN module to gradually reduce the spatial resolution; The spatial attention module is used to enhance key area features, supplement the deficiencies of global modeling, and highlight significant spatial information. The text features are embedded into each level of the decoder through the affine transformation module, so as to realize text-guided image enhancement. The affine transformation module uses the KANLinear layer and the LeakyReLU activation function to process the text features layer by layer, and generates transformation parameters adapted to the current decoder level. The transformation parameters include scaling factors and offset values, and the generated scaling factors and offset values ​​are combined with the image features in the decoder to realize text-based guidance. Each layer of DFN module in the decoder can receive the text feature guidance after the affine transformation module to ensure that the decoder outputs the fused text information layer by layer. After decoding and fusion of all levels, the final output image is generated through the convolution layer. Each upsampling module in the decoder is used to double the spatial resolution of the feature map, thereby gradually restoring it to the same resolution as the input image. After each upsampling module, the feature map in the decoder enters the next DFN module for further integration and processing of features; The convolution layer in the decoder is used to convert the output feature map of the decoder into the final fused image; the convolution layer uses a 3×3 convolution kernel.

6. The infrared and visible light image fusion method according to claim 5, characterized in that: The DFN module includes two branches, namely the Mamba Block module and the Multi-CNN module. The two branches are weightedly fused through learnable weights to achieve adaptive feature extraction of global semantic information and local texture information. The Mamba Block module is used to extract semantic information of the image from a global perspective. The Mamba Block module includes a Norm module, three MLP modules, two Conv modules, a Forward SSM module, a Backward SSM module and an Activation module. The three MLP modules are respectively a first MLP module, a second MLP module and a third MLP module. In the Mamba Block module, the input data is standardized through the Norm module to ensure stable feature distribution and improve numerical stability during training; The first MLP module is used to perform a preliminary nonlinear transformation on the input features; the features processed by the first MLP module are sent to the subsequent Conv module and state space model to enrich the feature expression; the Conv module extracts local information from the input features and extracts local temporal patterns or spatial features; The convolved features are sent to the Forward SSM module and the Backward SSM module; the Forward SSM module and the Backward SSM module are used to capture the forward and backward relations of sequence features respectively; the Forward SSM module extracts features through the state equation in the forward direction, while the Backward SSM module captures the reverse relation through the state equation in the backward direction; Next, the output is passed to the Activation module through the second MLP module, allowing the model to perform nonlinear transformations; the activation features output from the Activation module are multiplied element-by-element with the outputs of the Forward SSM module and the Backward SSM module respectively; After element-wise multiplication, the forward and backward features are added together to form a more robust intermediate feature that combines information from both directions; The combined features are further processed by the third MLP module to further abstract the features and help the model extract more useful information; The initial input features are added to the output of the third MLP module, and the original feature details are preserved through residual connections, so that low-level and high-level features can be integrated in the final output; The Multi-CNN module is used to extract local texture information, and it adopts a parallel structure of convolutional layers with different convolution kernel sizes to obtain multi-scale image features; The Multi-CNN module includes 3x3 convolution kernels, 5x5 convolution kernels, and 7x7 convolution kernels; the 3x3 convolution kernel is used to capture the detailed features of the image; the 5x5 convolution kernel has a larger receptive field than the 3x3 convolution kernel, can capture texture and edge information in a larger range, and can extract medium-scale image features; the 7x7 convolution kernel has a larger receptive field than the 5x5 convolution kernel, can capture global texture and structural information, and can extract larger-scale features; the Multi-CNN module can capture rich texture detail information at different scales by extracting features of different convolution kernel sizes in parallel; The multi-scale features are spliced ​​to form a fused feature representation with multi-level texture information; The DFN module also includes a weight fusion module; in the DFN module, the features extracted by the two branches of the Mamba Block module and the Multi-CNN module are fused through learnable weights; the weight fusion module is used to adaptively adjust the feature contributions of the two branches of the Mamba Block module and the Multi-CNN module; Through training, the network can learn the role of the Mamba Block module and the Multi-CNN module in different channels and spatial positions, thereby achieving better feature extraction; the features of the two branches of the Mamba Block module and the Multi-CNN module are weighted and added according to the weights to obtain the final fused feature representation.

7. The infrared and visible light image fusion method according to claim 1, characterized in that: The input of the image fusion network is a weak visible light image and an infrared image corresponding to the weak visible light image; The input weak visible light image and the infrared image corresponding to the weak visible light image are converted into a form suitable for encoder processing through the Flatten operation; Encoding the image features of the two modalities through an encoder, extracting their respective feature representations, and distilling them with the features extracted from the brightness adjustment network; The Aix module is used to initially fuse the features of different modalities, and the fused features of the Aix module are input into the spatial attention module to weight the features of the key areas, making the features of the key areas more prominent, thereby improving the fusion effect; Based on the text guidance, the Aix module supports different scene processing; the day scene text is used for processing under daytime conditions, emphasizing the image features of high brightness and high contrast; the night scene text is suitable for low brightness and cold environment conditions; by embedding affine transformation into the fusion process, the image fusion network's response to different scenes is controlled; The decoder receives the fused features and restores the image spatial resolution through multiple upsampling modules; After two 3x3 convolutional layers, the output is adjusted to the expected output image format to obtain a fused image.

8. The infrared and visible light image fusion method according to claim 7, characterized in that: The input of the Aix module is infrared features and visible light features, and the Aix module is a parallel branch design of Mamba and CNN; In the Mamba branch, the input features are flattened; layer normalization is performed to ensure the stability of the input; the Mamba block module is entered to capture the sequence features through the Mamba block module for global modeling; after serialization, the two modal features are concatenated in different orders to capture the impact of the sequence order on feature modeling; The splicing operation forms two different sequence combinations, namely a first combination and a second combination; the first combination is that the visible light is in front and the infrared is in the back; the second combination is that the infrared is in front and the visible light is in the back; The two sequences are deserialized back to the original two-dimensional feature form, and the serialized global features are remapped back to the spatial domain to restore the spatial information of H and W dimensions. Learnable weights are applied to the two split features to balance the fusion proportion of the two, and the contribution of different modalities to the final features is adaptively adjusted according to the specific scenario. In the CNN branch, the input features are pooled in the H and W dimensions to compress the spatial information into the channel dimension so that local information can be extracted in a smaller spatial range later. After pooling along the H dimension, features of shape B×C×1×W are obtained, and after pooling along the W dimension, features of shape B×C×H×1 are obtained. The four vectors are concatenated to form a tensor with a shape of B×C×(2(W+H))×1; the concatenated features are sent to MLP for channel attention learning, specifically: squeeze to reduce the number of channels, reduce the feature dimension from 2(W+H), and then excite to restore the dimension; learn the dependency between different pixels through the squeeze-excitation mechanism; split the MLP output into two parts, each used to enhance the input features; and use the Sigmoid activation function to generate channel attention weights, add a new dimension to the two enhanced features, and convert it to B×1×C×H×W through the unsqueeze operation, and then concatenate along the newly added dimensions to obtain a tensor with a shape of B×2×C×H×W; use 3D convolution to process the concatenated features, learn and fuse the information of the two modalities, and generate the fused features; remove the extra dimensions through the pooling operation to obtain the output; The two enhanced branch features are put together and fused through learnable weights; the weights are used to dynamically adjust the fusion ratio of the two modalities to adapt to the needs of different scenarios and features, and obtain the final fused output features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the infrared and visible light image fusion method as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the infrared and visible light image fusion method as described in any one of claims 1 to 8 is implemented.

Images