Infrared and visible light image fusion method and system for low-illumination scene

Abstract

The application provides an infrared and visible light image fusion method and system for a low-illumination scene, which comprises the following steps: calling a target fusion network based on a target text prompt to fuse an infrared image and a visible light image, and obtaining a target image; the training process of the target fusion network comprises the following steps: extracting features of an infrared image sample and a visible light image sample to obtain shared visual features; modulating the shared visual features based on a semantic modulation weight corresponding to an embedding vector of a sample text prompt to generate conditioned visual features; processing the conditioned visual features and the shared visual features based on the embedding vector to generate a fusion image; determining various losses based on the fusion image, a spatial weight map corresponding to the visible light image sample, and an infrared weight map, and training an initial fusion network to obtain the target fusion network. The application can improve the infrared and visible light image fusion efficiency under low illumination, realize pixel-level feature screening, has strong scene adaptability, and has high detail fidelity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for fusing infrared and visible light images in low-light scenes. Background Technology

[0002] Infrared and Visible Image Fusion (IVIF) is a widely used multimodal image processing technique that fuses visible light images and infrared images through image fusion technology to generate images that simultaneously possess thermal target salience and scene texture details.

[0003] With the development of deep learning technology, some deep learning-based IVIF technologies have emerged. Among these existing technologies, most are based on convolutional neural networks and Transformer architecture to build fusion frameworks. Some methods use the long-distance dependency capture capability of Transformer to improve the global feature extraction effect, while other technologies attempt to introduce vision-language pre-trained models to guide the image fusion process through text semantic information.

[0004] However, the aforementioned existing technologies still have some shortcomings, especially in handling complex real-world scenes, particularly in extreme low-light scenarios. Specifically, most existing technologies only perform signal processing at the pixel or feature level, ignoring high-level semantic information in the image. This results in key heat source targets such as pedestrians and vehicles becoming blurred during the fusion process due to the averaging of pixel intensity, making it difficult to effectively highlight them in complex backgrounds. Furthermore, in low-light scenes at night or with insufficient light, existing algorithms struggle to adaptively distinguish effective textures from low-light noise and dark areas in visible light images. Forced fusion can easily lead to infrared heat source targets being contaminated by the dark background of visible light or introducing a large amount of high-frequency noise.

[0005] Therefore, there is an urgent need to develop a semantically driven multimodal image fusion method for low-light scenes to make up for the shortcomings of existing technologies when facing complex and realistic low-light extreme scenes. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method and system for fusing infrared and visible light images in low-light scenes.

[0007] This invention provides a method for fusing infrared and visible light images in low-light scenes, comprising: Obtain the infrared image and the visible light image to be fused, as well as the target text prompts with explicit semantic guidance corresponding to the infrared image and the visible light image; Based on the target text prompt, the target fusion network is invoked to fuse the infrared image and the visible light image to obtain the target image; The training process of the target fusion network includes: Obtain sample images and sample text prompts for the sample images in low-light scenes, and encode the sample text prompts to obtain an embedding vector; Based on a dual-stream encoder, multi-scale feature extraction is performed on infrared and visible light image samples in the sample images. Then, the samples are stitched together in the channel dimension and cross-modal information is interacted and dimensionally compressed to obtain shared visual features. The embedded vector is processed by the text conditional modulation module to obtain semantic modulation weights. Based on the semantic modulation weights, the shared visual features are subjected to channel-dimensional text conditional modulation to generate semantically enhanced conditional visual features. Based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and the shared visual features. After multiple iterations, the final fusion feature is generated, and the fusion image is reconstructed based on the final fusion feature by the image decoder. The spatial weight map and the complementary infrared weight map are determined based on the gradient magnitude of the visible light image samples. Based on the fused image, the spatial weight map, and the infrared weight map, the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss are determined. Based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss, the initial fusion network is trained end-to-end until a preset convergence condition is met, thus obtaining the target fusion network.

[0008] According to the present invention, an infrared and visible light image fusion method for low-light scenes includes acquiring sample images and sample text prompts of the sample images in the low-light scene, and encoding the sample text prompts to obtain an embedding vector, comprising: Acquire infrared and visible light image data; Spatial registration and size normalization are performed on the infrared image data and the visible light image data respectively to obtain the infrared image sample and the visible light image sample; Based on the preset fusion task scenario, natural language keywords that can characterize the salient features of the source image are retrieved from the preset lexicon and embedded into the predefined instruction template to construct sample text prompts with explicit semantic guidance for the infrared image sample and the visible light image sample. The sample text prompts are encoded and mapped to a semantic latent space using a contrastive language image pre-trained text encoder to generate the high-dimensional embedding vector.

[0009] According to the present invention, an infrared and visible light image fusion method for low-light scenes is provided. The dual-stream encoder includes an infrared processing branch and a visible light processing branch. The infrared processing branch and the visible light processing branch include multiple cascaded Swing Transformer modules. The shared visual features are extracted by the l-th feature extraction layer in the dual-stream encoder. The l-th feature extraction layer includes the l-th Swing Transformer module in the infrared processing branch and the l-th Swing Transformer module in the visible light processing branch. The method involves multi-scale feature extraction of infrared and visible light image samples from the sample images based on a dual-stream encoder, followed by channel-level stitching, cross-modal information interaction, and dimensionality compression to obtain shared visual features, including: The infrared image sample is input into the infrared processing branch, and the visible light image sample is input into the visible light processing branch for parallel processing. The shift window self-attention mechanism is used to capture the multi-scale features of the infrared image sample and the visible light image sample, so as to obtain the infrared modal features of the l-th feature extraction layer and the visible light modal features of the l-th feature extraction layer. The infrared modal features and the visible light modal features are concatenated along the channel dimension, and a 1*1 convolutional layer is used to achieve cross-modal information interaction and dimensional compression, generating the shared visual features of the l-th feature extraction layer.

[0010] According to the infrared and visible light image fusion method for low-light scenes provided by the present invention, the process of obtaining the infrared modal features of the l-th feature extraction layer is as follows: ; in, The infrared modal features of the l-th feature extraction layer; This is the feature extraction mapping function of the l-th Swin Transformer module in the infrared processing branch; The process of obtaining the visible light modal features of the l-th feature extraction layer is as follows: ; in, The visible light modal features of the l-th feature extraction layer; This is the feature extraction mapping function of the l-th Swing Transformer module in the visible light processing branch; The process of generating the shared visual features in the l-th feature extraction layer is as follows: ; in, The shared visual features of the l-th feature extraction layer; This is a splicing operation at the channel level; This indicates a convolution operation with a kernel size of 1*1.

[0011] According to the present invention, an infrared and visible light image fusion method for low-light scenes includes processing the embedded vector through a text conditional modulation module to obtain semantic modulation weights, and performing channel-dimensional text conditional modulation on the shared visual features based on the semantic modulation weights to generate semantically enhanced conditional visual features, comprising: The embedded vector is projected onto the channel dimension space of the shared visual features using the lightweight projection head in the text conditional modulation module, and then processed by the Sigmoid activation function in the text conditional modulation module to obtain the semantically guided semantic modulation weights. The shared visual features are generated by performing element-wise multiplicative modulation of the channel dimension in residual form based on the semantic modulation weights.

[0012] According to the infrared and visible light image fusion method for low-light scenes provided by the present invention, the generation process of the semantic modulation weights is as follows: ; Wherein, W is the semantic modulation weight; This is a linear projection mapping operation; The Sigmoid activation function is used to normalize the semantic modulation weights to a range of 0 to 1. The process of generating the conditional visual features is as follows: ; in, For the conditional visual features; This is an element-wise multiplication operation for the channel dimension.

[0013] According to the present invention, an infrared and visible light image fusion method for low-light scenes is provided, wherein based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and the shared visual features, and after multiple iterations, a final fusion feature is generated. A fused image is then reconstructed based on the final fusion feature using an image decoder, including: Using the conditional visual features as query terms and the shared visual features as key and value terms, a window partitioning operation is performed on the conditional visual features and the shared visual features to decompose the feature maps of the conditional visual features and the shared visual features into non-overlapping local regions. Each of the local regions is mapped to a high-dimensional feature space through a linear projection layer to obtain the projected features. The projected features are subjected to window cross attention calculation to obtain fused residual increment features, which contain potential fusion information and spatial attention distribution guided by text semantics. The fused residual incremental features are mapped using a 3*3 convolutional layer and processed by a Sigmoid activation function to obtain a gated weight map. The fused residual incremental features are spatially weighted according to the gating weight map, and the shared visual features are updated through the residual connection mechanism. The shared visual features are updated layer by layer in a residual manner after semantically guided attention correction and gating screening to obtain the final fused features; The final fusion features are input into the image decoder to reconstruct the fused image with the same resolution as the infrared image sample and the visible light image sample.

[0014] According to the present invention, an infrared and visible light image fusion method for low-light scenes is provided, wherein determining a spatial weight map and a complementary infrared weight map based on the gradient magnitude of the visible light image samples includes: Calculate the gradient magnitude of the visible light image sample; The gradient magnitude is normalized to obtain the spatial weight map. The normalization process is as follows: ; in, This is the spatial weighting graph; The gradient magnitude of the visible light image sample; It is the maximum value operator; It is a constant; Subtracting 1 from the spatial weight map yields the infrared weight map.

[0015] According to the infrared and visible light image fusion method for low-light scenes provided by the present invention, the process of determining the adaptive gradient loss is as follows: ; in, The adaptive gradient loss; for Norm; For gradient operators; The infrared weighted map; Element-wise multiplication representing spatial dimensions; The fused image; The infrared image sample; This is the spatial weighting graph; The visible light image sample; The process for determining the weighted color loss is as follows: ; in, The weighted color loss; The process for determining the semantic contrast loss is as follows: ; in, The semantic contrast loss; Characterizes image encoding operations; The embedding vector; The process for determining the basic pixel loss is as follows: ; in, The loss is the basic pixel loss; For pixel-wise averaging operator, express Norm; The initial fusion network is trained end-to-end based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss until a preset convergence condition is met, resulting in the target fusion network, including: Based on the adaptive gradient loss and the weighted color loss, determine the spatial adaptive constraint loss; The spatial adaptive constraint loss, the basic pixel loss, and the semantic contrast loss are weighted and summed to obtain the total loss. Based on the total loss, an optimization algorithm is used to train the initial fusion network end-to-end until the preset convergence condition is met, thereby obtaining the target fusion network. The preset convergence condition is that the total loss reaches a convergent state.

[0016] The present invention also provides an infrared and visible light image fusion system for low-light scenes, comprising: The acquisition unit is configured to acquire the infrared image and the visible light image to be fused, as well as the target text prompts with explicit semantic guidance corresponding to the infrared image and the visible light image; The fusion unit is configured to, based on the target text prompt, invoke a target fusion network to fuse the infrared image and the visible light image to obtain a target image; The training unit is configured as follows: Obtain sample images and sample text prompts for the sample images in low-light scenes, and encode the sample text prompts to obtain an embedding vector; Based on a dual-stream encoder, multi-scale feature extraction is performed on infrared and visible light image samples in the sample images. Then, the samples are stitched together in the channel dimension and cross-modal information is interacted and dimensionally compressed to obtain shared visual features. The embedded vector is processed by the text conditional modulation module to obtain semantic modulation weights; based on the semantic modulation weights, the shared visual features are subjected to channel-dimensional text conditional modulation to generate semantically enhanced conditional visual features. Based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and the shared visual features. After multiple iterations, the final fusion feature is generated, and the fusion image is reconstructed based on the final fusion feature by the image decoder. Based on the gradient magnitude of the visible light image samples, a spatial weight map and a complementary infrared weight map are determined; based on the fused image, the spatial weight map, and the infrared weight map, basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss are determined. Based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss, the initial fusion network is trained end-to-end until the preset convergence condition is met, thus obtaining the target fusion network.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the infrared and visible light image fusion method for low-light scenes as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the infrared and visible light image fusion method for low-light scenes as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an infrared and visible light image fusion method for low-light scenes as described above.

[0020] This invention provides a method and system for fusing infrared and visible light images in low-light scenes. Unlike existing technologies that rely on statistical methods such as average and maximum pixel values ​​for fusion, this method introduces sample text prompts and their embedded vectors. Instead, it uses natural language prompts to enable the fusion network to explicitly understand the image content. Based on this, and using the fused image, spatial weight map, and infrared weight map, it automatically makes a dynamic trade-off between "preserving visible light texture" and "preserving infrared thermal radiation," effectively solving modal conflicts and noise pollution in low-light conditions. It addresses the problem of visible light images containing a lot of noise and invalid black backgrounds in nighttime or low-light scenes, and uses cross-modal cross-attention and dynamic spatial feature filtering to achieve pixel-level feature filtering. Compared with fixed-rule fusion methods, it has stronger scene adaptability and detail fidelity. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the infrared and visible light image fusion method for low-light scenes provided by the present invention.

[0023] Figure 2 This is one of the flowcharts of the training target fusion network provided by the present invention.

[0024] Figure 3 This is the second flowchart of the training target fusion network provided by the present invention.

[0025] Figure 4 This is a schematic diagram of the process for determining shared visual features provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the process for generating residual increment features provided by the present invention.

[0027] Figure 6 This is a schematic diagram of the process for generating fused images provided by the present invention.

[0028] Figure 7 This is a flowchart illustrating the spatial adaptive optimization strategy provided by the present invention.

[0029] Figure 8 This is a schematic diagram of the infrared and visible light image fusion system for low-light scenes provided by the present invention.

[0030] Figure 9This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] The following description, in conjunction with the accompanying drawings, describes a method and system for fusing infrared and visible light images in low-light scenes according to the present invention.

[0033] Figure 1 This is a flowchart illustrating the infrared and visible light image fusion method for low-light scenes provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain the infrared image and the visible light image to be fused, as well as the target text prompts with explicit semantic guidance corresponding to the infrared image and the visible light image; Step 102: Based on the target text prompt, call the target fusion network to fuse the infrared image and the visible light image to obtain the target image.

[0034] In practical applications, after obtaining the infrared and visible light images to be fused, explicit semantically guided target text prompts can be provided for the infrared and visible light images: natural language keywords that can characterize the salient features of the source images are retrieved from a preset thesaurus and embedded into a predefined instruction template to construct explicit semantically guided target text prompts for the infrared and visible light images respectively. For example, for an image involving an infrared camera, a target text prompt of "infrared thermal target" is embedded.

[0035] Furthermore, guided by the target text prompts, the trained target fusion network is used to fuse the infrared image and the visible light image to obtain the fused image, i.e., the target image.

[0036] It should be noted that, see Figure 2 , Figure 2 This is one of the flowcharts of the training target fusion network provided by the present invention, such as... Figure 2 As shown, the training process of the target fusion network includes the following steps 201 to 206.

[0037] Step 201: Obtain sample images and sample text prompts for low-light scenes, and encode the sample text prompts to obtain embedding vectors.

[0038] In practical applications, infrared and visible light image data to be fused can be obtained from publicly available infrared and visible light fusion datasets such as Multi-Spectral RoadScenarios (MSRS), RoadScene, and Low-Light Visible-Infrared Pairing Dataset (LLVIP). Representative image data covering various typical monitoring scenarios such as urban streets, rural roads, and indoor environments are selected. The infrared image data has corresponding visible light image data, and the two are combined into an image data pair. The obtained image data pairs are divided into training set, validation set, and test set in a ratio of 8:1:1.

[0039] For example, based on image data from representative surveillance scenes selected from MSRS, RoadScene, and LLVIP, a total of 2400 image data pairs were obtained. These were randomly and mutually exclusively partitioned statically according to a preset ratio of 8:1:1, with the specific size and sample size of each dataset remaining the same as described above. To ensure the robustness and generalization ability of the trained target fusion network in real-world complex environments, the 2400 image data pairs exhibited high diversity in scene and category distribution (including urban roads and traffic scenes during the day and night). Furthermore, when partitioning into training, validation, and test sets, the proportion of each scene remained consistent; that is, for each scene, the training, validation, and test sets were partitioned separately according to an 8:1:1 ratio. This ensures that the training, validation, and test sets remain isolated from each other in the data flow.

[0040] The training set is used to train the initial fusion network. After each preset number of training iterations on the initial fusion network based on the training set, the current network parameters are frozen. The validation set is then input into the current fusion network for forward inference to obtain the fused image. The average score of this fused image on a preset objective metric is calculated. This validation process does not trigger gradient backpropagation or update the network's internal parameters. After training is complete, the changes in the metrics of each network parameter on the validation set are compared. Finally, the network parameters that achieve the highest peak value in the overall metric on the validation set and whose loss function fully converges are selected. The network weight parameter matrix corresponding to the highest-performing iteration under these network parameters is locked as the target fusion model. The test set is used to test the fusion metrics of the target fusion model, such as fusion rate and fusion success rate.

[0041] Based on the obtained training set, it can be preprocessed to obtain sample images: the training set contains multiple image data pairs, each pair including infrared image data and visible light image data; for each image data pair, the infrared image data and visible light image data are preprocessed separately to obtain infrared image samples. and visible light image samples Among them, infrared image samples and visible light image samples This constitutes a sample image.

[0042] For the sample images, a sample text cue T with explicit semantic guidance is constructed. The sample text cue T is encoded using a Contrastive Language-Image Pre-training (CLIP) text encoder to generate embedding vectors. .

[0043] Step 202: Based on the dual-stream encoder, perform multi-scale feature extraction on the infrared image samples and visible light image samples in the sample images, and perform splicing in the channel dimension as well as cross-modal information interaction and dimensional compression to obtain shared visual features.

[0044] Specifically, the dual-stream encoder includes an infrared processing branch and a visible light processing branch, each branch integrating L cascaded Swing Transformer modules, where L is a positive integer greater than 1; the dual-stream encoder includes L feature extraction layers, the l-th feature extraction layer being composed of the l-th Swing Transformer module in the infrared processing branch and the l-th Swing Transformer module in the visible light processing branch, where l is any integer in [1, L].

[0045] The infrared processing branch and the visible light processing branch process the infrared image samples respectively. and visible light image samples Multi-scale feature extraction is performed, and concatenation, cross-modal information interaction, and dimensionality compression are carried out along the channel dimension to generate shared visual features corresponding to the l-th feature extraction layer. .

[0046] Step 203: Process the embedded vector through the text conditional modulation module to obtain semantic modulation weights, and perform channel-dimensional text conditional modulation on the shared visual features based on the semantic modulation weights to generate semantically enhanced conditional visual features.

[0047] Specifically, a text conditional modulation module can be used to embed vectors. Generate semantic modulation weights W, and then use the semantic modulation weights W to adjust the shared visual features. Perform channel-level text conditional modulation to generate semantically enhanced conditional visual features. .

[0048] Step 204: Based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on conditional visual features and shared visual features. After multiple iterations, the final fused features are generated, and the fused image is reconstructed based on the final fused features by the image decoder.

[0049] Specifically, in the embedding vector Under semantic guidance, conditional visual features With shared visual features Cross-modal cross-attention fusion and dynamic spatial feature selection are performed, and the final fused feature is generated after multiple iterations. And generate a fused image by reconstructing it using an image decoder. .

[0050] Step 205: Determine the spatial weight map and the complementary infrared weight map based on the gradient magnitude of the visible light image samples.

[0051] Specifically, calculate visible light image samples The gradient magnitude is calculated and normalized to obtain the spatial weight map. Sum and Spatial Weights Plot Complementary infrared weighted map .

[0052] Step 206: Based on the fused image, spatial weight map and infrared weight map, determine the basic pixel loss, adaptive gradient loss, weighted color loss and semantic contrast loss, and perform end-to-end training on the initial fusion network based on the basic pixel loss, adaptive gradient loss, weighted color loss and semantic contrast loss until the preset convergence condition is met to obtain the target fusion network.

[0053] The initial fusion network includes a dual-stream encoder, a text conditional modulation module, and an image decoder.

[0054] Specifically, it can be based on fused images Spatial weighting graph With infrared weighted map The basic pixel loss is calculated. Adaptive gradient loss Weighted color loss Compared with semantic contrast loss .

[0055] Then, based on the base pixel loss Adaptive gradient loss Weighted color loss Compared with semantic contrast loss Construct the final total loss This guides the initial fusion network through end-to-end training until the total loss is reached. Once convergence is achieved, the trained target fusion network can adaptively fuse infrared and visible light images under the guidance of textual semantics. The preset convergence condition is the total loss. It has reached a convergent state.

[0056] In this embodiment, a spatial adaptive optimization strategy (SA-Opt) can be introduced during the training phase of the initial fusion network to resolve modal conflicts under low illumination.

[0057] For example, see Figure 3 , Figure 3 This is the second flowchart of the training target fusion network provided by the present invention: Step 301: Obtain infrared image samples and visible light image samples, and construct text prompt samples; Step 302: Generate high-dimensional embedding vectors using a text encoder; Step 303: Construct a dual-stream encoder to extract shared visual features; Step 304: Perform text conditional modulation to generate conditional visual features; Step 305: Text-driven attention fusion is performed to conduct cross-modal feature alignment and spatial filtering; Step 306: Construct the spatial weight map and the infrared weight map; Step 307: Construct the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss; Step 308: Perform end-to-end training on the initial fusion network to obtain the target fusion network.

[0058] The infrared and visible light image fusion method for low-light scenes provided by this invention introduces sample text prompts and their embedded vectors. Unlike existing technologies that rely on statistical methods such as the average and maximum values ​​of underlying pixels for fusion, this method uses natural language prompts to enable the fusion network to explicitly understand the image content. Based on this, and using the fused image, spatial weight map, and infrared weight map, it automatically makes a dynamic trade-off between "preserving visible light texture" and "preserving infrared thermal radiation," effectively solving modal conflicts and noise pollution in low-light conditions. It addresses the problem of visible light images containing a lot of noise and invalid black backgrounds in nighttime or low-light scenes, and uses cross-modal cross-attention and dynamic spatial feature filtering to achieve pixel-level feature filtering. Compared with fixed-rule fusion methods, it has stronger scene adaptability and detail fidelity.

[0059] Optionally, sample images and sample text prompts for low-light scenes are obtained, and the sample text prompts are encoded to obtain an embedding vector, including: Acquire infrared and visible light image data; Infrared image data and visible light image data are spatially registered and size normalized respectively to obtain infrared image samples and visible light image samples. Based on the preset fusion task scenario, natural language keywords that can characterize the salient features of the source image are retrieved from the preset lexicon and embedded into the predefined instruction template to construct sample text prompts with explicit semantic guidance for infrared image samples and visible light image samples. The sample text prompts are encoded and mapped to the semantic latent space using a contrastive language image pre-trained text encoder, generating high-dimensional embedding vectors.

[0060] In practical applications, infrared and visible light image data from the training set can be acquired. Furthermore, various existing technologies can be used to perform spatial registration and size normalization on the infrared and visible light image data.

[0061] Specifically, affine transformation or feature-point-based alignment algorithms can be used to perform rigorous spatial registration of infrared and visible light image data. Subsequently, the dimensions of all infrared and visible light image data are normalized to 640×640 pixels, and the pixel grayscale values ​​are linearly mapped within the range [0,1] to obtain infrared and visible light image samples. This eliminates the impact of sensor gain differences on model training stability.

[0062] More specifically, during spatial registration, for infrared and / or visible light image data with small geometric displacements, fast alignment is achieved through an affine transformation matrix; for infrared and / or visible light image data with complex nonlinear deformations, a homography mapping is constructed using feature point matching techniques.

[0063] Furthermore, based on the preset fusion task scenario, natural language keywords that can characterize the salient features of the source image can be retrieved from a preset lexicon and embedded into a predefined instruction template for each infrared image sample. With visible light image samples Each sample text prompt T with explicit semantic guidance is constructed. The preset fusion task scenario can be target reconnaissance, environmental monitoring, etc., and the natural language keywords can be, for example, infrared thermal targets, visible light texture details, etc.

[0064] Then, the sample text cue T is encoded using the CLIP text encoder and mapped to the semantic latent space to generate a high-dimensional text embedding vector, i.e., the embedding vector. Embedded vector As a global semantic prior, it is used in subsequent steps for text modulation to achieve channel-level feature enhancement and further guides attention fusion, thereby achieving pixel-level semantic feature alignment.

[0065] Optionally, the dual-stream encoder includes an infrared processing branch and a visible light processing branch. The infrared processing branch and the visible light processing branch include multiple cascaded Swing Transformer modules. The shared visual features are extracted by the l-th feature extraction layer in the dual-stream encoder. The l-th feature extraction layer includes the l-th Swing Transformer module in the infrared processing branch and the l-th Swing Transformer module in the visible light processing branch. Multi-scale feature extraction is performed on infrared and visible light image samples from the sample images based on a dual-stream encoder. Then, the samples are stitched together along the channel dimension, and cross-modal information is interacted with and dimensionality compressed to obtain shared visual features, including: Infrared image samples are input into the infrared processing branch, and visible light image samples are input into the visible light processing branch for parallel processing. The shift window self-attention mechanism is used to capture the multi-scale features of the infrared image samples and the visible light image samples, so as to obtain the infrared modal features of the l-th feature extraction layer and the visible light modal features of the l-th feature extraction layer. Infrared modal features and visible light modal features are concatenated along the channel dimension, and a 1*1 convolutional layer is used to achieve cross-modal information interaction and dimensionality compression, generating shared visual features for the l-th feature extraction layer.

[0066] In practical applications, infrared image samples can be used. With visible light image samples The infrared processing branch and the visible light processing branch are processed in parallel, respectively, and infrared image samples are captured using a shift-window self-attention mechanism. With visible light image samples Multi-scale features are used to achieve multi-scale feature extraction, thereby obtaining the infrared modal features of the l-th layer (the l-th feature extraction layer). and the visible light mode characteristics of the lth layer Then, infrared modal features With visible light mode characteristics The data is concatenated along the channel dimension, and a 1*1 convolutional layer is used to achieve cross-modal information interaction and dimensionality compression, ultimately generating the shared visual features of the l-th layer. .

[0067] See Figure 4 , Figure 4This is a flowchart illustrating the process of determining shared visual features provided by the present invention: an infrared image sample and a visible light image sample are input into a dual-stream encoder; the infrared processing branch extracts features from the infrared image sample, and the visible light processing branch extracts features from the visible light image sample; then, feature channel splicing and mapping are performed, i.e., preliminary cross-modal feature interaction is executed to generate shared visual features.

[0068] Shared visual features The determination process actually uses a shift-window self-attention mechanism to capture infrared image samples. With visible light image samples Multi-scale features are processed through splicing and convolutional layers to obtain shared visual features. This shared visual feature It integrates physical structure information from both infrared and visible light modes, which is used as a visual benchmark for subsequent semantic guidance.

[0069] Specifically, the process of acquiring the infrared modal features of the l-th feature extraction layer is as follows: ; in, The infrared modal features of the l-th feature extraction layer; This is the feature extraction mapping function for the l-th Swin Transformer module in the infrared processing branch; The process of obtaining the visible light modal features of the l-th feature extraction layer is as follows: ; in, The visible light modal features are those of the l-th feature extraction layer; This is the feature extraction mapping function of the l-th Swin Transformer module in the visible light processing branch; The process of generating the shared visual features of the l-th feature extraction layer is as follows: ; in, The shared visual features of the l-th feature extraction layer; This is a splicing operation at the channel level; This indicates a convolution operation with a kernel size of 1*1.

[0070] Optionally, the embedded vector is processed by a text conditional modulation module to obtain semantic modulation weights. Based on these semantic modulation weights, the shared visual features are subjected to channel-dimensional text conditional modulation to generate semantically enhanced conditional visual features, including: The embedded vectors are projected onto the channel dimension space of shared visual features using the lightweight projection head in the text conditional modulation module, and then processed by the sigmoid activation function in the text conditional modulation module to obtain semantically oriented semantic modulation weights. Conditional visual features are generated by performing element-wise multiplicative modulation of the channel dimension of shared visual features in the form of residuals based on semantic modulation weights.

[0071] In practical applications, a lightweight projection head is used to embed the vector. Projected onto shared visual features The channel dimension space is defined, where the lightweight projection head is composed of a multilayer perceptron (MLP); then, semantically guided multiplicative semantic modulation weights W are generated through a sigmoid activation function. In this way, the semantic modulation weights W can have semantic guidance functionality.

[0072] Specifically, the process of generating semantic modulation weights is as follows: ; Where W is the semantic modulation weight; This is a linear projection mapping operation; The Sigmoid activation function is used to normalize semantic modulation weights to a range of 0 to 1.

[0073] Furthermore, semantic modulation weights W are used to adjust the shared visual features. Perform multiplicative modulation in residual form to generate conditional visual features. This process allows for the complete preservation of the original visual structure while dynamically adjusting key channels such as thermal targets using textual semantics.

[0074] Specifically, the process of generating conditional visual features is as follows: ; in, Conditional visual features; This is an element-wise multiplication operation for the channel dimension.

[0075] In the above formula, by introducing a residual structure with a constant of 1, the original visual structure is preserved intact while using textual semantics to dynamically enhance key channels such as hot targets.

[0076] Optionally, based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on conditional visual features and shared visual features. After multiple iterations, the final fused features are generated, and the fused image is reconstructed based on the final fused features using an image decoder, including: Using conditional visual features as query terms and shared visual features as key and value terms, a window partitioning operation is performed on the conditional visual features and shared visual features to decompose the feature maps of both conditionsal visual features and shared visual features into non-overlapping local regions. Each local region is mapped to a high-dimensional feature space through a linear projection layer to obtain the projected features. The projected features are subjected to window cross attention calculation to obtain fused residual increment features, which contain potential fusion information and spatial attention distribution guided by text semantics. The fused residual incremental features are mapped through a 3*3 convolutional layer and processed by a Sigmoid activation function to obtain a gated weight map. Spatial weighting is applied to the fused residual incremental features based on the gating weight graph, and the shared visual features are updated through the residual connection mechanism. Shared visual features are updated layer by layer in a residual manner after semantically guided attention correction and gating screening to obtain the final fused features; The final fused features are input into the image decoder to reconstruct a fused image with the same resolution as the infrared image sample and the visible light image sample.

[0077] In practical applications, this embodiment first constructs a Guided Adaptive Fusion (GAF) module, which aims to achieve precise alignment of visual features in the spatial dimension and key information filtering under the explicit guidance of text semantics.

[0078] Specifically, a window refers to dividing a feature map of size H*W into a series of non-overlapping local spatial regions. For example, if the feature map size of conditional visual features is 224*224, and a window size of 7*7 is used, then the entire feature map will be decomposed into 32*32 independent local windows.

[0079] More specifically, see Figure 5 , Figure 5 This is a flowchart illustrating the process of generating residual incremental features provided by the present invention. Conditional visual features are used as the query term (Q), and shared visual features are used as the key and value terms (K,V). Window segmentation and linear projection are performed. After mapping each local region to a high-dimensional feature space through a linear projection layer, the projected features are input into a customized window cross-attention module, i.e., the WindowCross-Attention module, to perform window cross-attention calculation and generate fused residual incremental features, i.e., guided attention fusion. The specific process is as follows: ; in, To integrate residual increment characteristics; This represents a feature recombination and filtering mechanism, which calculates query items. and key items The correlation is used to determine the degree of matching between the spatial locations of visual features and the semantics of the text, and to apply this to the value terms. Weighted aggregation is performed to extract key fusion incremental information; This is a probabilistic weighting tool in the prior art that maps attention scores to... The probability distribution between them, ensuring that the sum of the weights is 1; query item ; key item ; Value Item ; This is the scaling factor; For learnable relative position offset; This is a cross-dimensional feature mapping and spatial transformation in the prior art, which linearly combines input features through a learnable weight matrix and maps them to a unified high-dimensional attention space.

[0080] Through the aforementioned cross-attention mechanism, it is possible to utilize the injected semantic information. In the original visual flow Retrieve and enhance task-related textures or target regions to generate fused residual incremental features. This incremental feature includes spatial attention distribution guided by textual semantics, providing a basis for subsequent dynamic filtering.

[0081] Then, the fused residual incremental features are processed through a 3x3 convolutional layer. The mapping is performed, and the weights are processed by the Sigmoid activation function to generate a gated weight map. : ; in, It is a gated weight map with pixel-level weights in spatial dimension, used to represent the degree of preservation of fused information at the corresponding spatial location; This is used to map the values ​​of the convolution operation of the 3*3 convolutional layer to a range of 0 to 1, serving as a dynamic filtering factor.

[0082] residual increment characteristics Although it contains potential fusion information, it may introduce unnecessary noise or conflicting features in complex scenarios. Therefore, the above process introduces the existing technology of Dynamic Residual Gating (DRG), which can achieve fine control of the feature information flow and eliminate redundant interference.

[0083] Furthermore, according to the gating weight graph Incremental characteristics of fused residuals Spatial weighting is performed, and the shared visual features are updated using a residual connection mechanism to obtain the updated shared visual features. : ; in, Element-wise multiplication operation representing spatial dimensions.

[0084] Shared visual features After semantically guided attention correction and gating selection, the features are updated layer by layer in a residual manner; after multiple levels of progressive updates, the features at the final level are the final fused features. The final fusion features In the input image decoder, the infrared image sample is reconstructed. and visible light image samples Fusion images with consistent resolution .

[0085] Specifically, the final fusion features The input is fed into an image decoder consisting of cascaded upsampling and convolutional layers. This decoder gradually recovers the spatial resolution of the features through transposed convolutions and compresses the number of channels using convolution operations, ultimately reconstructing a fused image with the same resolution as the input source image. : ; Here, Decoder represents the decoding operation.

[0086] For example, see Figure 6 , Figure 6 This is a schematic diagram of the process for generating a fused image provided by the present invention, including: constructing guided attention fusion based on conditional visual features and shared visual features, performing global alignment between semantic text and visual space; applying dynamic residual gating, performing spatial pixel-level importance weight filtering, realizing visual feature residual update, and obtaining the final fused features; and then, reconstructing the fused image using an image decoder.

[0087] In this embodiment, This refers to the final single-channel or multi-channel fused image. Depending on the color space requirements of the application scenario, if the fusion is performed in grayscale space or using the luminance channel of YCbCr, then... For single-channel images; if fusion is performed directly using the RGB color space, then... This results in multi-channel images. This compatibility design ensures that this embodiment can be flexibly adapted to a variety of practical application environments, from basic thermal imaging monitoring to high-performance full-color visual enhancement.

[0088] Optionally, determining the spatial weight map and the complementary infrared weight map based on the gradient magnitude of the visible light image samples includes: Calculate the gradient magnitude of visible light image samples; The gradient magnitude is normalized to obtain the spatial weight map. The normalization process is as follows: ; in, Spatial weighted graph; The gradient magnitude of the visible light image sample; It is the maximum value operator; It is a constant; Subtracting 1 from the spatial weight map yields the infrared weight map.

[0089] Specifically, calculate visible light image samples The gradient magnitude is calculated and normalized to obtain the spatial weight map. This results in a visible light confidence map. Gradient magnitudes can be calculated using well-known and conventional methods in various image processing fields, such as calculating the gradient using the Sobel operator, Prewitt operator, etc., and then taking the modulus. This process constructs a visible light confidence map, which aims to dynamically allocate the contribution of different modalities in the fusion process based on the texture richness of the visible light image. Regions with higher values ​​represent high confidence and clear visible light textures, while regions with lower values ​​represent low confidence, dark areas, or smooth areas.

[0090] It should be noted that the gradient magnitude is used to extract edge and texture information. Use very small positive numbers, such as 10. -6 This is used to prevent the denominator from being zero and to ensure numerical stability.

[0091] Furthermore, the spatial weight map is calculated according to the normalization formula. and in accordance with Calculate the infrared weighted map .

[0092] In this embodiment of the invention, a weighting mechanism is introduced to ensure that, when calculating the loss function, the model can be adaptively guided to retain the fine texture of visible light or the thermal radiation information of infrared targets based on the scene content.

[0093] Optionally, the process for determining the adaptive gradient loss is as follows: ; in, Adaptive gradient loss; for Norm; For gradient operators; Infrared weighted map; Element-wise multiplication representing spatial dimensions; To merge images; Infrared image samples; Spatial weighted graph; These are visible light image samples.

[0094] This formula uses a spatial weighting map and weights the gradient error. In regions with high confidence in the visible light, it forces the fused image to approximate the gradient of the visible light image; in regions with low confidence, it approximates the gradient of the infrared image, thus effectively avoiding the introduction of noise from the dark areas of the visible light.

[0095] Optionally, the process for determining the weighted color loss is as follows: ; in, For weighted color loss.

[0096] This formula only constrains the fused image to maintain the color and intensity of visible light in areas with high confidence in visible light, i.e., areas with sufficient light and clear texture, preventing pixel values ​​of dark backgrounds in infrared images from contaminating visible light targets and ensuring the naturalness of image tones.

[0097] Optionally, semantic contrastive loss. The process of determining is as follows: ; in, For semantic contrast loss; Characterizes image encoding operations; This is the embedding vector.

[0098] Specifically, a CLIP image encoder that is pre-trained and whose weight parameters are frozen can be used. Extracting fused images High-level semantic feature vector ,Right now .

[0099] Then, based on the high-level semantic feature vector The semantic contrast loss is calculated using the following formula: ; in, It is the dot product of these two vectors. The semantic contrast loss, which represents the magnitude of the vector, ensures that the fusion result, while preserving the physical features of the source image, highlights visual representations that conform to high-level semantics such as "hot target" and "pedestrian" by maximizing the consistency between the fused image and the text description in the feature space.

[0100] In this embodiment of the invention, semantic contrast loss aims to guide the fused image to conform to a preset textual semantic description in visual representation through cross-modal semantic alignment.

[0101] Optionally, the process for determining the base pixel loss is as follows: ; in, Basic pixel loss; For pixel-wise averaging; express Norms are designed to provide a basic pixel-level reconstruction constraint.

[0102] In this embodiment of the invention, the basic pixel loss can constrain the fused image to maintain the balance of the source image in terms of global brightness.

[0103] Optionally, the initial fusion network is trained end-to-end based on basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss until a preset convergence condition is met, resulting in the target fusion network, including: Based on adaptive gradient loss and weighted color loss, determine the spatial adaptive constraint loss; The spatial adaptive constraint loss, basic pixel loss, and semantic contrast loss are weighted and summed to obtain the total loss. Based on the total loss, an optimization algorithm is used to train the initial fusion network end-to-end until a preset convergence condition is met, thus obtaining the target fusion network. The preset convergence condition is that the total loss reaches a convergent state.

[0104] In practical applications, spatial-adaptive constraint loss can be constructed. The aim is to guide the model to adaptively learn the texture and intensity information of the source image in different regions by using the confidence weights (spatial weight map) generated by S41.

[0105] Specifically, the adaptive gradient loss and weighted color loss can be substituted into the following formula for calculation to obtain the spatial adaptive constraint loss. : ; in, For use in balancing and The hyperparameter α is 1.0 in actual experiments, and the range of α is [0.1, 2.0].

[0106] After obtaining the spatial adaptive constraint loss Based on this, the spatial adaptive constraint loss, basic pixel loss, and semantic contrast loss are substituted into the following formula for calculation to obtain the total loss. : ; in, , as well as These are preset hyperparameter weights used to adjust the impact of different constraint terms on model convergence.

[0107] For example, in order to balance basic pixel alignment, spatial adaptive modal conflict suppression, and high-level semantic consistency, the hyperparameter weights are specifically set as follows: λ1=1.0, λ2=1.0, λ3=0.1.

[0108] , as well as The optimization method can be: grid search with controlled variables, that is, under fixed... as well as If it is 0, first check Adjust to the optimal level, then fix. Similarly, the other two hyperparameters are adjusted to find the relatively optimal hyperparameter design.

[0109] Furthermore, using an optimization algorithm, the initial fusion network is trained end-to-end until the total loss is reached. Once convergence is achieved, the target fusion network is obtained.

[0110] For example, see Figure 7 , Figure 7 This is a flowchart illustrating the spatial adaptive optimization strategy provided by the present invention, including: after obtaining the fused image, generating confidence weights based on visible light image samples, i.e., determining the spatial weight map and infrared weight map; further, calculating multi-criteria loss based on the spatial weight map and infrared weight map to obtain adaptive gradient loss and weighted color loss; calculating semantic contrast loss and basic pixel loss based on the fused image; weighting the adaptive gradient loss and weighted color loss based on the hyperparameter α to obtain the spatial adaptive constraint loss; then determining the total loss based on the spatial adaptive constraint loss, semantic contrast loss, and basic pixel loss, and then updating the model parameters of the initial fusion network based on the total loss.

[0111] The overall process differs from existing technologies that rely solely on low-level pixel statistics (such as average and maximum values) for fusion. It introduces CLIP text embedding and text conditional gating mechanisms. Through natural language cues such as "heat source target" and "background texture," the target fusion network model can explicitly understand image content. In the target images obtained using the target fusion network, the outlines of key heat source targets such as pedestrians and vehicles are clearer, and the contrast with the background is higher, significantly improving the accuracy of downstream detection tasks.

[0112] Furthermore, based on this, it can overcome modal conflicts and noise pollution under low illumination. Specifically, to address the problem that visible light images contain a lot of noise and invalid black backgrounds in nighttime or low-light scenes, a spatial adaptive optimization strategy is used. A confidence map is constructed using the visible light gradient, and a dynamic trade-off is made between "preserving visible light texture" and "preserving infrared thermal radiation". This allows the fused image to retain sufficient color details in well-lit areas, while automatically switching to infrared features in dark areas. This can eliminate ghosting, noise introduction, and target inundation problems common in existing technologies, thereby achieving high-quality imaging in all weather conditions.

[0113] Furthermore, unlike existing fusion networks that typically employ simple concatenation or addition operations and have weak feature interaction capabilities, the text conditional gating proposed in each embodiment utilizes cross-modal cross-attention and dynamic gating networks to achieve pixel-level feature selection. The network can adaptively learn how much infrared and visible light information should be retained at each location, exhibiting stronger scene adaptability and detail fidelity compared to fixed-rule fusion methods.

[0114] The infrared and visible light image fusion system for low-light scenes provided by the present invention will be described below. The infrared and visible light image fusion system for low-light scenes described below can be referred to in correspondence with the infrared and visible light image fusion method for low-light scenes described above.

[0115] Figure 8 This is a schematic diagram of the infrared and visible light image fusion system for low-light scenes provided by the present invention, as shown below. Figure 8 As shown, the infrared and visible light image fusion system for low-light scenes includes the following: The acquisition unit 801 is configured to acquire the infrared image and the visible light image to be fused, as well as the target text prompts with explicit semantic guidance corresponding to the infrared image and the visible light image; The fusion unit 802 is configured to, based on the target text prompt, invoke a target fusion network to fuse the infrared image and the visible light image to obtain a target image; Training unit 803 is configured as follows: Obtain sample images and sample text prompts for the sample images in low-light scenes, and encode the sample text prompts to obtain an embedding vector; Based on a dual-stream encoder, multi-scale feature extraction is performed on infrared and visible light image samples in the sample images. Then, the samples are stitched together in the channel dimension and cross-modal information is interacted and dimensionally compressed to obtain shared visual features. The embedded vector is processed by the text conditional modulation module to obtain semantic modulation weights; based on the semantic modulation weights, the shared visual features are subjected to channel-dimensional text conditional modulation to generate semantically enhanced conditional visual features. Based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and the shared visual features. After multiple iterations, the final fusion feature is generated, and the fusion image is reconstructed based on the final fusion feature by the image decoder. Based on the gradient magnitude of the visible light image samples, a spatial weight map and a complementary infrared weight map are determined; based on the fused image, the spatial weight map, and the infrared weight map, basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss are determined. Based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss, the initial fusion network is trained end-to-end until the preset convergence condition is met, thus obtaining the target fusion network.

[0116] The infrared and visible light image fusion system for low-light scenes provided by this invention introduces sample text prompts and their embedded vectors. Unlike existing technologies that rely on statistical methods such as the average and maximum values ​​of underlying pixels for fusion, this system uses natural language prompts to enable the fusion network to explicitly understand the image content. Based on this, and using the fused image, spatial weight map, and infrared weight map, it automatically makes a dynamic trade-off between "preserving visible light texture" and "preserving infrared thermal radiation," effectively solving modal conflicts and noise pollution in low-light conditions. It addresses the problem of visible light images containing a lot of noise and invalid black backgrounds in nighttime or low-light scenes, and uses cross-modal cross-attention and dynamic spatial feature filtering to achieve pixel-level feature filtering. Compared with fixed-rule fusion methods, it has stronger scene adaptability and detail fidelity.

[0117] Optionally, the training unit 803 is specifically configured as follows: Acquire infrared and visible light image data; Infrared image data and visible light image data are spatially registered and size normalized respectively to obtain infrared image samples and visible light image samples. Based on the preset fusion task scenario, natural language keywords that can characterize the salient features of the source image are retrieved from the preset lexicon and embedded into the predefined instruction template to construct sample text prompts with explicit semantic guidance for infrared image samples and visible light image samples. The sample text prompts are encoded and mapped to the semantic latent space using a contrastive language image pre-trained text encoder, generating high-dimensional embedding vectors.

[0118] Optionally, the dual-stream encoder includes an infrared processing branch and a visible light processing branch. The infrared processing branch and the visible light processing branch include multiple cascaded Swing Transformer modules. The shared visual features are extracted by the l-th feature extraction layer in the dual-stream encoder. The l-th feature extraction layer includes the l-th Swing Transformer module in the infrared processing branch and the l-th Swing Transformer module in the visible light processing branch. The training unit 803 is specifically configured as follows: Infrared image samples are input into the infrared processing branch, and visible light image samples are input into the visible light processing branch for parallel processing. The shift window self-attention mechanism is used to capture the multi-scale features of the infrared image samples and the visible light image samples, so as to obtain the infrared modal features of the l-th feature extraction layer and the visible light modal features of the l-th feature extraction layer. Infrared modal features and visible light modal features are concatenated along the channel dimension, and a 1*1 convolutional layer is used to achieve cross-modal information interaction and dimensionality compression, generating shared visual features for the l-th feature extraction layer.

[0119] Optionally, the process of acquiring the infrared modal features of the l-th feature extraction layer is as follows: ; in, The infrared modal features of the l-th feature extraction layer; This is the feature extraction mapping function for the l-th Swin Transformer module in the infrared processing branch; The process of obtaining the visible light modal features of the l-th feature extraction layer is as follows: ; in, The visible light modal features are those of the l-th feature extraction layer; This is the feature extraction mapping function of the l-th Swin Transformer module in the visible light processing branch; The process of generating the shared visual features of the l-th feature extraction layer is as follows: ; in, The shared visual features of the l-th feature extraction layer; This is a splicing operation at the channel level; This indicates a convolution operation with a kernel size of 1*1.

[0120] Optionally, the training unit 803 is specifically configured as follows: The embedded vectors are projected onto the channel dimension space of shared visual features using the lightweight projection head in the text conditional modulation module, and then processed by the sigmoid activation function in the text conditional modulation module to obtain semantically oriented semantic modulation weights. Conditional visual features are generated by performing element-wise multiplicative modulation of the channel dimension of shared visual features in the form of residuals based on semantic modulation weights.

[0121] Optionally, the semantic modulation weights are generated as follows: ; Where W is the semantic modulation weight; This is a linear projection mapping operation; The Sigmoid activation function is used to normalize the semantic modulation weights to a range of 0 to 1. The process of generating conditional visual features is as follows: ; in, Conditional visual features; This is an element-wise multiplication operation for the channel dimension.

[0122] Optionally, the training unit 803 is specifically configured as follows: Using conditional visual features as query terms and shared visual features as key and value terms, a window partitioning operation is performed on the conditional visual features and shared visual features to decompose the feature maps of both conditionsal visual features and shared visual features into non-overlapping local regions. Each local region is mapped to a high-dimensional feature space through a linear projection layer to obtain the projected features. The projected features are subjected to window cross attention calculation to obtain fused residual increment features, which contain potential fusion information and spatial attention distribution guided by text semantics. The fused residual incremental features are mapped through a 3*3 convolutional layer and processed by a Sigmoid activation function to obtain a gated weight map. Spatial weighting is applied to the fused residual incremental features based on the gating weight graph, and the shared visual features are updated through the residual connection mechanism. Shared visual features are updated layer by layer in a residual manner after semantically guided attention correction and gating screening to obtain the final fused features; The final fused features are input into the image decoder to reconstruct a fused image with the same resolution as the infrared image sample and the visible light image sample.

[0123] Optionally, the training unit 803 is specifically configured as follows: Calculate the gradient magnitude of visible light image samples; The gradient magnitude is normalized to obtain the spatial weight map. The normalization process is as follows: ; in, Spatial weighted graph; The gradient magnitude of the visible light image sample; It is the maximum value operator; It is a constant; Subtracting 1 from the spatial weight map yields the infrared weight map.

[0124] Optionally, the process for determining the adaptive gradient loss is as follows: ; in, Adaptive gradient loss; for Norm; For gradient operators; Infrared weighted map; Element-wise multiplication representing spatial dimensions; To merge images; Infrared image samples; Spatial weighted graph; Visible light image samples; The process for determining the weighted color loss is as follows: ; in, Weighted color loss; The process of determining semantic contrast loss is as follows: ; in, For semantic contrast loss; Characterizes image encoding operations; For embedding vectors; The process for determining the base pixel loss is as follows: ; in, Basic pixel loss; For pixel-wise averaging operator, express Norm; The training unit 803 is specifically configured as follows: Based on adaptive gradient loss and weighted color loss, determine the spatial adaptive constraint loss; The spatial adaptive constraint loss, basic pixel loss, and semantic contrast loss are weighted and summed to obtain the total loss. Based on the total loss, an optimization algorithm is used to train the initial fusion network end-to-end until a preset convergence condition is met, thus obtaining the target fusion network. The preset convergence condition is that the total loss reaches a convergent state.

[0125] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an infrared and visible light image fusion method for low-light scenes. This method includes: acquiring the infrared and visible light images to be fused, and the target text prompts with explicit semantic guidance corresponding to the infrared and visible light images; based on the target text prompts, calling a target fusion network to fuse the infrared and visible light images to obtain a target image; the training process of the target fusion network includes: acquiring sample images and sample text prompts for the sample images in low-light scenes, and encoding the sample text prompts to obtain embedding vectors; performing multi-scale feature extraction on the infrared and visible light image samples in the sample images based on a dual-stream encoder, and performing concatenation in the channel dimension as well as cross-modal information interaction and dimensional compression to obtain shared visual features; and performing text conditional modulation on the embedding vectors. The system performs line processing to obtain semantic modulation weights. Based on these weights, it performs channel-dimensional text conditional modulation on shared visual features to generate semantically enhanced conditional visual features. Based on the embedding vectors, it performs cross-modal cross-attention fusion and dynamic spatial feature filtering on the conditional visual features and shared visual features. After multiple iterations, it generates the final fused features, which are then reconstructed using an image decoder to generate a fused image. The system determines the spatial weight map and the complementary infrared weight map based on the gradient magnitude of the visible light image samples. Based on the fused image, the spatial weight map, and the infrared weight map, it determines the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss. Based on these losses, the initial fusion network is trained end-to-end until the preset convergence condition is met, resulting in the target fusion network.

[0126] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the infrared and visible light image fusion method for low-light scenes provided by the above methods. The method includes: acquiring infrared and visible light images to be fused, and target text prompts with explicit semantic guidance corresponding to the infrared and visible light images; based on the target text prompts, calling a target fusion network to fuse the infrared and visible light images to obtain a target image; the training process of the target fusion network includes: acquiring sample images and sample text prompts of the sample images in low-light scenes, and encoding the sample text prompts to obtain embedding vectors; performing multi-scale feature extraction on the infrared and visible light image samples in the sample images based on a dual-stream encoder, and performing concatenation and cross-modal information processing in the channel dimension. Interaction and dimensionality compression yield shared visual features. The embedded vectors are processed by a text conditional modulation module to obtain semantic modulation weights. Based on these weights, the shared visual features are subjected to channel-dimensional text conditional modulation, generating semantically enhanced conditional visual features. Based on the embedded vectors, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and shared visual features. After multiple iterations, the final fused features are generated, and the fused image is reconstructed using an image decoder. The spatial weight map and complementary infrared weight map are determined based on the gradient magnitude of visible light image samples. Based on the fused image, spatial weight map, and infrared weight map, the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss are determined. The initial fusion network is then trained end-to-end based on these losses until a preset convergence condition is met, resulting in the target fusion network.

[0128] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the infrared and visible light image fusion method for low-light scenes provided by the methods described above. This method includes: acquiring infrared and visible light images to be fused, and target text prompts with explicit semantic guidance corresponding to the infrared and visible light images; based on the target text prompts, calling a target fusion network to fuse the infrared and visible light images to obtain a target image; the training process of the target fusion network includes: acquiring sample images and sample text prompts for the sample images in low-light scenes, and encoding the sample text prompts to obtain embedding vectors; performing multi-scale feature extraction on infrared and visible light image samples in the sample images based on a dual-stream encoder, and performing concatenation in the channel dimension as well as cross-modal information interaction and dimensional compression to obtain shared visual features. The process involves: processing the embedded vectors using a text conditional modulation module to obtain semantic modulation weights; applying these weights to channel-dimensional text conditional modulation of shared visual features to generate semantically enhanced conditional visual features; performing cross-modal cross-attention fusion and dynamic spatial feature filtering on the conditional visual features and shared visual features based on the embedded vectors; generating the final fused features after multiple iterations; and reconstructing the fused image using an image decoder based on the final fused features; determining the spatial weight map and complementary infrared weight map based on the gradient magnitude of visible light image samples; and determining the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss based on the fused image, spatial weight map, and infrared weight map. The initial fusion network is then trained end-to-end based on these losses until a preset convergence condition is met, resulting in the target fusion network.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusing infrared and visible light images in low-light scenes, characterized in that, include: Obtain the infrared image and the visible light image to be fused, as well as the target text prompts with explicit semantic guidance corresponding to the infrared image and the visible light image; Based on the target text prompt, the target fusion network is invoked to fuse the infrared image and the visible light image to obtain the target image; The training process of the target fusion network includes: Obtain sample images and sample text prompts for the sample images in low-light scenes, and encode the sample text prompts to obtain an embedding vector; Based on a dual-stream encoder, multi-scale feature extraction is performed on infrared and visible light image samples in the sample images. Then, the samples are stitched together in the channel dimension and cross-modal information is interacted and dimensionally compressed to obtain shared visual features. The embedded vector is processed by the text conditional modulation module to obtain semantic modulation weights. Based on the semantic modulation weights, the shared visual features are subjected to channel-dimensional text conditional modulation to generate semantically enhanced conditional visual features. Based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and the shared visual features. After multiple iterations, the final fusion feature is generated, and the fusion image is reconstructed based on the final fusion feature by the image decoder. The spatial weight map and the complementary infrared weight map are determined based on the gradient magnitude of the visible light image samples. Based on the fused image, the spatial weight map, and the infrared weight map, the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss are determined. Based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss, the initial fusion network is trained end-to-end until a preset convergence condition is met, thus obtaining the target fusion network.

2. The infrared and visible light image fusion method for low-light scenes according to claim 1, characterized in that, The process of acquiring sample images and sample text prompts for low-light scenes, and encoding the sample text prompts to obtain an embedding vector, includes: Acquire infrared and visible light image data; Spatial registration and size normalization are performed on the infrared image data and the visible light image data respectively to obtain the infrared image sample and the visible light image sample; Based on the preset fusion task scenario, natural language keywords that can characterize the salient features of the source image are retrieved from the preset lexicon and embedded into the predefined instruction template to construct sample text prompts with explicit semantic guidance for the infrared image sample and the visible light image sample. The sample text prompts are encoded and mapped to a semantic latent space using a contrastive language image pre-trained text encoder to generate the high-dimensional embedding vector.

3. The infrared and visible light image fusion method for low-light scenes according to claim 1, characterized in that, The dual-stream encoder includes an infrared processing branch and a visible light processing branch. The infrared processing branch and the visible light processing branch include multiple cascaded Swing Transformer modules. The shared visual feature is extracted by the l-th feature extraction layer in the dual-stream encoder. The l-th feature extraction layer includes the l-th Swing Transformer module in the infrared processing branch and the l-th Swing Transformer module in the visible light processing branch. The method involves multi-scale feature extraction of infrared and visible light image samples from the sample images based on a dual-stream encoder, followed by channel-level stitching, cross-modal information interaction, and dimensionality compression to obtain shared visual features, including: The infrared image sample is input into the infrared processing branch, and the visible light image sample is input into the visible light processing branch for parallel processing. The shift window self-attention mechanism is used to capture the multi-scale features of the infrared image sample and the visible light image sample, so as to obtain the infrared modal features of the l-th feature extraction layer and the visible light modal features of the l-th feature extraction layer. The infrared modal features and the visible light modal features are concatenated along the channel dimension, and a 1*1 convolutional layer is used to achieve cross-modal information interaction and dimensional compression, generating the shared visual features of the l-th feature extraction layer.

4. The infrared and visible light image fusion method for low-light scenes according to claim 3, characterized in that, The process of acquiring the infrared modal features of the l-th feature extraction layer is as follows: ； in, The infrared modal features of the l-th feature extraction layer; This is the feature extraction mapping function of the l-th Swin Transformer module in the infrared processing branch; The process of obtaining the visible light modal features of the l-th feature extraction layer is as follows: ； in, The visible light modal features of the l-th feature extraction layer; This is the feature extraction mapping function of the l-th Swing Transformer module in the visible light processing branch; The process of generating the shared visual features in the l-th feature extraction layer is as follows: ； in, The shared visual features of the l-th feature extraction layer; This is a splicing operation at the channel level; This indicates a convolution operation with a kernel size of 1*1.

5. The infrared and visible light image fusion method for low-light scenes according to claim 1, characterized in that, The process of processing the embedded vector through a text conditional modulation module to obtain semantic modulation weights, and then performing channel-dimensional text conditional modulation on the shared visual features based on these semantic modulation weights to generate semantically enhanced conditional visual features, includes: The embedded vector is projected onto the channel dimension space of the shared visual features using the lightweight projection head in the text conditional modulation module, and then processed by the Sigmoid activation function in the text conditional modulation module to obtain the semantically guided semantic modulation weights. The shared visual features are generated by performing element-wise multiplicative modulation of the channel dimension in residual form based on the semantic modulation weights.

6. The infrared and visible light image fusion method for low-light scenes according to claim 5, characterized in that, The process of generating the semantic modulation weights is as follows: ； Wherein, W is the semantic modulation weight; This is a linear projection mapping operation; The Sigmoid activation function is used to normalize the semantic modulation weights to a range of 0 to 1. The process of generating the conditional visual features is as follows: ； in, For the conditional visual features; This is an element-wise multiplication operation for the channel dimension.

7. The infrared and visible light image fusion method for low-light scenes according to claim 1, characterized in that, The process involves performing cross-modal cross-attention fusion and dynamic spatial feature filtering on the conditional visual features and the shared visual features based on the embedding vector. After multiple iterations, a final fused feature is generated, and a fused image is reconstructed based on the final fused feature using an image decoder. This includes: Using the conditional visual features as query terms and the shared visual features as key and value terms, a window partitioning operation is performed on the conditional visual features and the shared visual features to decompose the feature maps of the conditional visual features and the shared visual features into non-overlapping local regions. Each of the local regions is mapped to a high-dimensional feature space through a linear projection layer to obtain the projected features. The projected features are subjected to window cross attention calculation to obtain fused residual increment features, which contain potential fusion information and spatial attention distribution guided by text semantics. The fused residual incremental features are mapped using a 3*3 convolutional layer and processed by a Sigmoid activation function to obtain a gated weight map. The fused residual incremental features are spatially weighted according to the gating weight map, and the shared visual features are updated through the residual connection mechanism. The shared visual features are updated layer by layer in a residual manner after semantically guided attention correction and gating screening to obtain the final fused features; The final fusion features are input into the image decoder to reconstruct the fused image with the same resolution as the infrared image sample and the visible light image sample.

8. The infrared and visible light image fusion method for low-light scenes according to claim 1, characterized in that, The determination of the spatial weight map and the complementary infrared weight map based on the gradient magnitude of the visible light image samples includes: Calculate the gradient magnitude of the visible light image sample; The gradient magnitude is normalized to obtain the spatial weight map. The normalization process is as follows: ； in, This is the spatial weighting graph; The gradient magnitude of the visible light image sample; It is the maximum value operator; It is a constant; Subtracting 1 from the spatial weight map yields the infrared weight map.

9. The infrared and visible light image fusion method for low-light scenes according to claim 1, characterized in that, The process for determining the adaptive gradient loss is as follows: ； in, The adaptive gradient loss; for Norm; For gradient operators; The infrared weighted map; Element-wise multiplication representing spatial dimensions; The fused image; The infrared image sample; This is the spatial weighting graph; The visible light image sample; The process for determining the weighted color loss is as follows: ； in, The weighted color loss; The process for determining the semantic contrast loss is as follows: ； in, The semantic contrast loss; Characterizes image encoding operations; The embedding vector; The process for determining the basic pixel loss is as follows: ； in, The loss is the basic pixel loss; For pixel-wise averaging operator, express Norm; The initial fusion network is trained end-to-end based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss until a preset convergence condition is met, resulting in the target fusion network, including: Based on the adaptive gradient loss and the weighted color loss, determine the spatial adaptive constraint loss; The spatial adaptive constraint loss, the basic pixel loss, and the semantic contrast loss are weighted and summed to obtain the total loss. Based on the total loss, an optimization algorithm is used to train the initial fusion network end-to-end until the preset convergence condition is met, thereby obtaining the target fusion network. The preset convergence condition is that the total loss reaches a convergent state.

10. An infrared and visible light image fusion system for low-light scenes, characterized in that, include: The acquisition unit is configured to acquire the infrared image and the visible light image to be fused, as well as the target text prompts with explicit semantic guidance corresponding to the infrared image and the visible light image; The fusion unit is configured to, based on the target text prompt, invoke a target fusion network to fuse the infrared image and the visible light image to obtain a target image; The training unit is configured as follows: Obtain sample images and sample text prompts for the sample images in low-light scenes, and encode the sample text prompts to obtain an embedding vector; Based on a dual-stream encoder, multi-scale feature extraction is performed on infrared and visible light image samples in the sample images. Then, the samples are stitched together in the channel dimension and cross-modal information is interacted and dimensionally compressed to obtain shared visual features. The embedded vector is processed by the text conditional modulation module to obtain semantic modulation weights; based on the semantic modulation weights, the shared visual features are subjected to channel-dimensional text conditional modulation to generate semantically enhanced conditional visual features. Based on the embedding vector, cross-modal cross-attention fusion and dynamic spatial feature filtering are performed on the conditional visual features and the shared visual features. After multiple iterations, the final fusion feature is generated, and the fusion image is reconstructed based on the final fusion feature by the image decoder. Based on the gradient magnitude of the visible light image samples, a spatial weight map and a complementary infrared weight map are determined; based on the fused image, the spatial weight map, and the infrared weight map, basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss are determined. Based on the basic pixel loss, adaptive gradient loss, weighted color loss, and semantic contrast loss, the initial fusion network is trained end-to-end until the preset convergence condition is met, thus obtaining the target fusion network.

Images