Infrared and visible image fusion method and system based on multi-granularity word elements

Abstract

The application provides an infrared and visible light image fusion method and system based on multi-granularity word elements, and comprises the following steps: S1, acquiring infrared images and visible light images, and decomposing the infrared images and the visible light images on no less than two different scales; multi-granularity word element global feature extraction: calculating long-range dependence of the infrared images and the visible light images through no less than two independent Transformer branches; designing a loss function by using a preset logic, and using the loss function to supervise and train a preset multi-granularity word element fusion model; and fusing the infrared images and the visible light images through a multi-granularity word element fusion module to obtain a multi-granularity word element fusion output image. The application solves the technical problems of poor fusion model performance, high complexity, limited feature extraction and representation.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and more specifically to various types of image fusion technology. Background Technology

[0002] To comprehensively depict real-world scenes, combining multi-source images acquired by different sensors is crucial for applications. Therefore, infrared and visible light image fusion has been widely used in intelligence gathering and analysis, such as in military, public security, and smart cities. Infrared sensors aim to capture thermal radiation emitted by heat sources, enhancing the image of thermal infrared targets. However, infrared sensors cannot detect details or texture information in the background because objects in the background have almost identical thermal information. Visible light sensors, on the other hand, capture reflected light to generate images, preserving rich background texture or detail information. Therefore, image fusion can leverage the advantages of multiple source images to simultaneously synthesize a single information image, and many fusion methods have been proposed, including traditional methods and deep learning-based methods. Traditional methods mainly extract features through mathematical transformations and then combine these features by designing fusion strategies. Traditional methods include multi-scale transformation methods, sparse representation-based methods, saliency-based methods, subspace-based methods, and other hybrid methods. Traditional fusion methods aim to obtain satisfactory fused images and meet certain application requirements to a certain extent. However, traditional fusion methods still have bottlenecks.

[0003] For example, the existing invention patent application document CN110332934A, entitled "A Robot Tracking System and Method under Hybrid Network Fusion," describes a visual image sensor, a wireless transmitter, a wireless receiver, a visible light source, an optical communication device, an inertial navigation device, and a hybrid data fusion unit. The visual image sensor, wireless transmitter, and visible light source are installed in the robot's moving area; the wireless receiver, optical communication device, and inertial navigation device are installed on the robot's body; and the track tracking data is calculated and evaluated in the hybrid data processing unit at the control center. This prior art fuses multi-source hybrid data acquired by multiple sensors. From the specific implementation of this prior art, it can be seen that the visible light image is only used as a coarse unit, and then multiple corrections are performed using data collected from other multi-source sensors. The single representation ignores the differences between multi-source images, limiting the performance of the fusion model. Furthermore, the fusion method of this prior art mainly involves combining various types of data processing processes, increasing the algorithm complexity. In addition, traditional fusion methods also require the design of complex fusion strategies. Therefore, deep learning (DL) has been introduced to solve these tasks.

[0004] Deep learning-based methods possess nonlinear fitting capabilities, enabling them to model complex correlations in source images. Based on differences in fusion frameworks, DL-based fusion methods can be categorized into CNN-based fusion methods and Generative Adversarial Network (GAN)-based fusion methods. CNN-based methods utilize parallel convolutional kernels to extract multiple features and reconstruct the fusion result using a refined loss function. Furthermore, the GAN framework can also be used for image fusion tasks, employing an adversarial game to simulate the distribution of source images. Typically, Ma et al. first used GANs to fuse infrared and visible light images, and then proposed various GAN-based methods, such as the dual-discriminator GAN method and the multi-classification constrained GAN method.

[0005] For example, the existing invention patent application CN114240736A, entitled "A Method for Simultaneous Generation and Editing of Arbitrary Facial Attributes Based on VAE and cGAN," focuses on an encoder-decoder architecture based on Variational Autoencoder (VAE) and Conditional Adversarial Neural Network (cGAN), and develops a bidirectional feedback generative network for simultaneously generating new faces and performing attribute editing. Attribute classification constraints are used on the generated images to ensure the correct variation of specified attributes, and face images with multiple attributes are generated by sampling attribute encodings from the latent space. This method includes modeling attribute strength to support attribute interpolation and flexibly handle multiple facial attributes. As mentioned above, such CNN or GAN-based methods utilize convolutional operations to extract image features from a small receptive field, and the uniform convolutional operation limits feature extraction and representation. Furthermore, while CNN or GAN-based methods aim to extract local features through convolutional kernels, these existing schemes also have drawbacks, namely, the inability to extract long-range dependency information. Therefore, Transformer can be introduced to address these issues. For example, Li et al. and Vibashan et al. combined Transformer and CNN to extract local features and long-range dependency information of images.

[0006] However, existing Transformer-based methods ignore the difference in attention weights between infrared and visible light words at the same position, which affects fusion performance because infrared and visible light words have different importance at the same position.

[0007] In summary, existing technologies suffer from technical problems such as poor performance of fusion models, high complexity, and limitations in feature extraction and representation. Summary of the Invention

[0008] The technical problem to be solved by this invention is how to solve the technical problems of poor performance, high complexity, and limited feature extraction and representation of existing fusion models.

[0009] This invention solves the above-mentioned technical problems by employing the following technical solution: A method for fusion of infrared and visible light images based on multi-granularity terms includes:

[0010] S1. Acquire infrared and visible light images, and decompose the infrared and visible light images at no less than two different scales respectively;

[0011] S2. Multi-granularity term global feature extraction: Using at least two independent Transformer branches, calculate the long-range dependencies of infrared and visible light images respectively. For each independent Transformer branch, design at least two Transformer models to extract comprehensive multi-scale long-range dependencies. Step S2 includes:

[0012] S21. Divide the infrared and visible light images into multi-scale regional patches;

[0013] S22. Convert infrared and visible light images into infrared sequences. and visible light sequence

[0014] S23. Embedding and processing infrared sequences using a preset linear projection E. and visible light sequence Encoded position information is added to each sequence to obtain the encoded infrared sequence. and encoding visible light sequences

[0015] S24. Utilize a fully connected layer to encode the infrared sequence using pre-embedded logic. and encoding visible light sequences Perform an embedding operation to obtain relation extraction parameters;

[0016] S25. Utilizing the Multi-Head Self-Attention (MSA) mechanism, parameters are extracted using pre-defined logical processing relationships to obtain long-range dependencies from infrared and visible light images. Based on this, multi-head self-attention fusion parameters are obtained, including: infrared lexical units. and visible light term

[0017] S3. Design a loss function using pre-defined logic to supervise and train a pre-defined multi-granularity word fusion model.

[0018] S4. The infrared image and the visible light image are fused through a multi-granularity word fusion module to obtain a multi-granularity word fusion output image. Step S4 includes:

[0019] S41. Learnable attention weights are obtained by defining pre-defined weights, and then infrared lexical units are captured using pre-defined relation capture logic. With visible light term Multi-granularity terminology relevance;

[0020] S42. Use preset reconstruction logic to process multi-granularity word correlation and differential scale features to obtain multi-granularity word fusion output image.

[0021] This invention captures the relevance of corresponding lexical units by introducing learnable attention weights, thereby fusing infrared and visible light images at a multi-granularity lexical dimension. This method can perceive the differences in multimodal information present at the same location for infrared and visible light lexical units. This invention extracts long-range dependencies of each image at multiple scales, embeds multi-granularity lexical units into sub-images through local multi-scale segmentation, reconstructs multi-scale features using multi-granularity fusion, and calculates the fused image through dimensionality reduction mapping operations. This invention can achieve the fusion of infrared and visible light images at a multi-scale lexical dimension based on a pure Transformer, and has better fusion performance compared to other methods.

[0022] In a more specific technical solution, step S21 includes: using the following logic, dividing the infrared image into N P at each scale. s 2 Multi-scale regional patches of varying sizes:

[0023]

[0024] In the formula, s is the scale applied to the segmented image, and s is defined as 1, 2, and 3 respectively.

[0025] In a more specific technical solution, in step S23, the infrared sequence is embedded using a preset linear projection E according to the following logic. and visible light sequence And add encoded position information to each sequence:

[0026]

[0027] In the formula, and This represents the encoded sequences of two original images at different scales of s.

[0028] In a more specific technical solution, in step S24, a fully connected layer is used to encode the infrared sequence using the following logic. and encoding visible light sequences Perform embedding operations:

[0029]

[0030] In the formula, and This represents the query, key, and value of infrared and visible light image sequences, and LN represents a fully connected layer.

[0031] In a more specific technical solution, in step S25, parameters are extracted using the following logical processing relationship to obtain long-range dependencies from the infrared and visible light images:

[0032]

[0033] In the formula, and This is the output of MSA.

[0034] This invention extends fusion to multi-granularity lexical fusion to extract multi-scale long-range dependencies of each source image and capture the attentional relevance of corresponding lexical units at different scales. Furthermore, lexical-based fusion can also extract local region features because the lexical units embedded in the locally segmented sub-images themselves contain local region features, optimizing model fusion performance and improving the representational quality of the fused image features.

[0035] In a more specific technical solution, step S3 includes:

[0036] S31. Calculate the loss in the intensity domain between the infrared image and the fused image using the following logic:

[0037]

[0038] S32. The loss L1 and total loss are obtained using the following logic processing, thereby preserving the details and brightness information of the visible light image:

[0039]

[0040]

[0041] In the formula, L represents the total loss value, M represents the total number of pixels, and f, I, and V represent the fusion result, the infrared image, and the visible light image, respectively. F Let λ denote the Forbenius norm of the matrix, and λ be the balance parameter.

[0042] This invention extracts the long-range dependency between infrared and visible light images through two independent Transformer branches. In the fusion module, the optimal fusion result is found by considering the intensity loss of the infrared image and the L1 loss of the visible light image, while retaining more details and brightness information of the visible light image. Ultimately, the fusion of infrared and visible light images is achieved, while optimizing the image fusion accuracy and fusion effect.

[0043] In a more specific technical solution, step S41 includes:

[0044] S411. Using the following logic, infrared lexical units are captured with the learnable attention weights. With visible light term Multi-granularity term relevance:

[0045]

[0046] In the formula, f represents the features calculated by fusing lexical units at the s-scale, and R represents the reshaping operation. and Indicating infrared morphemes and visible light term Learnable attention weights;

[0047] S412, Balanced Infrared Terminology and visible light term The importance of being in the same position.

[0048] In a more specific technical solution, step S412 utilizes the following logic to balance infrared terms. and visible light term Importance of being in the same location:

[0049]

[0050] In a more specific technical solution, in step S42, the following logic is used to process multi-granularity word correlation and difference scale features to obtain a multi-granularity word fusion output image:

[0051] f = g(f1,…f s ),

[0052] In the formula, f represents the fused image, g represents the dimensionality reduction mapping operation of the convolutional layer, and the kernel is 1×1.

[0053] In more specific technical solutions, infrared and visible light image fusion systems based on multi-granularity terms include:

[0054] The differential scale decomposition module is used to acquire infrared and visible light images and decompose the infrared and visible light images at no less than two differential scales.

[0055] A multi-granularity lexical global feature extraction module is used to calculate the long-range dependencies of infrared and visible light images through at least two independent Transformer branches. At least two Transformer models are designed in each independent Transformer branch to extract comprehensive multi-scale long-range dependencies. The multi-granularity lexical global feature extraction module is connected to the differential scale decomposition module. The multi-granularity lexical global feature extraction module includes:

[0056] The image segmentation module is used to segment infrared and visible light images into multi-scale regional patches;

[0057] The image conversion module is used to convert infrared and visible light images into infrared sequences. and visible light sequence The image conversion module is connected to the image segmentation module;

[0058] A linear projection embedding module is used to embed infrared sequences using a preset linear projection E. and visible light sequence Encoded position information is added to each sequence to obtain the encoded infrared sequence. and encoding visible light sequences The linear projection embedding module is connected to the image conversion module;

[0059] The relation extraction module is used to leverage the fully connected layer to encode infrared sequences with pre-embedded logic. and encoding visible light sequences An embedding operation is performed to obtain relation extraction parameters; the relation extraction module is connected to the linear projection embedding module.

[0060] The multi-head self-attention fusion module utilizes the Multi-Head Self-Attention (MSA) mechanism to extract parameters based on pre-defined logic processing relationships. This allows for the extraction of long-range dependencies from infrared and visible light images, thereby obtaining MSA fusion parameters. These parameters include infrared lexical units. and visible light term The multi-head self-attention fusion module is connected to the relationship extraction module;

[0061] The word fusion model training module is used to design a loss function using pre-defined logic, thereby supervising and training a pre-defined multi-granularity word fusion model. The word fusion model training module is connected to the multi-head self-attention fusion module.

[0062] A multi-granularity word fusion output module is used to fuse infrared and visible light images to obtain a multi-granularity word fusion output image. This module is connected to the word fusion model training module and includes:

[0063] The lexical relevance module is used to obtain learnable attention weights based on predefined weight definition logic, and then use predefined relationship capture logic to capture infrared lexical units. With visible light term Multi-granularity terminology relevance;

[0064] The image reconstruction module is used to process multi-granularity word correlation and differential scale features using preset reconstruction logic to obtain a multi-granularity word fusion output image. The image reconstruction module is connected to the word correlation module.

[0065] Compared with existing technologies, this invention has the following advantages: It captures the relevance of corresponding lexical units by introducing learnable attention weights, thereby fusing infrared and visible light images at a multi-granularity lexical dimension. This method can perceive the differences in multimodal information present at the same location for infrared and visible light lexical units. This invention extracts the long-range dependencies of each image at multiple scales, embeds multi-granularity lexical units into sub-images through local multi-scale segmentation, reconstructs multi-scale features using multi-granularity fusion, and calculates the fused image through dimensionality reduction mapping operations. This invention can achieve the fusion of infrared and visible light images at multiple scales based on a pure Transformer, and has better fusion performance than other schemes.

[0066] This invention extends fusion to multi-granularity lexical fusion to extract multi-scale long-range dependencies of each source image and capture the attentional relevance of corresponding lexical units at different scales. Furthermore, lexical-based fusion can also extract local region features because the lexical units embedded in the locally segmented sub-images themselves contain local region features, optimizing model fusion performance and improving the representational quality of the fused image features.

[0067] This invention extracts the long-range dependency between infrared and visible light images through two independent Transformer branches. In the fusion module, it searches for the optimal fusion result by applying intensity loss from the infrared image and L1 loss from the visible light image, while preserving more detail and brightness information from the visible light image, ultimately achieving the fusion of infrared and visible light images. This also optimizes image fusion accuracy and effect. This invention solves the technical problems of poor performance, high complexity, and limited feature extraction and representation in existing fusion models. Attached Figure Description

[0068] Figure 1This is a schematic diagram of the Transformer-based infrared and visible light image fusion framework in the multi-granularity term-based infrared and visible light image fusion method of Embodiment 1 of the present invention.

[0069] Figure 2 This is a schematic diagram of the basic steps of the infrared and visible light image fusion method based on multi-granularity terms in Embodiment 1 of the present invention;

[0070] Figure 3 This is a schematic diagram illustrating the principle of multi-granularity word global feature extraction in Embodiment 1 of the present invention.

[0071] Figure 4 This is a schematic diagram illustrating the specific steps of multi-granularity lexical fusion in Embodiment 1 of the present invention;

[0072] Figure 5 This is a comparison chart of the ablation experiment results of the learnable attention module in Embodiment 2 of the present invention;

[0073] Figure 6 This is a comparison chart of the ablation experiment results of the multi-granularity word module in Embodiment 2 of the present invention;

[0074] Figure 7 This is a schematic diagram comparing the processing effects of various methods on the TNO dataset in Embodiment 2 of the present invention;

[0075] Figure 8a This is a schematic diagram of the first index analysis of the fused images of each method on the TNO dataset in Embodiment 2 of the present invention;

[0076] Figure 8b This is a schematic diagram of the second index analysis of the fused images of each method on the TNO dataset in Embodiment 2 of the present invention;

[0077] Figure 8c This is a schematic diagram of the third index analysis of the fused images of each method on the TNO dataset in Embodiment 2 of the present invention;

[0078] Figure 8d This is a schematic diagram of the fourth index analysis of the fused images of each method on the TNO dataset in Embodiment 2 of the present invention;

[0079] Figure 8e This is a schematic diagram of the fifth index analysis of the images fused by various methods on the TNO dataset in Embodiment 2 of the present invention;

[0080] Figure 8f This is a schematic diagram of the sixth index analysis of the fused images of each method on the TNO dataset in Embodiment 2 of the present invention;

[0081] Figure 9 The images shown are the results of image fusion for various methods in a nighttime scene on the Roadscene dataset in Embodiment 2 of this invention.

[0082] Figure 10 This is a comparison of the image fusion effects of various methods on the LLVIP dataset in nighttime scenes according to Embodiment 2 of the present invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Example 1

[0085] like Figure 1 As shown, the infrared and visible light image fusion method based on multi-granularity terms provided by this invention involves a Transformer-based infrared and visible light image fusion framework. It extracts the long-range dependencies between infrared and visible light images through two independent Transformer branches. In the fusion module, it searches for the optimal fusion result by considering the intensity loss of the infrared image and the L1 loss of the visible light image, ultimately achieving the fusion of the infrared and visible light images.

[0086] In this embodiment, the infrared and visible light image fusion method based on multi-granularity terms includes the following steps:

[0087] Step S1: Image multi-scale decomposition: Decompose the infrared image and the visible light image at three different scales to obtain three different scale branches.

[0088] Step S2: Multi-granularity lexical global feature extraction;

[0089] like Figure 3 As shown, in this embodiment, step S2 further includes the following specific steps:

[0090] S21. Divide the infrared and visible light images into multi-scale regional patches;

[0091] S22. Convert infrared and visible light images into infrared sequences. and visible light sequence

[0092] S23. Embedding and processing infrared sequences using a preset linear projection E. and visible light sequence Encoded position information is added to each sequence to obtain the encoded infrared sequence. and encoding visible light sequences

[0093] S24. Utilize a fully connected layer to encode the infrared sequence using pre-embedded logic. and encoding visible light sequences Perform an embedding operation to obtain relation extraction parameters;

[0094] S25. Utilize the Multi-Head Self-Attention (MSA) mechanism to extract parameters based on pre-defined logical processing relationships, thereby extracting long-range dependencies from infrared and visible light images to obtain multi-head self-attention fusion parameters. These parameters include: infrared lexical units. and visible light term

[0095] In this embodiment, the long-range dependencies of infrared and visible light images are calculated using two independent Transformer branches. To comprehensively extract multi-scale long-range dependencies, this invention designs three Transformer models for each branch. Given an infrared image (I) and a visible light image (V), where I,V∈R H×W×C H, W, and C represent the height, width, and channel size of the source image, respectively. This invention first divides the two original images into multi-scale patches, further dividing the infrared image into N Ps at each scale. s 2 Size Where s is the scale applied to the segmented image, and s is defined as 1, 2, and 3 respectively. Additionally, in this invention, P... S This also satisfies P1=P2=P3. Based on this, the infrared and visible light images are converted into sequences. and Then embedded via linear projection E and Encoded position information is added to each sequence. and Its expression is shown in formula (1):

[0096]

[0097] In the formula, and This represents the encoded sequences of two original images at different scales of s.

[0098] In addition, the present invention utilizes a fully connected layer to and Embedded in the query, key, and value, as shown below:

[0099]

[0100] in and Let LN represent the query, key, and value of the infrared and visible light image sequences, and LN represent a fully connected layer. Furthermore, this invention utilizes a multi-head self-attention (MSA) mechanism to extract the long-range dependencies between the infrared and visible light images, as shown in equation (3):

[0101]

[0102] in and This is the output of MSA, which is applied to the next fusion stage.

[0103] Step S3: Set the loss function: Design a loss function to supervise the training of the proposed method to simulate the data distribution of the original image. Since infrared images are obtained by capturing thermal radiation, the content in infrared images has pixel intensity characteristics. Therefore, the loss between the infrared image and the fused image in the intensity domain is calculated, i.e. Since visible light sensors describe a scene by capturing reflected light, to preserve more detail and brightness information from the visible light image, L1 loss is used to constrain the fused image to have a data distribution similar to the visible light image, defined as... The total loss function is shown in equation (4):

[0104]

[0105] Where L represents the total loss value, M represents the total number of pixels, and f, I, and V represent the fusion result, the infrared image, and the visible light image, respectively. F Let λ denote the Forbenius norm of the matrix, and λ is designed to balance these two terms.

[0106] like Figure 4 As shown, step S4, multi-granularity lemma fusion: The infrared image and the visible light image are fused using a multi-granularity lemma fusion module. In this embodiment, step S4 further includes:

[0107] In this embodiment, learnable attention weights are introduced to capture... and The correlations between them are defined as shown in formula (5):

[0108]

[0109] Where f represents the features calculated by fusing lexical units at scale s, and R represents the reshaping operation. and Indicating infrared morphemes and visible light term Learnable attention weights, as defined in this invention To balance the importance of infrared and visible terms being in the same position.

[0110] This invention reconstructs the fused image using features at different scales, as defined in formula (6):

[0111] f = g(f1,…f s (6)

[0112] Where f represents the fused image, g represents the dimensionality reduction mapping operation of the convolutional layer, and the kernel is 1×1.

[0113] Example 2

[0114] This invention demonstrates the effectiveness of its fusion strategy through ablation experiments, and conducts qualitative and quantitative comparisons and generalization experiments on three public datasets.

[0115] A. Ablation test

[0116] 1) Ablation Analysis of Learnable Attention Weights: This invention introduces learnable attention weights to estimate the importance of corresponding words in the word-based fusion process. Therefore, the model is trained by discarding the learnable attention weights (unweighted), and the effectiveness is demonstrated through comparison.

[0117] like Figure 5 As shown, learnable attention weights play a crucial role in word-based fusion, for example, in Figure 5 The red square shows that the fusion model with learnable attention weights contains more details and edge information, which proves the effectiveness of learnable attention weights.

[0118] 2) Multi-granularity lexical fusion ablation analysis: In this study, infrared and visible light images are fused to extract multi-scale long-range dependencies and capture the attentional relevance of corresponding lexical units at different scales. Therefore, the effectiveness of the fusion model can be demonstrated by training it with the multi-granularity operation removed (without multi-granularity).

[0119] like Figure 6 As shown, in this embodiment, the introduction of multi-granularity modules results in richer thermal infrared details and better visual effects, such as... Figure 6 The region present in the red square. Furthermore, the method of this invention captures more background texture information compared to the absence of multi-granularity tokens, demonstrating the rationality and necessity of multi-granularity token fusion.

[0120] B. Comparative Experiment

[0121] In the experiments of this invention, the method of this invention is evaluated qualitatively and quantitatively. Qualitative analysis uses the human visual system to evaluate the fusion results, primarily focusing on brightness, sharpness, and contrast. Quantitative analysis employs six evaluation metrics to assess these methods. These metrics include mutual information (MI), standard deviation (SD), average gradient (AG), spatial frequency (SF), edge intensity (EI), and peak signal-to-noise ratio (PSNR). The purpose of fused images is to measure the information retained by the fused image from the source image. SD reflects the contrast of the fused image, while AG primarily measures the texture information. SF measures the grayscale change rate of the fused image, reflecting its sharpness. EI is used to evaluate the edge information of the fused image. For these metrics, higher values ​​indicate better performance of the fusion method.

[0122] 1) Qualitative analysis: In the comparative experiment, the method of the present invention was compared with 8 fusion methods on 35 TNO image pairs.

[0123] like Figure 7 As shown, traditional methods, CNN-based methods, and GAN-based methods all fail to preserve clear background texture information, while Transformer-based methods perform better in preserving background details. Furthermore, the method of this invention retains even more detail and texture information compared to existing methods combining CNN and Transformer (CGTF). For example, the area magnified by the red square in this invention reflects the sharpest edges and details in the result. In addition, compared to other methods, the method of this invention also preserves more significant thermal infrared information. This invention selects a region from each image and scales it within green squares, demonstrating that the processing result of this invention has higher brightness in the face region than other methods.

[0124] 2) Quantitative Analysis: In the quantitative experiment, 35 TNO image pairs were selected for comparison with these methods. This invention selected six indicators to objectively evaluate its method. As shown in Table 1, the method of this invention performs well on all six indicators.

[0125] As shown in Figure 8, the figure illustrates more details of the quantitative analysis of 35 image pairs. The largest MI value indicates that the method retains rich original image information, and the largest SD value indicates that the method has higher contrast than other methods. Furthermore, the model of this invention also has the largest values ​​in AG, SF, EI, and PSNR, indicating that the method of this invention retains more texture and detail information with lower noise levels.

[0126] Table 1: Metrics of different schemes on the TNO dataset

[0127]

[0128] C. Generalization experiment

[0129] To verify the generalization ability of the proposed model, in addition to the TNO dataset, this invention selected 100 infrared and 100 visible light images from the Roadscene dataset and the LLVIP dataset respectively to conduct qualitative and quantitative experiments.

[0130] like Figure 9 As shown in the figure, qualitative experiments on the Roadscene dataset demonstrate that the method of this invention not only preserves more background texture in nighttime scenes but also exhibits more pronounced thermal infrared characteristics and details; in daytime scenes, it can preserve even more thermal infrared characteristics and details, with nighttime scene comparison results as shown in the figure. Figure 9 As shown in Table 2, quantitative experiments show that the results of this invention have the highest values ​​for MI, SD, AG, EI, and SF, while remaining acceptable in terms of PSNR.

[0131] Table 2: Metrics for different schemes on the Roadscene dataset

[0132]

[0133]

[0134] like Figure 10 As shown in Figure 8, qualitative experiments on the LLVIP dataset demonstrate that the method of this invention contains more detail and higher contrast in nighttime scenes, and it can also provide clearer contours and edges than other methods. In daytime scenes, it can simultaneously preserve a certain amount of texture and significant infrared targets, with the nighttime scene comparison results shown in Figure 8. Quantitative experiments, as shown in Table 3, show that the results of this invention have the highest values ​​in MI, SD, AG, EI, SF, and PSNR, which proves the superiority of the method of this invention.

[0135] Table 3: Metrics for different schemes on the LLVIP dataset

[0136]

[0137] D. Efficiency Comparison

[0138] In this work, the invention also compares efficiency by providing the average runtime of each method on three datasets. Traditional methods are implemented on CPUs, while other methods are implemented on GPUs. Table 4 shows that traditional MSVD-based and wavelet-based methods are less time-consuming than most DL-based methods, while Transformer-based methods are more time-consuming than CNN-based and GAN-based methods.

[0139] Table 4: Average runtime of different methods on three datasets

[0140]

[0141] In summary, the ablation experiments demonstrate the rationality and necessity of introducing learnable attention weights and a multi-granularity word fusion module. Comparative and generalization experiments show that the proposed method has advantages in both quantitative and qualitative aspects. While it offers some computational efficiency compared to CGTF, which is also based on the Transformer framework, it is significantly less efficient than CNN-based and GAN-based methods. However, due to its superior processing performance, the method of this invention has a wide range of applications.

[0142] This invention captures the relevance of corresponding lexical units by introducing learnable attention weights, thereby fusing infrared and visible light images at a multi-granularity lexical dimension. This method can perceive the differences in multimodal information present at the same location for infrared and visible light lexical units. This invention extracts long-range dependencies of each image at multiple scales, embeds multi-granularity lexical units into sub-images through local multi-scale segmentation, reconstructs multi-scale features using multi-granularity fusion, and calculates the fused image through dimensionality reduction mapping operations. This invention can achieve the fusion of infrared and visible light images at a multi-scale lexical dimension based on a pure Transformer, and has better fusion performance compared to other methods.

[0143] This invention extends fusion to multi-granularity lexical fusion to extract multi-scale long-range dependencies of each source image and capture the attentional relevance of corresponding lexical units at different scales. Furthermore, lexical-based fusion can also extract local region features because the lexical units embedded in the locally segmented sub-images themselves contain local region features, optimizing model fusion performance and improving the representational quality of the fused image features.

[0144] This invention extracts the long-range dependency between infrared and visible light images through two independent Transformer branches. In the fusion module, the optimal fusion result is found by considering the intensity loss of the infrared image and the L1 loss of the visible light image, while retaining more details and brightness information of the visible light image, thus ultimately achieving the fusion of infrared and visible light images.

[0145] This invention optimizes image fusion accuracy and fusion effect. It solves the technical problems of poor performance, high complexity, and limited feature extraction and representation in existing fusion models.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for infrared and visible image fusion based on multi-granularity word elements, characterized in that, The method comprises: S1, acquiring infrared images and visible light images, and decomposing the infrared images and the visible light images on not less than two difference scales respectively; S2, multi-granularity word global feature extraction: calculating long-range dependency of the infrared images and the visible light images through two independent Transformer branches, wherein not less than two Transformer models are designed in each independent Transformer branch, and the step S2 comprises: S21, dividing the infrared images and the visible light images into multi-scale sub-regional patches; S22, converting the infrared image and the visible light image into an infrared sequence and a visible light sequence ; S23, using the pre-set linear projection E Embedding the infrared sequence and the visible light sequence and adding position information in each sequence to obtain an encoded infrared sequence and an encoded visible light sequence ; S24. Using a fully connected layer, the encoded infrared sequence is pre-embedded with logic. and the encoded visible light sequence Perform an embedding operation to obtain relation extraction parameters; S25, using a multi-head self-attention mechanism MSA to process the relationship extraction parameters by preset logic to extract the long-range dependency relationship from the infrared image and the visible light image to obtain multi-head self-attention fusion parameters, wherein the multi-head self-attention fusion parameters include: infrared word units and visible light word units ; S3, designing a loss function by using a preset logic, to supervise and train a preset multi-granularity word fusion model; S4, fusing the infrared images and the visible light images through a multi-granularity word fusion module, to obtain a multi-granularity word fusion output image, and the step S4 comprises: S41、with pre-set weight definition logic to obtain learnable attention weight, which is used to capture the infrared word element by using pre-set relationship capture logic correlation of the multi-granularity word element with the visible light word element ​ S42, processing the multi-granularity word correlation and difference scale features by using a preset reconstruction logic, to obtain the multi-granularity word fusion output image.

2. The multi-granularity token-based infrared and visible image fusion method according to claim 1, characterized in that, The step S21 comprises: dividing the infrared image into N patches of different sizes at each scale by using the following logic the multi-scale sub-regional patch: In the formula, s is a scale applied to the divided image, and s is defined as 1, 2 and 3 respectively. 3.The multi-granularity token based infrared and visual image fusion method of claim 1, wherein, In the step S23, the pre-established linear projection is used with the following logic E Embedding the infrared sequence and the visible light sequence with the addition of coded position information in each sequence: wherein and denote the encoding sequences of the two original images at different scales s . 4.The multi-granularity token based infrared and visual image fusion method of claim 1, wherein, In the step S24, using the fully connected layer, an embedding operation is performed on the encoded infrared sequence and the encoded visible light sequence with the following logic: wherein , , and , , query, key and value of the infrared and visible light image sequence, LN represents a fully connected layer. 5.The multi-granularity token based infrared and visual image fusion method of claim 1, wherein, In the step S25, the relationship extraction parameters are processed by using the following logic, to extract the long-range dependency from the infrared images and the visible light images: wherein and is the output of MSA, , , and , , query, key and value of the infrared and visible image sequence, LN represents a fully connected layer. 6.The multi-granularity token based infrared and visual image fusion method of claim 1, wherein, The step S3 comprises: S31, calculating the loss of the infrared images and the fusion images in the intensity domain by using the following logic: ； S32, the loss is obtained by using the following logic processing and total loss, so as to retain the detail and brightness information of the visible light image: In the formula, L denotes the total loss value, M denotes the total number of pixels, The step S41 comprises: respectively denote the fusion result, the infrared image and the visible light image, denotes the matrix Forbenius norm, is a balance parameter. 7.The multi-granularity token based infrared and visual image fusion method of claim 1, wherein, In the step S42, the multi-granularity word correlation and difference scale features are processed by using the following logic, to obtain the multi-granularity word fusion output image: S411, capture the infrared tokens with learnable attention weights using the following logic and the multi-granularity token correlation of the visible light tokens ​ wherein, denotes a scale s a feature computed from the lower fusion wordpiece, denotes a reshaping operation, and denotes a learnable attention weight for the infrared wordpiece and the visible light wordpiece . S412, balancing the infrared tokens and the visible light tokens importance in the same location.

8. The multi-granularity token-based infrared and visible image fusion method according to claim 7, characterized in that, In the step S412, the infrared tokens are balanced with the following logic and the visible light tokens importance in the same location: 。 9. The multi-granularity token based infrared and visible image fusion method according to claim 7, wherein, The system comprises: In the formula, denotes a fusion image, denotes a dimension reduction mapping operation of the convolution layer, and the kernel is 1x1.

10. A multi-granularity lexicon based infrared and visible image fusion system, characterized in that, A difference scale decomposition module is configured to acquire infrared images and visible light images, and decompose the infrared images and the visible light images on not less than two difference scales respectively; A multi-granularity word global feature extraction module is configured to calculate long-range dependency of the infrared images and the visible light images through two independent Transformer branches, wherein not less than two Transformer models are designed in each independent Transformer branch, the multi-granularity word global feature extraction module is connected with the difference scale decomposition module, and the multi-granularity word global feature extraction module comprises: An image segmentation module is configured to divide the infrared images and the visible light images into multi-scale sub-regional patches; A word fusion model training module is configured to design a loss function by using a preset logic, to supervise and train a preset multi-granularity word fusion model, and the word fusion model training module is connected with the multi-head self-attention fusion module; An image conversion module is used to convert the infrared image and the visible light image into an infrared sequence. and visible light sequence The image conversion module is connected to the image segmentation module; a linear projection embedding module, configured to utilize a preset linear projection E to embed the infrared sequence and the visible light sequence , and add coded position information in each sequence to obtain a coded infrared sequence and a coded visible light sequence , the linear projection embedding module being connected with the image conversion module; a relationship extraction module configured to utilize a fully connected layer to perform an embedding operation on the encoded infrared sequence and the encoded visible light sequence to obtain relationship extraction parameters, the relationship extraction module being connected to the linear projection embedding module. The multi-head self-attention fusion module is configured to utilize a multi-head self-attention mechanism (MSA) to process the relationship extraction parameter by using preset logic to extract the long-range dependency relationship from the infrared image and the visible light image, so as to obtain a multi-head self-attention fusion parameter, wherein the multi-head self-attention fusion parameter includes: an infrared word item and a visible light word item , and the multi-head self-attention fusion module is connected with the relationship extraction module. A multi-granularity word fusion output module is configured to fuse the infrared images and the visible light images through a multi-granularity word fusion module, to obtain a multi-granularity word fusion output image, and the multi-granularity word fusion output module is connected with the word fusion model training module, and the multi-granularity word fusion output module comprises: ​ a word correlation module, configured to define a preset weight definition logic to obtain a learnable attention weight, and to capture the infrared word by using a preset relationship capture logic and the visible light word multi-granularity word correlation; An image reconstruction module is configured to process the multi-granularity wordpiece correlation and the difference scale feature by using preset reconstruction logic to obtain a multi-granularity wordpiece fusion output image. The image reconstruction module is connected to the wordpiece correlation module.

Images