A multi-modal frequency domain cross image fusion method

Abstract

This invention provides a multimodal frequency domain cross-image fusion method, relating to the field of image fusion technology. The method includes: S1, constructing a multimodal image fusion network; wherein the multimodal image fusion network includes: an encoder network, a cross-modal fusion network, and a decoder; the execution steps of the multimodal image fusion network include: using the encoder network to extract different frequency domain features from each input modal image; using the cross-modal fusion network to fuse the frequency domain features of different modal images through cross-layer connections and cross-connections; inputting the fused features into the decoder to decode the fused features and output a fused image; S2, training the constructed multimodal image fusion network using multimodal images and outputting a fused image containing features from each modal image. Using this invention can improve the fusion effect of multimodal images, resulting in fusion results with clearer details and richer information.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a multimodal frequency domain cross-image fusion method. Background Technology

[0002] In modern multi-sensor systems, the fusion technology of infrared and visible light images is rapidly developing and has become a key tool in many applications. This technology improves overall perception capabilities by combining information acquired from different spectral ranges, not only enhancing image details and features but also expanding application possibilities, especially in target detection, surveillance, navigation, and driver assistance. Infrared imaging devices can detect objects in low-light or nighttime conditions by capturing thermal radiation, but they typically suffer from low resolution and insufficient contrast. Visible light imaging relies on ambient light, acquiring structural and color information of objects through reflected light. It offers high image resolution and rich detail, but image quality deteriorates significantly in low-light or complex environments. The fusion of infrared and visible light images combines the advantages of both, improving the overall performance of the system. The fused image demonstrates significant advantages in target recognition, environmental adaptability, and reliability.

[0003] Infrared and visible light image fusion methods can be broadly categorized into three types. Image-level fusion directly synthesizes the two images pixel-by-pixel, generating a fused image through transformation and reconstruction. This method is simple and intuitive, but may introduce noise and distortion from different data sources. Feature-level fusion extracts features from both images, fuses the feature information, and then makes a decision. It is highly effective in information compression and feature enhancement, but it also requires addressing the complexity of feature extraction. Decision-level fusion fuses decision results from different sensors based on high-level information, thereby improving the accuracy of the judgment. This is typically used in applications where only the final decision result is important. With the increasing maturity of artificial intelligence and deep learning technologies, the future of infrared and visible light image fusion technology may increasingly rely on data-driven intelligent models. Through large amounts of training data and complex network structures, deep learning methods have the potential to significantly improve the automation level and fusion effect of image fusion. At the same time, effectively processing and interpreting multimodal sensing information, and providing efficient fusion algorithms in real-time applications, remain significant challenges we will face. Summary of the Invention

[0004] This invention provides a multimodal frequency domain cross-image fusion method, which can improve the fusion effect of multimodal images, resulting in fusion results with clearer details and richer information. The technical solution is as follows:

[0005] This invention provides a multimodal frequency domain cross-image fusion method, comprising:

[0006] S1. Construct a multimodal image fusion network; wherein, the multimodal image fusion network includes: an encoder network, a cross-modal cross-fusion network, and a decoder; the execution steps of the multimodal image fusion network include:

[0007] An encoder network is used to extract different frequency domain features from the input modal images;

[0008] A cross-modal cross-fusion network is used to fuse the frequency domain features of images from different modalities through cross-layer connections and cross-connections;

[0009] The fused features are input into the decoder to decode the fused features and output the fused image;

[0010] S2. Train the constructed multimodal image fusion network using multimodal images, and output a fused image containing features of each modality.

[0011] Furthermore, the encoder network includes: a shared feature encoder, a basic transformation network encoder, and a detail convolutional neural network encoder;

[0012] Modal images include: infrared images and visible light images;

[0013] The step of using an encoder network to extract different frequency domain features from the input modal images includes:

[0014] Cross-modal shallow feature extraction is achieved by using a shared feature encoder to obtain shallow features from infrared and visible light images. in, These represent the shallow features of infrared and visible light images, respectively.

[0015] The shallow features are further extracted using a basic transformation network encoder to obtain low-frequency basic features. in, These represent the low-frequency fundamental features of infrared and visible light images, respectively.

[0016] A detail convolutional neural network encoder is used to further extract features from shallow features to obtain high-frequency detail features. in, These represent the high-frequency detail features of infrared and visible light images, respectively.

[0017] Furthermore, the shared feature encoder includes: two converter layers based on recovery converter blocks;

[0018] Each shared feature encoder consists of two stacked transformer layers based on recovery transformer blocks, with the parameters of the two shared feature encoders being independent of each other;

[0019] The formula for a shared feature encoder is expressed as:

[0020]

[0021] Where S(·) represents the process of feature extraction by the shared feature encoder, and {I,V} represent the input paired infrared image and visible light image, respectively.

[0022] Furthermore, the basic transformation network encoder includes a transformation network module based on a lightweight converter for extracting low-frequency basic features from shallow features;

[0023] Each transformation network module includes: 2 Norm layers, 1 basic Attention layer, and 1 MLP layer; where Norm is normalization, Attention is attention, and MLP is multilayer perceptron.

[0024] The formula for the basic transformation network encoder is expressed as:

[0025] Norm(Attention(Norm(MLP(·))))=B(·)

[0026]

[0027] Where B(·) represents the process of feature extraction by the basic transformation network encoder.

[0028] Furthermore, the detail convolutional neural network encoder includes three INNs for extracting high-frequency detail features from shallow features; wherein, the INN is a reversible neural network;

[0029] In each processing step of the reversible neural network, the transformation process is as follows:

[0030]

[0031] Where ⊙ represents the Hadamard product operation. This represents the 1st to cth channels in the input feature of the kth reversible layer, where k = 1, 2, ..., K, K represents the number of reversible layers, C represents the number of input feature channels, and CAT(·) is the channel concatenation operation. i It is an arbitrary mapping function, i = 1,...,3, where i represents different mapping function identifiers. After the reversible neural network is processed repeatedly 3 times, the output is the feature extraction result of the detail convolutional neural network encoder, i.e., high-frequency detail features.

[0032] The formula for the entire detail convolutional neural network encoder is expressed as follows:

[0033]

[0034] Where D(·) represents the process of feature extraction by the detail convolutional neural network encoder.

[0035] Furthermore, the cross-modal fusion network includes high-frequency and low-frequency branches and channel attention modules to fuse the frequency domain features extracted by the encoder; each high-frequency and low-frequency branch consists of three layer feature extraction modules representing different depths.

[0036] The layer feature extraction process of the layer feature extraction module is represented as follows:

[0037] LFC(Θ B ′)=Θ B ",LFC(Θ B ′,Θ B ")=Θ B "′,

[0038] LFC(Θ D ′)=Θ D ",LFC(Θ D ′,Θ D ")=Θ D "′

[0039] in, These represent the low-frequency fundamental features of infrared and visible light images, respectively. These represent the high-frequency detail features of infrared and visible light images, respectively. LFC(·) represents layer feature extraction, used to extract high and low frequency features at different levels. Depending on the number of processing iterations, different stages of features from shallow to deep can be generated; {Θ B ′,Θ B ",Θ B "′" represents the characteristics of each stage of low frequency; {Θ D ′,Θ D ",Θ D "″′} represents the characteristics of each stage of high frequency.

[0040] Furthermore, the method of using a cross-modal cross-fusion network to fuse various frequency domain features of different modal images through cross-layer connections and cross-connections includes:

[0041] For the frequency domain features of the obtained images of different modalities, the layer feature extraction module is used to extract features from different stages from shallow to deep.

[0042] In the process of processing stage features, a cross-layer connection method is adopted to link features at different levels and pass shallow features to various deep features.

[0043] Based on the requirements for processing different frequency domain feature information, frequency domain feature fusion is achieved by using a cross-connection approach to extract primary features {F}. B ,F D} and link them with the cross-layer processing results to obtain the heteromodal co-frequency fusion features {Θ} modified by hetero-frequency domain features. B ,Θ D}; where, the process of cross-layer connections and cross-connections is represented as:

[0044]

[0045] LFC(Θ B ′,Θ B ",Θ B "′,F D )=Θ B ,LFC(Θ D ′,Θ D ",Θ D "′,F B )=Θ D

[0046] Among them, F(·) base With F(·) detail These represent the extraction processes of primary basic features and primary detailed features, respectively.

[0047] The heteromodal co-frequency fusion features are fed into the channel attention module for fusion to obtain the final fusion feature map.

[0048] Furthermore, a hybrid gradient loss is added during the training process. This hybrid gradient loss is composed of a surface layer and a gradient layer, ensuring that the multimodal image fusion network can learn more complete fusion gradient information during training.

[0049] Furthermore, the hybrid gradient loss converts the visible light image into the Lab color space and extracts the L luminance channel as the visible light processing channel; the L luminance channel of the visible light image is then mixed with the infrared image channel using a Mix(·) process to generate the surface layer loss result. Simultaneously, the Sobel operator is used to extract gradients from the L-channel of the visible light image and the infrared image channel. The extracted gradient maps are then subjected to a gradient mixing process, MixGrad(·), to produce gradient layer loss results. The obtained surface layer loss and gradient layer loss are summed according to their weights to obtain the final hybrid gradient loss, Loss. Mixed_Grad , is represented as:

[0050]

[0051] Among them, {V LI, F} represent the L brightness channel of the visible light image, the infrared image, and the fused image, respectively. Let L represent the gradient map of the visible light image, the gradient map of the infrared image, and the gradient map of the fused image, respectively, and λ be the summation weight of the surface layer loss and the gradient layer loss.

[0052] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described multimodal frequency domain cross-image fusion methods.

[0053] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0054] In this embodiment, a multimodal image fusion network is constructed, comprising an encoder network, a cross-modal fusion network, and a decoder. The encoder network extracts different frequency domain features from the input modal images. By designing encoder networks tailored to different frequency domain features, the high- and low-frequency feature extraction process of different modal images is achieved. An efficient cross-modal fusion network is used to fuse the frequency domain features of different modalities. Through cross-layer connections and cross-connections, the network effectively utilizes and efficiently transmits high- and low-frequency information from different levels, improving information transmission efficiency and utilization. The fused features are input to the decoder for decoding, outputting a fused image. The entire multimodal image fusion network is trained to output a fused image containing features from each modal image. Using this invention, the fusion effect of multimodal images can be improved, resulting in fusion results with clearer details and richer information. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of a multimodal frequency domain cross-image fusion method provided by an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the structure of the multimodal image fusion network provided in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the structure of the detail convolutional neural network encoder provided in an embodiment of the present invention;

[0059] Figure 4This is a schematic diagram of the cross-modal cross-fusion network provided in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the hybrid gradient loss structure provided in an embodiment of the present invention. Detailed Implementation

[0061] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0062] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0063] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0064] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0065] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0066] This invention provides a multimodal frequency domain cross-image fusion method, such as... Figure 1 and Figure 2 As shown, the processing flow of this method may include the following steps:

[0067] S1. Construct a multimodal image fusion network; wherein, the multimodal image fusion network includes: an encoder network, a cross-modal cross-fusion network (also known as: a high- and low-frequency feature cross-fusion network, such as...) Figure 2 (as shown) and decoder; the execution steps of the multimodal image fusion network include:

[0068] An encoder network is used to extract different frequency domain features from the input modal images; thus, by designing encoder networks for different frequency domain features, the high and low frequency feature extraction process of different modalities can be realized.

[0069] Cross-modal cross-fusion networks are used to fuse the frequency domain features of different modal images through cross-layer and cross-connection, so as to effectively utilize and efficiently transmit high and low frequency information at different levels, thereby improving the efficiency and utilization of information transmission.

[0070] The fused features are input into the decoder to decode the fused features and output the fused image;

[0071] S2. Train the constructed multimodal image fusion network using multimodal images, and output a fused image containing features of each modality.

[0072] In a preferred embodiment, the encoder network in this embodiment includes: a shared feature encoder, a basic transformation network encoder, and a detail convolutional neural network encoder;

[0073] Modal images include: infrared images and visible light images;

[0074] The step of using an encoder network to extract different frequency domain features from the input modal images includes:

[0075] A1 achieves cross-modal shallow feature extraction by using a shared feature encoder, obtaining shallow features from infrared and visible light images. in, These represent the shallow features of infrared and visible light images, respectively.

[0076] In this embodiment, the shared feature encoder includes: two converter layers based on the Restormer block;

[0077] Each shared feature encoder consists of two stacked transformer layers based on recovery transformer blocks, with the parameters between the two shared feature encoders being independent of each other to better extract shallow features from different modalities of infrared and visible light images;

[0078] The formula for a shared feature encoder is expressed as:

[0079]

[0080] Where S(·) represents the process of feature extraction by the shared feature encoder, and {I,V} represent the input paired infrared image and visible light image, respectively.

[0081] A2 further extracts features from the shallow features using a basic transformation network encoder to obtain low-frequency basic features. in, These represent the low-frequency fundamental features of infrared and visible light images, respectively.

[0082] In this embodiment, the basic transformation network encoder includes a transformation network module based on a lightweight transformer, used to extract low-frequency basic features from shallow features;

[0083] Each transformation network module includes: 2 Norm layers, 1 basic Attention layer, and 1 MLP layer; where Norm is normalization, Attention is attention, and MLP is multilayer perceptron.

[0084] The formula for the basic transformation network encoder is expressed as:

[0085] Norm(Attention(Norm(MLP(·))))=B(·)

[0086]

[0087] Where B(·) represents the process of feature extraction by the basic transformation network encoder.

[0088] A3 uses a detail convolutional neural network encoder to further extract features from shallow features, obtaining high-frequency detail features. in, These represent the high-frequency detail features of infrared and visible light images, respectively.

[0089] In this embodiment, the detail convolutional neural network encoder includes three INNs (Invertible Neural Networks) used to extract high-frequency detail features from shallow features; wherein, INN is an invertible neural network;

[0090] like Figure 3 As shown, the transformation process in each step of the invertible neural network INN is as follows:

[0091]

[0092] Where ⊙ represents the Hadamard product operation. This represents the 1st to cth channels (k = 1, 2, ..., K) in the input features of the kth reversible layer, where K represents the number of reversible layers, C represents the number of input feature channels, h × w × c represents the feature scale, and CAT(·) is the channel concatenation operation. i (i = 1, ..., 3) is any mapping function, where i represents different mapping function identifiers. In this embodiment, the bottleneck residual block (BRB) in the MobileNetV2 network is selected as the mapping function. After the reversible neural network is processed three times, the feature extraction result of the detail convolutional neural network encoder is output, which is the high-frequency detail feature.

[0093] It should be noted that: Figure 3 The document provides a detailed flowchart of the reversible neural network (INN). After repeating the reversible neural network process three times, the output result is the result of the detail convolutional neural network encoder.

[0094] The formula for the entire detail convolutional neural network encoder is expressed as follows:

[0095]

[0096] Where D(·) represents the process of feature extraction by the detail convolutional neural network encoder.

[0097] In this embodiment, a shared feature encoder, a basic transformation network encoder, and a detail convolutional neural network encoder are used to encode the input infrared and visible light images to obtain the low-frequency basic features and high-frequency detail features of each of the two modalities. In this way, by designing encoder networks for different frequency domain features, the high and low frequency feature extraction process of different modalities can be realized.

[0098] As a preferred embodiment, in this embodiment, such as Figure 4 As shown, the cross-modal fusion network includes high-frequency and low-frequency branches and channel attention modules, which are used to fuse the frequency domain features extracted by the encoder; each high-frequency and low-frequency branch consists of three layer feature extraction modules representing different depths.

[0099] The layer feature extraction process of the layer feature extraction module is represented as follows:

[0100] LFC(Θ B ′)=Θ B ",LFC(Θ B ′,Θ B ")=Θ B "′,

[0101] LFC(Θ D ′)=Θ D ",LFC(Θ D ′,Θ D ")=Θ D "′

[0102] in, These represent the low-frequency fundamental features of infrared and visible light images, respectively. These represent the high-frequency detail features of infrared and visible light images, respectively. LFC(·) represents layer feature extraction, used to extract high and low frequency features at different levels. Depending on the number of processing iterations, different stages of features from shallow to deep can be generated; {Θ B ′,ΘB ",Θ B "′" represents the characteristics of each stage of low frequency; {Θ D ′,Θ D ",Θ D "″′} represents the characteristics of each stage of high frequency.

[0103] It should be noted that: Figure 4 The fourth layer feature extraction module only represents the feature extraction result and does not include the processing process.

[0104] In a preferred embodiment, the use of a cross-modal cross-fusion network to fuse frequency domain features of different modal images through cross-layer and cross-connection methods includes:

[0105] For the frequency domain features of the obtained images of different modalities, the layer feature extraction module is used to extract features from different stages from shallow to deep.

[0106] In the process of processing stage features, a cross-layer connection method is adopted to link features at different levels and pass shallow features to various deep features. Through this processing method, the cross-modal cross-fusion network can improve the fusion depth of features with the same frequency while retaining the original features.

[0107] Based on the requirements for processing different frequency domain feature information, frequency domain feature fusion is achieved by using a cross-connection approach to extract primary features {F}. B ,F D} and link them with the cross-layer processing results to obtain the heteromodal co-frequency fusion features {Θ} modified by hetero-frequency domain features. B ,Θ D This improves the fusion depth between frequency domain features, enabling the mutual transfer of information between different frequency domain features; the processes of cross-layer connections and cross-connections are represented as follows:

[0108]

[0109] LFC(Θ B ′,Θ B ",Θ B "′,F D )=Θ B ,LFC(Θ D ′,Θ D ",Θ D "′,F B )=Θ D

[0110] Among them, F(·) base With F(·) detailThese represent the extraction processes of primary basic features and primary detailed features, respectively. This approach enhances the adaptability of the cross-modal cross-fusion network to cross-frequency domain feature fusion, improving the fusion effect. After cross-layer and cross-connections, the cross-modal cross-fusion network generates highly adaptive, original feature-preserving, and deeply fused heteromodal same-frequency fusion features {Θ}. B ,Θ D};

[0111] The heteromodal co-frequency fusion features are fed into the channel attention module for fusion to obtain the final fusion feature map.

[0112] In this embodiment, a cross-modal cross-fusion network is built to achieve an efficient feature fusion process. The unique cross-layer connection method brings a more complete modal information integration effect to the fused image, and the cross-connection design provides higher robustness for the fused features.

[0113] As a preferred embodiment, in this embodiment, a hybrid gradient loss is added to the commonly used detail loss (MSE loss) and pixel loss (SSIM loss) during the training process. The hybrid gradient loss is composed of a surface layer and a gradient layer, which ensures that the multimodal image fusion network can learn more complete fusion gradient information during the training process.

[0114] As a preferred embodiment, in this embodiment, such as Figure 5 As shown, the hybrid gradient loss, in order to preserve visible light image information as much as possible and reduce information loss, converts the visible light image into the Lab color space. Since the human eye is more sensitive to changes in brightness, and the brightness component directly affects the contrast and structural details of the image, the L brightness channel is extracted as the visible light processing channel. The L brightness channel of the visible light image is then mixed with the infrared image channel using Mix(·) to produce the surface layer loss result. Simultaneously, the Sobel operator is used to extract gradients from the L-channel of the visible light image and the infrared image channel. The extracted gradient maps are then subjected to a gradient mixing process, MixGrad(·), to produce gradient layer loss results. The obtained surface layer loss and gradient layer loss are summed according to their weights to obtain the final hybrid gradient loss, Loss. Mixed_Grad (Right now: Figure 2 L in Mixed_Grad ), represented as:

[0115]

[0116] Among them, {V L I, F} represent the L brightness channel of the visible light image, the infrared image, and the fused image, respectively. Let L represent the gradient maps of the visible light image, the infrared image, and the fused image, respectively. λ is the summation weight of the surface layer loss and the gradient layer loss, which is set to 10 here. Finally, the loss function is expressed as:

[0117] Loss = MSEloss + SSIMloss + Loss Mixed_Grad

[0118] Among them, MSEloss is the detail loss, which is obtained by calculating the MSE values ​​of the infrared image, the visible light image and the fused image respectively, and then weighting the two; SSIMloss is the pixel loss, which is obtained by calculating the SSIM values ​​of the infrared image, the visible light image and the fused image respectively, then inverting the inverted results and weighting the results.

[0119] In this embodiment, the hybrid gradient loss maximizes the utilization of information from both visible and infrared images in the loss function calculation by preserving the brightness and darkness details of the visible light image to the greatest extent. This provides better guidance for the training process of the entire multimodal image fusion network, helping the multimodal image fusion network to better utilize modal information while bringing clearer details to the fusion result.

[0120] In this embodiment, the multimodal image fusion network fuses visible light and infrared images, extracts low-frequency basic features and high-frequency detail features of the two modalities respectively, and achieves the fusion process of the two features between the two modalities through an efficient cross-modal fusion network. The fused image is output by the decoder, and a hybrid gradient loss that can improve the multimodal image fusion effect is added during the training process to help the network train. The infrared image information and the brightness information of the visible light image are taken into account, and the final output is a fused image with clearer details and richer information.

[0121] In this embodiment, a well-trained multimodal image fusion network can be used to efficiently extract features from each modality, obtain high-frequency detail features and low-frequency basic features from infrared and visible light images, and achieve effective feature fusion. The final output is a fused image with clearer details and richer information, including infrared and visible light features.

[0122] In this embodiment, to verify the effectiveness of the multimodal frequency domain cross-image fusion method provided by the present invention, its performance is evaluated and tested using the MSRS dataset. The test environment and parameters are set as follows: 100 training iterations, 16 training batch sizes, 320×240 training image sizes, Adam optimizer for training, StepLR scheduler for learning rate optimization, and two RTX2080Ti GPUs for training. The evaluation metrics and results are described below:

[0123] The evaluation results of the multimodal frequency domain cross-image fusion method (hereinafter referred to as the "method of the present invention") provided in this embodiment of the invention are compared with other methods in Table 1. To reflect the advantages of the method of the present invention in terms of fusion results, multiple indicators are selected for comparison, including EN (Entropy) entropy, SD (Standard Deviation) standard deviation, SSIM (Structural Similarity), AG (Average Gradient) average gradient, and PSNR (Peak Signal-to-Noise Ratio).

[0124] In this embodiment, entropy (EN) is an evaluation metric based on information theory, representing the amount of information in an image, and its mathematical definition is as follows:

[0125]

[0126] Where L represents the number of gray levels, p l This represents the normalized histogram of the corresponding gray levels in the fused image. The higher the entropy, the more information the fused image contains, and the better the performance of the fusion method.

[0127] In this embodiment, the standard deviation (SD) metric is based on a statistical concept that reflects the distribution and contrast of the fused image. The mathematical definition of standard deviation is as follows:

[0128]

[0129] Where M represents the length of the processed image, N represents the width of the processed image, F(i,j) represents the pixel value at position (ij), and μ represents the average value of the fused image. Because the human visual system is sensitive to contrast, high-contrast regions always attract human attention. Therefore, high-contrast fused images often produce a large standard deviation, meaning the fused image can achieve good visual results.

[0130] In this embodiment, the Structural Similarity Index (SSIM) is used to simulate image loss and distortion. This index consists of three main components: correlation loss and brightness and contrast distortion. The product of these three components is the evaluation result of the fused image, defined as follows:

[0131]

[0132] Among them, SSIM X,F σ represents the structural similarity between the source image X and the fused image F; x and f represent image patches of the source image and the fused image within the sliding window, respectively; xf σ represents the covariance between the source image and the fused image; x and σ fμ represents the standard deviation (SD). x and μ f Let C1, C2, and C3 represent the average values ​​of the source and fused images, respectively; C1, C2, and C3 are parameters used to stabilize the algorithm; when C1 = C2 = C3 = 0, SSIM is simplified to a general image quality index. Therefore, the structural similarity between all source and fused images can be written as follows:

[0133] SSIM = SSIM A,F +SSIM B,F

[0134] Among them, SSIM A,F and SSIM B,F These represent the structural similarity between infrared / visible light and the fused image, respectively.

[0135] In this embodiment, the average gradient (AG) metric quantifies the gradient information of the fused image and represents its detail and texture. The average gradient metric is defined as follows:

[0136]

[0137] in, These represent the horizontal and vertical gradients of the pixel at position (i,j), respectively. The larger the average gradient metric, the more gradient information the fused image contains, and the better the performance of the fusion algorithm.

[0138] In this embodiment, the Peak Signal-to-Noise Ratio (PSNR) is the ratio of peak power to noise power in the fused image, thus reflecting the distortion during the fusion process. The PSNR is defined as follows:

[0139]

[0140] Among them, MSE AF and MSE BF Let A(i,j) represent the differences between the fused image and the infrared and visible light images, respectively, where A(i,j) represents the pixel value of the infrared image and B(i,j) represents the pixel value of the visible light image; r represents the peak value of the fused image. The higher the peak signal-to-noise ratio, the closer the fused image is to the source image, and the smaller the distortion produced by the fusion method.

[0141] Table 1 compares the method of this invention with other methods in the MSRS dataset.

[0142]

[0143] As shown in Table 1, compared with other methods, the method of this invention achieved satisfactory results in all indicators, indicating that the infrared-visible light fusion result obtained by the method of this invention performs excellently in terms of image quality. The method of this invention shows significant improvements in EN and AG values ​​compared to other methods, indicating that the fused image obtained by the method of this invention possesses richer information and better gradient results; the excellent result of the method of this invention in SD value also indicates that the obtained fused image has better visual effects.

[0144] To verify the significance of each part of the method described in this embodiment, an ablation experiment was also conducted in this embodiment.

[0145] This embodiment verifies the effectiveness of the cross-modal cross-fusion network and the hybrid gradient loss through ablation experiments. Table 2 shows the fusion result metrics for the summation fusion network (Ori+Add) without cross-modal cross-fusion network and hybrid gradient loss, the original cross-modal cross-fusion network (Ori+Cross), and the original cross-modal cross-fusion network with added hybrid gradient loss (Ori+Cross+G). Here, we select SF (Spatial Frequency), SSIM (Structural Similarity), AG (Average Gradient), and PSNR (Peak Signal-to-Noise Ratio) for comparison:

[0146] In this embodiment, spatial frequency (SF) is a gradient-based image quality metric, namely the horizontal gradient and the vertical gradient, also known as row frequency (RF) and column frequency (CF), respectively. The spatial frequency metric effectively measures the gradient distribution of an image, thereby revealing the image's details and texture. This metric is defined as follows:

[0147]

[0148] According to the human visual system, fused images with large SFs are sensitive to human perception and have rich edges and textures.

[0149] Table 2 shows the ablation experiments of the method of this invention in the MSRS dataset.

[0150]

[0151] As can be seen from the results in Table 2, by comparing with other ablation experiment results, the cross-modal cross-fusion network and hybrid gradient loss provided by the method of this invention both improve the network fusion effect; after the network adopts the cross-modal cross-fusion network and hybrid gradient loss, the fusion result has more gradient information, clearer details and less distortion.

[0152] The multimodal frequency domain cross-image fusion method described in this embodiment of the invention has at least the following advantages:

[0153] 1) This invention is a multimodal frequency domain cross-image fusion method. By extracting the high and low frequency features of infrared images and visible light images, a fused image with richer information is produced after fusion.

[0154] 2) The designed efficient cross-modal cross-fusion network calculates high and low frequency feature results at different levels, and then combines cross-layer connections and cross-connections to complete the feature fusion process, thereby improving information transmission efficiency and fusion effect;

[0155] 3) A loss function calculation method is designed to improve the multimodal image fusion effect, thereby enhancing the detail and texture representation of the fused image;

[0156] 4) This invention validates the method on the MSRS dataset and achieves better results than state-of-the-art methods.

[0157] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0158] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0159] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0160] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0161] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0163] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0166] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal frequency domain cross-image fusion method, characterized in that, The method includes: S1. Construct a multimodal image fusion network; wherein, the multimodal image fusion network includes: an encoder network, a cross-modal cross-fusion network, and a decoder; the execution steps of the multimodal image fusion network include: An encoder network is used to extract different frequency domain features from the input modal images; A cross-modal cross-fusion network is used to fuse the frequency domain features of images from different modalities through cross-layer connections and cross-connections; The fused features are input into the decoder to decode the fused features and output the fused image; S2. Train the constructed multimodal image fusion network using multimodal images and output a fused image containing features of each modality; The cross-modal fusion network includes high-frequency and low-frequency branches and channel attention modules, which are used to fuse the frequency domain features extracted by the encoder. Each high-frequency and low-frequency branch consists of three layer feature extraction modules representing different depths. The layer feature extraction process of the layer feature extraction module is represented as follows: ； in, , These represent the low-frequency fundamental features of infrared and visible light images, respectively. , These represent the high-frequency detail features of infrared and visible light images, respectively. Representative layer feature extraction is used to extract high and low frequency features at different levels. Depending on the number of processing iterations, different stages of features from shallow to deep can be generated. Indicates the characteristics of each stage of low frequency; Indicates the characteristics of each stage of high frequency; The method of using a cross-modal cross-fusion network to fuse frequency domain features of different modal images through cross-layer and cross-connection includes: For the frequency domain features of the obtained images of different modalities, the layer feature extraction module is used to extract features from different stages from shallow to deep. In the process of processing stage features, a cross-layer connection method is adopted to link features at different levels and pass shallow features to various deep features. To achieve frequency domain feature fusion based on the requirements for processing different frequency domain feature information, a cross-connection approach is adopted to extract primary features. This is then linked with the cross-layer processing results to obtain heteromodal co-frequency fusion features modified with hetero-frequency domain features. The processes of cross-layer connections and cross-connections are represented as follows: ； in, and These represent the extraction processes of primary basic features and primary detailed features, respectively. The heteromodal co-frequency fusion features are fed into the channel attention module for fusion to obtain the final fusion feature map.

2. The multimodal frequency domain cross-image fusion method according to claim 1, characterized in that, The encoder network includes: a shared feature encoder, a basic transformation network encoder, and a detail convolutional neural network encoder; Modal images include: infrared images and visible light images; The step of using an encoder network to extract different frequency domain features from the input modal images includes: Cross-modal shallow feature extraction is achieved by using a shared feature encoder to obtain shallow features from infrared and visible light images. ;in, , These represent the shallow features of infrared and visible light images, respectively. The shallow features are further extracted using a basic transformation network encoder to obtain low-frequency basic features. ;in, , These represent the low-frequency fundamental features of infrared and visible light images, respectively. A detail convolutional neural network encoder is used to further extract features from shallow features to obtain high-frequency detail features. ;in, , These represent the high-frequency detail features of infrared and visible light images, respectively.

3. The multimodal frequency domain cross-image fusion method according to claim 2, characterized in that, The shared feature encoder includes: two converter layers based on recovery converter blocks; Each shared feature encoder consists of two stacked transformer layers based on recovery transformer blocks, with the parameters of the two shared feature encoders being independent of each other; The formula for a shared feature encoder is expressed as: ； in, This represents the process of a shared feature encoder extracting features. These represent the input pairs of infrared and visible light images, respectively.

4. The multimodal frequency domain cross-image fusion method according to claim 3, characterized in that, The basic transformation network encoder includes a transformation network module based on a lightweight converter for extracting low-frequency basic features from shallow features; Each transformation network module includes: 2 Norm layers, 1 basic Attention layer, and 1 MLP layer; where Norm is normalization, Attention is attention, and MLP is multilayer perceptron. The formula for the basic transformation network encoder is expressed as: ； in, This represents the process of feature extraction by the basic transformation network encoder.

5. The multimodal frequency domain cross-image fusion method according to claim 4, characterized in that, The detail convolutional neural network encoder includes three INNs for extracting high-frequency detail features from shallow features; wherein, the INN is an invertible neural network; In each processing step of the reversible neural network, the transformation process is as follows: ； in, This represents the Hadamard product operation. Indicates the first The first to the first of the input features of the reversible layer One channel, , C represents the number of reversible layers and the number of input feature channels. This is a channel connection operation. It is an arbitrary mapping function. , Different mapping function identifiers are used to output the feature extraction results of the detail convolutional neural network encoder after the reversible neural network is processed three times, which are high-frequency detail features. The formula for the entire detail convolutional neural network encoder is expressed as follows: ； in, This describes the process of feature extraction by a detail convolutional neural network encoder.

6. The multimodal frequency domain cross-image fusion method according to claim 1, characterized in that, A hybrid gradient loss, consisting of a surface layer and a gradient layer, is added during the training process to ensure that the multimodal image fusion network can learn more complete fusion gradient information during training.

7. The multimodal frequency domain cross-image fusion method according to claim 6, characterized in that, The hybrid gradient loss converts the visible light image into the Lab color space and extracts the L brightness channel as the visible light processing channel. The visible light image L brightness channel and the infrared image channel are mixed. This results in surface layer loss. Simultaneous use The operator extracts gradients from the L-channel of the visible light image and the infrared image channel, and then performs gradient blending on the extracted gradient maps. This produces gradient layer loss results. The obtained surface layer loss result and gradient layer loss result are summed according to their weights to obtain the final hybrid gradient loss. , represented as: ； in, These represent the L-luminance channel of the visible light image, the infrared image, and the fused image, respectively. These represent the gradient maps of the L-channel brightness of the visible light image, the infrared image, and the fused image, respectively. The summation weights are the surface layer loss and the gradient layer loss.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images