Multi-modal image fusion method and device based on frequency domain perception, medium and product

Abstract

The invention discloses a multi-modal image fusion method and device based on frequency domain perception, a medium and a product, and relates to the field of image fusion, and the method comprises the steps: obtaining a to-be-fused target infrared image and a to-be-fused target visible light image; according to the to-be-fused target infrared image and the to-be-fused target visible light image, determining a reconstructed image and a fused image of the target by using a multi-modal image fusion model; wherein the multi-modal image fusion model is obtained by training a convolutional neural network by using a training data set; the training data set comprises an infrared image and a visible light image of the target for training and a corresponding actual reconstruction image and an actual fusion image. According to the method, multiple frequency domain components are obtained by using discrete wavelet transform, and the components are fused by using a frequency domain fusion strategy so as to generate a high-quality fusion result. The quality of the fused image is improved.

Description

Technical Field

[0001] The present application relates to the field of image fusion, and in particular to a multimodal image fusion method, device, medium and product based on frequency domain perception. Background Art

[0002] In recent years, multimodal image fusion, such as infrared image and visible light image fusion, has attracted extensive attention from researchers in computer vision. It provides complementary scene descriptions by integrating the advantages of different visual modes. Specifically, thermal radiation-sensitive objects captured from infrared images can break the limitations of harsh conditions (e.g., lighting and smoke), but these images are affected by low resolution. Visible light images have rich texture details but become fragile with changes in illumination. Therefore, the goal of fusion of infrared images and visible light images is to make the fused image retain the thermal radiation information in the infrared image and the detailed texture information in the visible light image. The fused image can avoid the shortcomings of visible light images being sensitive to lighting conditions and infrared images being noisy and having low resolution, thereby achieving better performance than using a single mode alone in actual downstream tasks. Downstream recognition tasks, such as multimodal saliency detection, object detection, and semantic segmentation can obtain clearer scene and object representations from the fused images of infrared and visible light images. In recent years, although some preliminary results have been achieved in multimodal image fusion research, the following problems and challenges still exist based on the existing literature reports:

[0003] (1) Limited fusion quality: Most of the current fusion methods are based on the spatial domain, ignoring the impact of frequency domain information on the fusion results, and cannot provide users with a good fusion visual experience. In addition, most fusion methods are based on unsupervised methods, and the fusion quality is limited.

[0004] (2) Lack of multi-task joint datasets: Current infrared image and visible light image datasets are limited, with few samples and lack of real labels. Summary of the invention

[0005] The purpose of this application is to provide a multimodal image fusion method, device, medium and product based on frequency domain perception to improve the quality of fused images.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a multimodal image fusion method based on frequency domain perception, comprising:

[0008] Acquire a target infrared image to be fused and a target visible light image to be fused;

[0009] According to the target infrared image to be fused and the target visible light image to be fused, a reconstructed image and a fused image of the target are determined using a multimodal image fusion model; wherein the multimodal image fusion model is obtained by training a convolutional neural network using a training data set; the training data set includes an infrared image and a visible light image of the training target and the corresponding actual reconstructed image and actual fused image.

[0010] Optionally, the convolutional neural network is trained using a training data set, specifically including:

[0011] Get the training dataset;

[0012] Build a convolutional neural network;

[0013] Taking the infrared image and the visible light image of the training target as input and the corresponding actual reconstructed image and the actual fused image as output, the convolutional neural network is trained for multiple rounds to obtain a multimodal image fusion model.

[0014] Optionally, taking the infrared image and the visible light image of the training target as input and taking the corresponding actual reconstructed image and the actual fused image as output, performing any round of training on the convolutional neural network specifically includes:

[0015] Inputting the infrared image and the visible light image of the training target into the convolutional neural network, wherein the first convolution block extracts a first feature of the infrared image and a second feature of the visible light image;

[0016] Based on the first feature and the second feature, using discrete wavelet transform, determine the low-frequency feature and the high-frequency feature of the infrared image and the low-frequency feature and the high-frequency feature of the visible light image;

[0017] Determine the fused low-frequency features by using L1 norm and softmax operation according to the low-frequency features of the infrared image and the low-frequency features of the visible light image;

[0018] Determine the fused high-frequency features by using a high-frequency difference fusion module according to the high-frequency features of the infrared image and the high-frequency features of the visible light image;

[0019] Determine a predicted fused image and a predicted reconstructed image of a training target by using an inverse discrete wavelet transform according to the fused low-frequency features and the fused high-frequency features;

[0020] Determine a loss function value by using a frequency loss function according to the predicted fused image, the predicted reconstructed image, the actual fused image and the actual reconstructed image;

[0021] The loss function value is used to adjust the model parameters of the convolutional neural network for the next round of training.

[0022] Optionally, according to the low-frequency features of the infrared image and the low-frequency features of the visible light image, using L1 norm and softmax operation, determining the fused low-frequency features specifically includes:

[0023] Performing L1 norm on the low-frequency features of the infrared image in the channel dimension to obtain a first activity level map;

[0024] Performing L1 norm on the low-frequency features of the visible light image in the channel dimension to obtain a second activity level map;

[0025] Generate a first feature-level weight map using a softmax operation according to the first activity level map;

[0026] Generating a second feature-level weight map using a softmax operation according to the second activity level map;

[0027] A fused low-frequency feature is determined according to the first feature-level weight map, the low-frequency feature of the infrared image, the second feature-level weight map, and the low-frequency feature of the visible light image.

[0028] Optionally, according to the high-frequency features of the infrared image and the high-frequency features of the visible light image, a high-frequency difference fusion module is used to determine the fused high-frequency features, specifically including:

[0029] Performing element-wise subtraction operation on the high-frequency features of the infrared image and the high-frequency features of the visible light image to obtain a difference feature map;

[0030] Performing a global average pooling operation on the difference feature map to obtain a difference feature vector;

[0031] According to the difference feature vector, a channel attention vector is obtained using a fully connected layer and a Sigmoid activation function;

[0032] Obtaining an enhanced first difference feature by using a channel multiplication operation according to the channel attention vector and the high-frequency features of the infrared image;

[0033] Obtaining an enhanced second difference feature by using a channel multiplication operation according to the channel attention vector and the high-frequency feature of the visible light image;

[0034] According to the high-frequency features of the infrared image, a first weight mapping feature is determined using a first lightweight sub-network and a Sigmoid activation function; the lightweight sub-network includes a second convolution block, a third convolution block and a first 3×3 convolution layer;

[0035] Determining a second weight mapping feature according to the high-frequency features of the visible light image using a second lightweight subnetwork and a Sigmoid activation function;

[0036] According to the enhanced first difference feature, the enhanced second difference feature, the first weight mapping feature and the second weight mapping feature, a fused high-frequency feature is determined by using element addition and a second 3×3 convolutional layer.

[0037] Optionally, determining a fused high-frequency feature according to the enhanced first difference feature, the enhanced second difference feature, the first weight mapping feature, and the second weight mapping feature by using element addition and a second 3×3 convolution layer specifically includes:

[0038] Using the formula Determine the high-frequency features after fusion; among them, is the high-frequency feature after fusion; The first difference feature is enhanced; The second difference feature is enhanced; is the first weight mapping feature; is the second weight mapping feature; is the sum of the enhanced first difference feature and the first weight mapping feature; is the sum of the enhanced second difference feature and the second weight mapping feature; is the element and operation; Conv 3×3 It is the second 3×3 convolutional layer.

[0039] Optionally, the frequency loss function is:

[0040]

[0041]

[0042] L aux =||∑HF F -∑HF inf ||1+||∑HF F -∑HF vis ||1,HF∈{LH,HL,HH};

[0043] in, is the frequency L1 loss; L aux is the high frequency alignment detail loss; Fre F is the frequency domain component of the fused image; Fre inf is the frequency domain component of the infrared image; Fre vis is the frequency domain component of the visible light image; HF F is the high frequency component of the fused image; HF infHF is the high frequency component of the infrared image; vis is the high-frequency component of the visible light image.

[0044] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned multimodal image fusion methods based on frequency domain perception.

[0045] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned multimodal image fusion methods based on frequency domain perception.

[0046] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned multimodal image fusion methods based on frequency domain perception.

[0047] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0048] The present application provides a multimodal image fusion method, device, medium and product based on frequency domain perception, by obtaining the target infrared image to be fused and the target visible light image to be fused; according to the target infrared image to be fused and the target visible light image to be fused, the multimodal image fusion model is used to determine the reconstructed image and the fused image of the target; wherein the multimodal image fusion model is obtained by training the convolutional neural network using a training data set; the training data set includes the infrared image and the visible light image of the training target and the corresponding actual reconstructed image and the actual fused image. The present application extracts the feature representation of the image through a wavelet encoder, and uses a discrete wavelet transform to obtain multiple frequency domain components. Next, these components are fused using a frequency domain fusion strategy, and further, multiple frequency domain components are sent to a wavelet decoder to obtain a fused image and a reconstructed image. Finally, the model is jointly optimized by frequency domain loss and self-supervised reconstruction loss to drive the model to generate high-quality fusion results. The present application improves the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1A schematic diagram of a flow chart of a multimodal image fusion method based on frequency domain perception provided in one embodiment of the present application;

[0051] Figure 2 This is a flowchart of the multimodal image fusion method based on frequency domain perception in practical application;

[0052] Figure 3 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0055] Most of the existing multimodal image fusion algorithms based on deep learning mainly focus on learning feature representation in the spatial domain, ignoring the impact of frequency information on the fusion results, and cannot fully retain significant structures and important texture details. The core of this application is based on the following observation: after the image is converted in the frequency domain, the low-frequency component of the image mainly contains significant structures, while the texture detail information is located in the high-frequency component. Based on this observation, this application proposes a frequency-domain-aware multimodal image fusion method. The frequency domain jump connection enables the fused image to retain more low-frequency contrast and high-frequency detail information. In addition, this application proposes to reduce the loss of frequency domain information of the source image by constraining the low-frequency and high-frequency components between the fused image and the source image. Specifically, the infrared image, the visible light image and the fused image are decomposed into low / high frequencies using discrete wavelet transform to obtain the low / high frequency components of the infrared image, the visible light image and the fused image. Subsequently, the L1 norm is used to calculate the distance between the low / high frequency components of the fused image and the low / high frequency components of the above infrared image and visible light image. Although the frequency L1 loss improves the frequency perception ability of the model, it can only synthesize arbitrary frequency signals, and the fused image cannot maintain the optimal high-frequency detail distribution. To this end, this application further proposes an auxiliary high-frequency detail alignment loss function.

[0056] It is also a key to consider how to effectively retain the important information of the source image during the feature extraction process. This application introduces image reconstruction as an auxiliary task to model image fusion (main task) and image reconstruction (auxiliary task) in a joint optimization manner. The network is encouraged to learn a richer and more comprehensive feature representation of unlabeled input images through self-supervision.

[0057] The low-frequency weight fusion strategy makes the fusion result have the target high contrast. The high-frequency difference fusion module aims to mine and integrate the most discriminative common and difference information in the high-frequency features of multimodal images, generate features with more detailed information, and make the fusion result have rich texture details. First, the high-frequency features perform element-wise subtraction operations to obtain difference features. Then global average pooling is performed to generate global statistical information under different views, and the difference feature vector is sent to the fully connected layer and the sigmoid activation function to generate the channel attention vector. Next, the difference attention vector is multiplied with the input feature in a channel multiplication manner to obtain the enhanced difference feature. Further, a lightweight subnetwork contains two convolution blocks, a convolution layer (kernel size is 3) and a sigmoid function to obtain the high-frequency component weight map and multiply it with the input. Finally, the cross-modal high-frequency fusion feature is generated by performing element-wise addition and a 3×3 convolution layer with the enhanced difference feature.

[0058] In an exemplary embodiment, Figure 1 As shown, a multimodal image fusion method based on frequency domain perception is provided, comprising the following steps:

[0059] S1: Acquire a target infrared image to be fused and a target visible light image to be fused.

[0060] S2: Determine the reconstructed image and the fused image of the target using a multimodal image fusion model according to the target infrared image to be fused and the target visible light image to be fused; wherein the multimodal image fusion model is obtained by training a convolutional neural network using a training data set; the training data set includes the infrared image and the visible light image of the training target and the corresponding actual reconstructed image and the actual fused image.

[0061] As an optional implementation, the convolutional neural network is trained using a training data set, specifically including:

[0062] Get the training dataset.

[0063] Build a convolutional neural network.

[0064] Taking the infrared image and visible light image of the training target as input and the corresponding actual reconstructed image and actual fused image as output, the convolutional neural network is trained for multiple rounds to obtain a multimodal image fusion model. Specifically, it includes:

[0065] S201: Inputting the infrared image and the visible light image of the training target into the convolutional neural network, and extracting the first feature of the infrared image and the second feature of the visible light image by a first convolution block.

[0066] S202: Based on the first feature and the second feature, using discrete wavelet transform, determine the low-frequency feature and high-frequency feature of the infrared image and the low-frequency feature and high-frequency feature of the visible light image.

[0067] In practical applications, the image fusion process is as follows: Figure 2 shown.

[0068] Step 1: Based on the input infrared image to be fused and the visible light image to be fused, the convolutional neural network is used to extract the feature representation of the input image, and then the discrete wavelet transform is performed on these feature representations to obtain four different frequency components (LL, LH, HL and HH), where LL represents the low-frequency component, and LH, HL and HH represent the high-frequency component. Specifically, the convolution block extracts the feature representation of the input image, and then performs a discrete wavelet transform to obtain multiple frequency components. The discrete wavelet transform contains 4 kernels, {LL T ,LH T ,HL T ,HH T}, defined as follows:

[0069]

[0070] Among them, L represents a low-pass filter, capturing low-frequency signals including image contours and surfaces, and H represents a high-pass filter, focusing on detail information such as vertical, horizontal, and diagonal edges. For simplicity, this application defines the frequency components of the four filter kernel decompositions as: i=1, 2, 3, k=1, 2, 3, 4, im=inf, vis, fused image, where Inf represents infrared image, vis represents visible light image, and fused represents fused image. Here, taking visible light image as an example, the visible light image is fed into the encoder (the first convolutional block) to obtain the feature Then the features Decompose into frequency components is a low frequency feature, The decomposed frequency domain components have three applications: 1) they are sent to step 2 for low-frequency weight fusion and high-frequency difference fusion strategy for feature fusion; 2) they are directly fed to step 3 to reconstruct the input by connecting low-frequency and high-frequency frequency hopping connections. 3) In step 1, the low-frequency component is element-wise added to the feature of the current convolution block to perform an element-wise addition operation on the low-frequency feature. and Features Element-wise addition is used as input to the next convolutional block.

[0071]

[0072] E represents the convolution block, pool represents the pooling operation, Indicates element addition. i indicates the sequence number of the convolution block.

[0073] S203: Determine fused low-frequency features according to the low-frequency features of the infrared image and the low-frequency features of the visible light image by using L1 norm and softmax operation.

[0074] As an optional implementation, S203 specifically includes:

[0075] An L1 norm is performed on the low-frequency features of the infrared image in the channel dimension to obtain a first activity level map.

[0076] An L1 norm is performed on the low-frequency features of the visible light image in the channel dimension to obtain a second activity level map.

[0077] A first feature-level weight map is generated using a softmax operation according to the first activity level map.

[0078] According to the second activity level map, a second feature level weight map is generated using a softmax operation.

[0079] A fused low-frequency feature is determined according to the first feature-level weight map, the low-frequency feature of the infrared image, the second feature-level weight map, and the low-frequency feature of the visible light image.

[0080] Step 2: The low-frequency weight fusion strategy and the high-frequency difference fusion strategy fuse multiple frequency domain components from step 1 respectively. In the low-frequency weight fusion strategy, considering that the extracted low-frequency features mainly contain the saliency information of the source image, this application uses the L1 norm combined with the softmax operation to integrate the low-frequency saliency features to calculate the activity level weighted map. The activity level weighted map is obtained by performing the L1 norm on the channel dimension of the low-frequency features decomposed by wavelet, where higher response values ​​represent significant areas in the low-frequency components. Subsequently, the softmax operation is used in the activity level weighted map to generate a feature-level weight map. Finally, the feature-level weight map is multiplied with the input features to obtain a fused low-frequency feature map. The fused low-frequency features are sent to step 3 for inverse transformation to generate a fusion result map.

[0081] S204: Determine fused high-frequency features according to the high-frequency features of the infrared image and the high-frequency features of the visible light image using a high-frequency difference fusion module.

[0082] As an optional implementation, S204 specifically includes:

[0083] An element-wise subtraction operation is performed on the high-frequency features of the infrared image and the high-frequency features of the visible light image to obtain a difference feature map.

[0084] A global average pooling operation is performed on the difference feature map to obtain a difference feature vector.

[0085] According to the difference feature vector, a channel attention vector is obtained using a fully connected layer and a Sigmoid activation function.

[0086] According to the channel attention vector and the high-frequency features of the infrared image, a channel multiplication operation is utilized to obtain an enhanced first difference feature.

[0087] According to the channel attention vector and the high-frequency features of the visible light image, an enhanced second difference feature is obtained by utilizing a channel multiplication operation.

[0088] According to the high-frequency features of the infrared image, a first weight mapping feature is determined using a first lightweight sub-network and a Sigmoid activation function; the lightweight sub-network includes a second convolution block, a third convolution block and a first 3×3 convolution layer.

[0089] According to the high-frequency features of the visible light image, a second weight mapping feature is determined using a second lightweight sub-network and a Sigmoid activation function.

[0090] According to the enhanced first difference feature, the enhanced second difference feature, the first weight mapping feature and the second weight mapping feature, the fused high-frequency feature is determined by using element addition and a second 3×3 convolution layer. Specifically, it includes:

[0091] Using the formula Determine the high-frequency features after fusion; among them, is the high-frequency feature after fusion; The first difference feature is enhanced; The second difference feature is enhanced; is the first weight mapping feature; is the second weight mapping feature; is the sum of the enhanced first difference feature and the first weight mapping feature; is the sum of the enhanced second difference feature and the second weight mapping feature; is the element and operation; Conv 3×3 It is the second 3×3 convolutional layer.

[0092] In practical applications, each high-frequency component corresponds to detail information, and effectively retaining the high-frequency components can obtain a fused image with rich texture.

[0093] However, directly using high-frequency information may lead to hidden problems, such as noise. In this embodiment, an attempt is made to generate a more reliable weight map through their commonalities and differences to merge cross-modal information. In order to integrate cross-modal detail information, this application designs a high-frequency fusion strategy-a high-frequency difference fusion module. The high-frequency difference fusion module proposed in this application aims to mine and combine the most discriminative commonalities and differences in infrared and visible light high-frequency features to generate features with more detailed information. Taking LH as an example, given two input features and is a high-frequency feature of infrared images. is a high-frequency feature of the visible light image. First, this application uses element subtraction to generate and The difference feature map between them is then global average pooling is performed to obtain global statistics under different views. The difference feature vector is then fed to the fully connected layer and the Sigmoid activation function to generate a C×1×1 channel attention vector Through the above operations, The contribution of channel differential features to the fusion result can be directly regulated. Therefore, a higher value indicates that channel C plays an important role. In other words, reveals the degree of difference in the information contained in each channel. For example, if channel m has a high feature variance, the corresponding It will also be relatively high.

[0094]

[0095] Among them, Sig represents the Sigmoid activation function, Fc represents the fully connected layer, and GAP represents the global average pooling. represents element-wise subtraction. Finally, the channel attention vector Multiply the input features by channels and Multiply them to get enhanced difference features and

[0096]

[0097] In addition, a weight map is obtained through a lightweight subnetwork containing two convolution blocks and one convolution layer (with a kernel size of 3) and a sigmoid function, and multiplied with the input to generate a weight map feature. and

[0098]

[0099] Next, and The final cross-modal fusion features (the fused high-frequency features) are generated by element addition and 3×3 convolutional layers. This process can be described as follows:

[0100]

[0101] Through the above operations, the complementary detail information of different high-frequency components is integrated into the fused features. Finally, the fused high-frequency features are sent to step 3 through skip connections, and after inverse transformation, the fusion results are output through the convolutional neural network.

[0102] S205: Determine a predicted fused image and a predicted reconstructed image of the training target by using an inverse discrete wavelet transform according to the fused low-frequency features and the fused high-frequency features.

[0103] S206: Determine a loss function value using a frequency loss function according to the predicted fused image, the predicted reconstructed image, the actual fused image and the actual reconstructed image.

[0104] S207: Using the loss function value to adjust the model parameters of the convolutional neural network for the next round of training.

[0105] Step 3: Receive the frequency feature representation from step 1 and step 2, and step 3 obtains the reconstructed image and the fused image through inverse transformation and convolutional neural network. In order to reduce the loss of important features (i.e., saliency and details) in the fusion and reconstruction results, this application directly feeds the frequency component ( i=1,2,3,k=1,2,3,4) and the fused frequency signal generated in step 2 They are fed into step 3 through frequency skip connections respectively. This will prompt step 3 to generate an image containing more information, thereby improving the fusion and reconstruction performance. Then, the inverse discrete wavelet transform is used to reconstruct these frequency features and restore them to spatial features. More specifically, the present application performs a transposed convolution TransConv on each frequency component (low frequency and high frequency), and then adds the four transposed convolution features to accurately restore the frequency signal to the original feature, and the formula is:

[0106]

[0107] The whole process can be defined as:

[0108]

[0109] After step 3 is completed, the fusion result and the reconstruction result are used to train the convolutional neural network. The present application innovatively proposes a frequency L1 loss. The frequency L1 loss is responsible for constraining multiple frequency components. Specifically, the present application introduces a discrete wavelet transform to decompose the fused image and the source image into a low-frequency and three high-frequency components at the first level. Then, the L1 norm loss is used in multiple frequency subbands, expressed as:

[0110]

[0111] Although the proposed frequency L1 loss significantly improves the frequency perception ability of the model, it can only synthesize arbitrary frequency signals, and the fused image cannot maintain the optimal high-frequency detail distribution. To this end, this application further introduces an auxiliary high-frequency alignment detail loss, which is expressed as:

[0112] L aux =||∑HF F -∑HF inf ||1+||∑HF F -∑HF vis ||1,HF∈{LH,HL,HH} (14)

[0113] Here, HF represents three high frequency components for performing element-wise addition operations.

[0114] Finally, the frequency loss function consists of the frequency L1 loss and the high-frequency alignment auxiliary loss, and the formula is as follows:

[0115]

[0116] in, is the frequency L1 loss; L aux is the high frequency alignment detail loss; Fre F is the frequency domain component of the fused image; Fre inf is the frequency domain component of the infrared image; Fre vis is the frequency domain component of the visible light image; HF F is the high frequency component of the fused image; HF inf HF is the high frequency component of the infrared image; vis is the high-frequency component of the visible light image.

[0117] The L1 norm constrains the model by calculating the absolute value difference between the frequency components of the fused image and the source image. In the frequency domain loss, the L1 norm helps to preserve the high-frequency details and low-frequency contrast of the source image and reduces the sensitivity of the model to outliers because it does not over-penalize errors. Compared with the L1 norm, the L2 norm is more sensitive to outliers because it amplifies the errors and may cause the model to lose important information. The goal of this application is to preserve the important features in the source image. Therefore, applying the L1 norm in the frequency domain can better preserve the contrast and texture information of the fused image.

[0118] In order to further encourage the model to better learn the features of the source image, the present application adopts self-supervised reconstruction loss to measure the difference between the reconstructed image and the input image, and optimizes the model through back propagation. The unlabeled input image is used as the true label for training to obtain supervision, which is crucial for learning the representation of unlabeled data. In other words, image reconstruction is introduced as an auxiliary task to achieve better performance in feature representation. Of course, given the poor quality of image reconstruction, the self-supervised loss will be higher, indicating that the network has not effectively learned the features required to accurately reconstruct the input. In order to enhance the robustness of the reconstruction process, the present application adopts frequency skip connections to reduce the loss of intermediate information, so that the reconstructed image contains more low-frequency structural information and high-frequency detail information, thereby improving the quality of the reconstructed image. In addition, the reconstruction difference is calculated by using the L1 norm, allowing for smaller reconstruction errors. In general, through these two strategies, the present application can alleviate the problem of inaccurate image reconstruction that may result. Therefore, the model can learn more comprehensive features of the source image. The reconstruction results are defined as and The self-supervised reconstruction loss is defined as:

[0119]

[0120] After the above steps, given the acquired infrared image and visible light image to be fused, the present application can generate a fused image with high contrast and rich texture details of the foreground target, which can be used for subsequent applications. Such as target tracking: the fused image can identify and track the target under various weather and lighting conditions, enhancing the target detection and recognition capabilities; intelligent transportation: fused images are used for pedestrian detection, vehicle identification and road obstacle classification to improve traffic management and safety; security monitoring: in dark or complex environments, fused images can improve the accuracy of face recognition and behavior analysis. Images with high contrast and rich texture details help improve target detection and recognition capabilities in complex backgrounds and reduce false alarms and missed reports. Retaining more detailed information, improving the visual effect and information content of the image, will help with more accurate analysis and decision-making.

[0121] The present application has the following advantages over the related art:

[0122] The multimodal fusion method based on frequency domain perception can largely retain the thermal radiation information and detailed texture information of the source image. The fusion performance is attributed to the model starting from the frequency domain. The fusion strategy proposed based on frequency characteristics can effectively fuse the complementary information of the source image, and the frequency domain jump connection reduces the loss of intermediate information. The frequency domain loss retains the low-frequency contrast and high-frequency detail information of the source image as much as possible.

[0123] High model operation efficiency: This application jointly optimizes the wavelet encoder by sharing the weights of the fusion branch and the reconstruction branch, and the reconstruction branch is only used in the training phase, which does not increase the reasoning complexity of the model. Secondly, the channel dimensions of the wavelet encoder and the wavelet decoder are uniformly set to 16, which greatly reduces the amount of calculation. In addition, the high-frequency difference fusion module also uses a lightweight structure for feature fusion. These three factors may help to achieve the fastest running time.

[0124] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal image fusion method based on frequency domain perception is implemented.

[0125] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the above-mentioned multimodal image fusion method based on frequency domain perception is implemented.

[0126] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the multimodal image fusion method based on frequency domain perception is implemented.

[0127] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned multimodal image fusion method based on frequency domain perception when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0130] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multimodal image fusion method based on frequency domain perception, characterized in that: include: Acquire a target infrared image to be fused and a target visible light image to be fused; According to the target infrared image to be fused and the target visible light image to be fused, a reconstructed image and a fused image of the target are determined using a multimodal image fusion model; wherein the multimodal image fusion model is obtained by training a convolutional neural network using a training data set; the training data set includes an infrared image and a visible light image of the training target and the corresponding actual reconstructed image and actual fused image.

2. The multimodal image fusion method based on frequency domain perception according to claim 1, characterized in that: The convolutional neural network is trained using the training data set, including: Get the training dataset; Build a convolutional neural network; Taking the infrared image and the visible light image of the training target as input and the corresponding actual reconstructed image and the actual fused image as output, the convolutional neural network is trained for multiple rounds to obtain a multimodal image fusion model.

3. The multimodal image fusion method based on frequency domain perception according to claim 1, characterized in that: Taking the infrared image and the visible light image of the training target as input and the corresponding actual reconstructed image and the actual fused image as output, the convolutional neural network is trained in any round, specifically including: Inputting the infrared image and the visible light image of the training target into the convolutional neural network, wherein the first convolution block extracts a first feature of the infrared image and a second feature of the visible light image; Based on the first feature and the second feature, using discrete wavelet transform, determine the low-frequency feature and the high-frequency feature of the infrared image and the low-frequency feature and the high-frequency feature of the visible light image; Determine the fused low-frequency features by using L1 norm and softmax operation according to the low-frequency features of the infrared image and the low-frequency features of the visible light image; Determine the fused high-frequency features by using a high-frequency difference fusion module according to the high-frequency features of the infrared image and the high-frequency features of the visible light image; Determine a predicted fused image and a predicted reconstructed image of a training target by using an inverse discrete wavelet transform according to the fused low-frequency features and the fused high-frequency features; Determine a loss function value by using a frequency loss function according to the predicted fused image, the predicted reconstructed image, the actual fused image and the actual reconstructed image; The loss function value is used to adjust the model parameters of the convolutional neural network for the next round of training.

4. The multimodal image fusion method based on frequency domain perception according to claim 3 is characterized in that: According to the low-frequency features of the infrared image and the low-frequency features of the visible light image, the fused low-frequency features are determined by using L1 norm and softmax operation, specifically including: Performing L1 norm on the low-frequency features of the infrared image in the channel dimension to obtain a first activity level map; Performing L1 norm on the low-frequency features of the visible light image in the channel dimension to obtain a second activity level map; Generate a first feature-level weight map using a softmax operation according to the first activity level map; Generating a second feature-level weight map using a softmax operation according to the second activity level map; A fused low-frequency feature is determined according to the first feature-level weight map, the low-frequency feature of the infrared image, the second feature-level weight map, and the low-frequency feature of the visible light image.

5. The multimodal image fusion method based on frequency domain perception according to claim 3, characterized in that: According to the high-frequency features of the infrared image and the high-frequency features of the visible light image, a high-frequency difference fusion module is used to determine the fused high-frequency features, specifically including: Performing element-wise subtraction operation on the high-frequency features of the infrared image and the high-frequency features of the visible light image to obtain a difference feature map; Performing a global average pooling operation on the difference feature map to obtain a difference feature vector; According to the difference feature vector, a channel attention vector is obtained using a fully connected layer and a Sigmoid activation function; Obtaining an enhanced first difference feature by using a channel multiplication operation according to the channel attention vector and the high-frequency features of the infrared image; Obtaining an enhanced second difference feature by using a channel multiplication operation according to the channel attention vector and the high-frequency feature of the visible light image; According to the high-frequency features of the infrared image, a first weight mapping feature is determined using a first lightweight sub-network and a Sigmoid activation function; the lightweight sub-network includes a second convolution block, a third convolution block and a first 3×3 convolution layer; Determining a second weight mapping feature according to the high-frequency features of the visible light image using a second lightweight subnetwork and a Sigmoid activation function; According to the enhanced first difference feature, the enhanced second difference feature, the first weight mapping feature and the second weight mapping feature, a fused high-frequency feature is determined by using element addition and a second 3×3 convolutional layer.

6. The multimodal image fusion method based on frequency domain perception according to claim 5, characterized in that: Determining a fused high-frequency feature according to the enhanced first difference feature, the enhanced second difference feature, the first weight mapping feature, and the second weight mapping feature by using element addition and a second 3×3 convolution layer, specifically includes: Using the formula Determine the high-frequency features after fusion; among them, is the high-frequency feature after fusion; The first difference feature is enhanced; The second difference feature is enhanced; is the first weight mapping feature; is the second weight mapping feature; is the sum of the enhanced first difference feature and the first weight mapping feature; is the sum of the enhanced second difference feature and the second weight mapping feature; is the element and operation; Conv 3×3 It is the second 3×3 convolutional layer.

7. The multimodal image fusion method based on frequency domain perception according to claim 3, characterized in that: The frequency loss function is: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> aux <h2 style=";text-align:left;direction:ltr"> =||∑HF<h2 style=";text-align:left;direction:ltr"> F <h2 style=";text-align:left;direction:ltr"> -∑HF<h2 style=";text-align:left;direction:ltr"> inf <h2 style=";text-align:left;direction:ltr"> ||1+||∑HF<h2 style=";text-align:left;direction:ltr"> F <h2 style=";text-align:left;direction:ltr"> -∑HF<h2 style=";text-align:left;direction:ltr"> vis <h2 style=";text-align:left;direction:ltr"> ||1,HF∈{LH,HL,HH}; in, is the frequency L1 loss; L aux is the high frequency alignment detail loss; Fre F is the frequency domain component of the fused image; Fre inf is the frequency domain component of the infrared image; Fre vis is the frequency domain component of the visible light image; HF F is the high frequency component of the fused image; HF inf It is the high frequency component of infrared image; HF vis is the high-frequency component of the visible light image.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal image fusion method based on frequency domain perception as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal image fusion method based on frequency domain perception described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal image fusion method based on frequency domain perception described in any one of claims 1 to 7 is implemented.

Images