Multimodal image fusion method and system based on joint evaluation of feature uncertainty

Abstract

The present invention discloses a multimodal image fusion method and system based on joint feature uncertainty assessment. A pair of input images are spliced together to obtain a spliced image. A diffusion model is input to extract S features of different scales from N time steps. The resulting N×S features are input into a fully connected neural network model to obtain the feature potential uncertainty weights of the features at different time steps. Features of the same scale at different time steps are weighted and spliced and compressed according to the feature potential uncertainty weights to obtain S preliminary fused features of different scales. The system then upsamples layer by layer based on the scale and uses an attention module to adjust the number of channels to obtain the final fused features. The decoder then obtains the fused image. The present invention aims to address the issues of insufficient feature extraction and unstable fusion results caused by modal differences and feature potential uncertainty in image fusion, and to overcome the issues of missing details and unstable performance in fusion results caused by insufficient feature extraction and unreasonable fusion.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal image fusion method and system based on joint evaluation of feature uncertainty. Background Art

[0002] With the advancement of science and technology, image fusion, as a key research area in image processing, has been applied to all aspects of life. Image fusion effectively integrates image information from different sensors, modalities, or time periods. The fusion of visible light and thermal infrared images combines the advantages of visible light sensors, such as high resolution and sharp texture details, with the functional highlights provided by thermal infrared sensors. This not only fully preserves the texture details of the scene background, but also makes salient objects in the resulting image more prominent, significantly improving human visual perception. It also helps improve the performance of subsequent downstream tasks such as object detection, tracking, or recognition. Existing visible light and thermal infrared image fusion methods typically involve three stages: feature extraction, fusion strategy, and reconstruction. Most methods extract unique features from both the visible light and infrared images separately during the feature extraction stage. Then, in the fusion stage, these features are combined to obtain a more informative joint representation, and the final fused image is generated during the reconstruction stage. However, the following problems still exist: features cannot be extracted effectively and completely. Due to the different characteristics of infrared and visible light images, existing methods often find it difficult to fully retain the salient target information in infrared and the background texture details in visible light; due to the interference of potential uncertainty and noise of features, it is difficult to achieve reasonable fusion. When extracting and merging image features, different features have essential differences, and the interference of potential uncertainty and noise of features will amplify this difference, resulting in missing details and unstable performance of the fusion results. Summary of the Invention

[0003] Technical problem to be solved by the present invention: In response to the above-mentioned problems in the prior art, a multimodal image fusion method and system based on joint evaluation of feature uncertainty are provided. The present invention aims to solve the problems of insufficient feature extraction and unstable fusion results caused by modal differences and potential feature uncertainties in image fusion, and overcome the problems of missing details and unstable performance of fusion results due to insufficient feature extraction and unreasonable fusion.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A multimodal image fusion method based on joint evaluation of feature uncertainty, including and thermal infrared images The stitched image of the two The features of S different scales extracted from N time steps are used to extract the feature potential uncertainty weights of the features of different time steps using a fully connected layer neural network model, and the features of different time steps at the same scale are weighted and concatenated based on the feature potential uncertainty weights to obtain S preliminary fusion features of different scales, and then the final fusion features are obtained by upsampling layer by layer and adjusting the number of channels using the attention module. And use the decoder to decode and get the output fusion image The fully connected layer neural network model is used to extract features of S different scales for N time steps and obtain the mean and standard deviation of the feature potential distribution, and calculate the feature potential uncertainty weights of features at different time steps according to the following formula:

[0006] ,

[0007] in, is the t time step, The characteristic potential uncertainty weight of the scale feature, is the t time step The average standard deviation of the latent distribution of features at all time steps under the scale, For all time steps The average of the standard deviations of the underlying distribution of features across all time steps at this scale.

[0008] Optionally, the fully connected layer neural network model is used to extract features of S different scales for N time steps and obtain the mean and standard deviation of the feature potential distribution, including: first, performing channel compression on the features of S different scales for N time steps through a channel compression module, and the channel compression module is composed of a 1×1 convolution, an instance normalization layer and a ReLU activation function connected in sequence, and then processing the channel-compressed features through an adaptive average pooling layer, a feature extraction module, and an adaptive global average pooling layer, and finally passing through two parallel fully connected branch networks to respectively obtain the mean and standard deviation of the feature potential distribution, and the fully connected branch networks are composed of a linear layer, an instance normalization layer, a ReLU activation function, a random dropout layer and a linear layer connected in sequence, and one of the fully connected branch networks is used to output the mean of the feature potential distribution and the other outputs the standard deviation of the feature potential distribution.

[0009] Optionally, when training the fully connected layer neural network model, the calculation function expression of the feature potential distribution adopted is:

[0010] ,

[0011] in, For a given feature Under the condition, the potential probability distribution of the feature, For the The characteristics of the scale, for The characteristic potential uncertainty weight of and Characteristics The mean and standard deviation of the feature potential uncertainty weights of features at different time steps; and Represents the module that obtains the mean and standard deviation of the feature potential distribution in the fully connected layer neural network model.

[0012] Optionally, when training the fully connected layer neural network model, the function expression of the training loss function used is:

[0013] ,

[0014] in, is the training loss function used in training the fully connected layer neural network model. represents a specific characteristic scale, S is the total number of characteristic scales, is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions; For a given feature The potential probability distribution of the feature under the condition, For a given feature The potential probability distribution of the feature under the condition, is a standard normal distribution.

[0015] Optionally, the method of weighting the splicing and compressing the features of different time steps at the same scale based on the feature potential uncertainty weights to obtain S kinds of preliminary fusion features at different scales includes: weighting the features of the same scale at different time steps in the channel dimension according to the feature potential uncertainty weights of the features of the different time steps, and then using a convolution module formed by a combination of 1×1 convolution and ReLU activation function to perform channel compression on the spliced multi-channel features and enhance their expression ability; after completing the channel compression, using a convolution module formed by a combination of 3×3 convolution and ReLU activation function to further extract the features after channel compression, thereby obtaining a preliminary fusion feature that is consistent with the original input in spatial size and has the number of channels mapped to the specified output dimension. ,in is the time step The preliminary fusion features at the i-th scale.

[0016] Optionally, the final fusion feature is obtained by upsampling layer by layer and adjusting the number of channels using the attention module. When , the function expression of upsampling and using the attention module to adjust the number of channels is:

[0017] ，

[0018] in, To integrate Scale characteristics After the fusion feature, is the attention module; For interpolation upsampling, The resolution is doubled; To integrate Scale characteristics The fusion features after Scale characteristics The fusion features As the final fusion feature .

[0019] Optionally, the decoder is used to decode the output fused image Including: using a convolution module formed by a combination of 3×3 convolution and LeakyReLU activation function to compress the high-dimensional features and enhance their expressiveness; using a convolution module formed by a combination of 3×3 convolution and Tanh activation function to map the compressed features to the three-channel output range of the target to generate the output fused image , and the function expression of the training loss function used by the decoder during training is:

[0020] ,

[0021] ,

[0022] ,

[0023] in, is the training loss function used by the decoder during training, and is the intermediate variable, For visible light images of high, For visible light images width, To fuse the image No. Channel image, For thermal infrared images, The visible light image Channel image, is the L1 norm; is the gradient operator, Indicates taking the maximum value.

[0024] In addition, the present invention also provides a multimodal image fusion system based on feature uncertainty joint evaluation, comprising:

[0025] The feature potential uncertainty weight acquisition program unit is used to convert the feature potential uncertainty weight into the feature potential uncertainty weight. and thermal infrared images The stitched image of the two The features of S different scales extracted at N time steps are used to extract the feature potential uncertainty weights of the features at different time steps using a fully connected layer neural network model; the fully connected layer neural network model is used to extract features of S different scales at N time steps and obtain the mean and standard deviation of the feature potential distribution, and calculate the feature potential uncertainty weights of the features at different time steps according to the following formula:

[0026] ,

[0027] in, is the t time step, The characteristic potential uncertainty weight of the scale feature, is the t time step The average standard deviation of the latent distribution of features at all time steps under the scale, For all time steps The average of the standard deviations of the underlying distribution of features across all time steps at the scale;

[0028] The feature fusion program unit is used to weight the features of different time steps at the same scale based on the potential uncertainty weight of the features to obtain S kinds of preliminary fusion features of different scales, and then obtain the final fusion features by upsampling layer by layer and adjusting the number of channels using the attention module. ;

[0029] Decoding program unit, used to finally fusion features The decoder is used to decode the output fusion image .

[0030] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainty through a processor.

[0031] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainties through a processor.

[0032] Compared with the existing technology, the present invention can mainly achieve the following beneficial effects: considering the different characteristics of thermal infrared images and visible light images, and the interference of potential uncertainty and noise of the features, it is very challenging to obtain high-quality and robust fused images. The present invention adopts a diffusion model in the feature extraction stage. The gradual iterative mechanism of the diffusion model can continuously suppress noise, extract and improve the key features of visible light and infrared images in the process of time steps, which helps to more carefully combine the feature information of different modalities; the present invention adopts a fully connected layer neural network model to learn the potential representation of the features and evaluate their potential uncertainty, which can more scientifically balance the texture details of the visible light image and the salient target information of the infrared image during the fusion process, avoiding the loss of details and unstable performance of the fusion results caused by noise or uncertainty; the present invention can solve the problems of difficult feature extraction and difficult quantification of potential uncertainty and noise interference in the image fusion process, and realize high-quality and robust fusion of multimodal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0034] Figure 2 Schematic diagram of the network structure in an embodiment of the present invention.

[0035] Figure 3 Schematic diagram of the network structure of the fully connected neural network model in an embodiment of the present invention.

[0036] Figure 4 Schematic diagram of the process of splicing and compressing according to the potential uncertainty weights of features in an embodiment of the present invention.

[0037] Figure 5 Schematic diagram of the structure of the attention module in an embodiment of the present invention.

[0038] Figure 6 Schematic diagram of the structure of a decoder in an embodiment of the present invention.

[0039] Figure 7 The figure shows the test results of an example on the MSRS dataset in an embodiment of the present invention, where (a) to (d) are visible light images on the MSRS dataset, (a1) to (d1) are thermal infrared images on the MSRS dataset, and (a2) to (d2) are the image fusion results of the method of this embodiment. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0041] like Figure 1 As shown, the multimodal image fusion method based on joint evaluation of feature uncertainty in this embodiment includes the following steps:

[0042] S1, input visible light image and thermal infrared images Stitch by channel to get the stitched image ;

[0043] S2, stitch the images Input the trained diffusion model (DDPM) and extract S features of different scales from each of the N time steps of the diffusion model to obtain N×S features;

[0044] S3: Input the obtained N×S features into the trained fully connected layer neural network model to obtain the feature potential uncertainty weights of the features at different time steps. For the features of the same scale at different time steps, weighted splicing and compression are performed according to the feature potential uncertainty weights of the features at different time steps to obtain S preliminary fusion features at different scales. ;

[0045] S4, preliminary fusion features of S different scales Upsample layer by layer according to the scale and use the attention module to adjust the number of channels to obtain the final fusion feature ;

[0046] S5, the final fusion feature Input the trained decoder to get the output fused image .

[0047] In step S1 of this embodiment, the input visible light image and thermal infrared images Stitch by channel to get the stitched image Includes: Input visible light image and thermal infrared images Convert them into RGB three-channel matrices respectively to get visible light image input and thermal infrared imaging input ; Input the visible light image Three channels and thermal infrared imaging input The first channel of is spliced channel by channel to obtain a four-channel spliced image For example, in this embodiment, the visible light image input The scale is (3, 640, 480), thermal infrared imaging input The scale is (1, 640, 480), and the visible light and thermal infrared mosaic images are obtained by channel stitching The scale is (4, 640, 480), where the first element of the scale represents the channel, the second element represents the length, and the third element represents the width.

[0048] like Figure 2 As shown, the diffusion model of this embodiment is a denoising network of a U-Net network structure. The U-Net network structure is composed of a multi-level forward process module and a multi-level reverse process module. The adjacent forward process modules contain a downsampling module, and the adjacent reverse process modules contain an upsampling module. The forward process module is used to stitch the images at different time steps. Add random Gaussian noise, and the reverse process module is based on the image of time step t And the time step t prediction and gradual removal of noise to finally get the original image , the forward process adds random Gaussian noise to the image at different time steps. This process can be regarded as a Markov chain. Therefore, the function expression of the forward process module in this embodiment is:

[0049] ,

[0050] in, For the forward process, given the initial stitching image condition Under this condition, an image is generated at time step t The probability distribution of is the image at time step t, To stitch images The sampling, is the normal distribution function, is the cumulative noise weight of time step t, is the identity matrix, is the noise weight at time step t, is the noise weight at time step t, is the noise weight of the s time step; the function expression of the reverse process module is:

[0051] ,

[0052] in, is the image at time step t-1, For the denoising network based on The predicted Gaussian noise of the output at time step t, is the standard deviation of Gaussian noise, is a random variable sampled from a standard normal distribution, used to inject noise into the inverse process. In this embodiment, the diffusion model is a denoising network with a U-Net network structure. Compared to traditional encoder-decoder networks, the U-Net network structure more specifically preserves feature information and achieves deeper feature fusion through skip connections during the downsampling and upsampling stages. U-Net first gradually downsamples in the encoder to extract multi-scale deep features; then, in the decoder, it gradually upsamples and concatenates the feature maps using skip connections, effectively reducing the loss of high-resolution details during the downsampling process. Skip connections are similar to the residual concept to a certain extent, alleviating the vanishing or exploding gradient problems that can occur in deep networks while also preserving more local details and edge information, making network training easier and more effective. Other encoders can also be used.

[0053] In this embodiment, the function expression of the training loss function used in the diffusion model training is:

[0054] ,

[0055] in, is the training loss function used by the diffusion model during training, is the true value of the Gaussian noise randomly sampled in the forward process, For the denoising network based on The predicted Gaussian noise of the output t time steps; the extracting S features of different scales from each of the N time steps of the diffusion model refers to extracting features from the upsampling modules between S adjacent reverse process modules in each of the N time steps of the diffusion model, thereby obtaining S features in each time step:

[0056] ,

[0057] in, represents the features of the i-th scale at time step t, is the moment when the diffusion model starts denoising, The moment when the diffusion model completes denoising. From the N time steps of the diffusion model DDPM, S different scales of U-Net features are extracted from the upsampling process of the U-Net network. , including N S eigenvectors; for example, in the diffusion model DDPM in this embodiment, the total time step is 100, such as Figure 2As shown in the figure, N is 3, which represents the 0th time step (t=0), the 50th time step (t=50), and the 100th time step (t=100) of the diffusion model; S is 5, so the five scale features at the 0th time step (t=0) are , , , and , the features of the five scales at the 50th time step (t=50) are , , , and , the five scale features at the 100th time step (t=100) are , , , and ,like Figure 1 As shown; N is extracted from the upsampling process of the U-Net network S types of U-Net features at different time steps and scales , where the superscript Indicates the Time step, in the subscript Indicates the scales; the five different scales are (1024, 40, 30), (1024, 80, 60), (512, 160, 120), (256, 320, 240) and (128, 640, 480), corresponding to , , , and scale.

[0058] like Figure 3As shown, the fully connected neural network model in this embodiment includes a channel compression module, an adaptive average pooling layer, a feature extraction module, an adaptive global average pooling layer, and two parallel fully connected branch networks. The channel compression module, adaptive average pooling layer, feature extraction module, and adaptive global average pooling layer are sequentially connected, and the adaptive global average pooling layer is respectively connected to the input ends of the two parallel fully connected branch networks. The channel compression module is composed of a 1×1 convolution, an instance normalization layer (InstanceNorm), and a ReLU activation function connected in sequence. The fully connected branch networks are composed of a linear layer, an instance normalization layer (InstanceNorm), a ReLU activation function, a random dropout layer, and a linear layer connected in sequence. One of the fully connected branch networks is used to output the mean of the feature potential distribution, and the other outputs the standard deviation of the feature potential distribution. The fully connected neural network model first uses a channel compression module composed of a 1×1 convolution, an InstanceNorm, and a ReLU activation function to enhance feature expression for the input multi-channel image features. Subsequently, the output features are aligned to the specified spatial scale through an adaptive average pooling layer. Next, a pre-trained ResNet-152 network is introduced as the backbone network for deep feature extraction, and its first convolutional layer is adapted to handle features after channel compression. After completing deep feature extraction, a compact image representation is obtained through global adaptive average pooling. Finally, two parallel fully connected branch networks (internal components of a linear layer, InstanceNorm, and ReLU activation function) are used to output the mean and standard deviation vectors of the latent distribution, respectively, completing the learning of the feature latent representation.

[0059] The feature extraction module (backbone network) can adopt a desired network model as needed. For example, as an optional implementation, this embodiment specifically uses the ResNet-152 neural network. Compared to traditional convolutional neural networks, the ResNet-152 neural network has a deeper network layer structure and introduces the concept of residual connections, which solves the gradient vanishing and gradient exploding problems in deep network training. Residual connections allow the network to learn identity mapping, making network training easier.

[0060] According to the standard deviation vector of each feature, the standard deviation vector is used to quantify the intrinsic uncertainty of the modal to obtain the feature potential uncertainty weight. In this embodiment, the calculation function expression of the feature potential uncertainty weight of the features at different time steps is:

[0061] ,

[0062] in, is the t time step, The characteristic potential uncertainty weight of the scale feature, is the t time step The average standard deviation of the latent distribution of features at all time steps under the scale, For all time steps The average value of the standard deviation of the potential distribution of all time-step features under the scale; the calculation function expression of the feature potential distribution used in the training of the fully connected layer neural network model is:

[0063] ,

[0064] in, For a given feature Under the condition, the potential probability distribution of the feature, For the The characteristics of the scale, for The characteristic potential uncertainty weight of and Characteristics The mean and standard deviation of the feature potential uncertainty weights of features at different time steps; and They represent two parallel fully connected branch networks and their upstream shared channel compression module, adaptive average pooling layer, feature extraction module and adaptive global average pooling layer respectively; for example, in this embodiment, the number of potential features is 128, so the output potential feature mean is and standard deviation are all 128-dimensional vectors. In this embodiment, the total number of characteristic scales S is 5, so the scale .

[0065] In this embodiment, the function expression of the training loss function used in the training of the fully connected layer neural network model is:

[0066] ,

[0067] in, is the training loss function used in training the fully connected layer neural network model. represents a specific characteristic scale, S is the total number of characteristic scales, is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions; For a given feature The potential probability distribution of the feature under the condition, For a given feature The potential probability distribution of the feature under the condition, is a standard normal distribution.

[0068] like Figure 4 As shown, in this embodiment, the features of the same scale at different time steps are weighted and spliced and compressed according to the feature potential uncertainty weights of the features at different time steps to obtain S kinds of preliminary fusion features at different scales. It includes: weighted splicing of features of the same scale at different time steps in the channel dimension according to the potential uncertainty weights of the features of the features at different time steps, and then using a convolution module formed by a combination of 1×1 convolution and ReLU activation function to compress the channels of the spliced multi-channel features and enhance their expressiveness; after completing the channel compression, a convolution module formed by a combination of 3×3 convolution and ReLU activation function is used to further extract the features after channel compression, thereby obtaining a preliminary fusion feature that is consistent with the original input in spatial size and has the number of channels mapped to the specified output dimension For example, in this embodiment, firstly, for the first scale feature at different time steps, Perform weighted splicing. Specifically, the 0th time step and the 1st scale feature , the 50th time step, the first scale feature And the 100th time step, the first scale feature Multiply by the corresponding weights respectively, then splice and compress by channel to obtain the first scale preliminary fusion features .in, The scales of the two channels are (1024, 40, 30), and the scale after channel splicing is (3072, 40, 30). After compression, the initial fusion features of the first scale are obtained. The scale is (1024, 40, 30) and In this embodiment, the S kinds of preliminary fusion features of different scales are Upsample layer by layer according to the scale and use the attention module to adjust the number of channels to obtain the final fusion feature When , the function expression of upsampling and using the attention module to adjust the number of channels is:

[0069] ，

[0070] in, To integrate Scale characteristics After the fusion feature, is the attention module; For interpolation upsampling, The resolution is doubled; To integrate Scale characteristics The fusion features after Scale characteristics The fusion features As the final fusion feature .like Figure 5 As shown in the figure, the attention module first uses a convolution module composed of a 3×3 convolution and a ReLU activation function to adjust the features and enhance their expressiveness. After completing the channel adjustment, a channel space layer composed of a channel and spatial attention mechanism is introduced, which contains two parallel branches: the channel attention branch and the spatial attention branch. The channel attention branch first performs global average pooling (channel direction) on the input features, and then learns the importance of each channel through a two-layer fully connected network (including nonlinear activation and Sigmoid function), a Sigmoid function, and a linear layer. Finally, the learned channel weights are multiplied channel by channel with the original features of the input channel attention branch. The spatial attention branch uses 1×1 convolution and Sigmoid function to learn the weight matrix for each pixel position (across channels) of the input features, and multiplies it pixel by pixel with the original features of the input spatial attention branch. Finally, through weighted fusion, the number of channels of the feature is adjusted to the expected value while maintaining the spatial resolution unchanged.

[0071] It should be noted that the interpolation used in interpolation upsampling is a well-known data processing method, and the required interpolation algorithm can be used as needed. For example, the bilinear interpolation algorithm is used in this embodiment. The bilinear interpolation comprehensively considers the distribution changes of pixel values in the horizontal and vertical directions in a local range, and estimates the unknown pixel value by weighted average. Its characteristic is that linear interpolation is first performed in the horizontal direction, and then in the vertical direction, so as to generate the target pixel value, which can retain the details and smoothness of the original image as much as possible when the image is enlarged. Specifically, the bilinear interpolation algorithm will find the four known pixel points (denoted as ) closest to the target pixel position in the original image. ), and then perform two-step interpolation based on these four pixel values. The first step is Interpolate in the direction of the axis and find the point with the same The two middle values of the coordinates and , its calculation can be expressed as:

[0072] ,

[0073] ,

[0074] in, is the interpolation position, and Respectively represent the endpoints of the horizontal coordinate interval where the adjacent known pixel points are located. After completing the horizontal interpolation, the second step is Axis direction and Perform linear combination to obtain the final interpolation result:

[0075] .

[0076] In this way, the influence of surrounding pixels can be integrated in both vertical and horizontal dimensions to obtain the target pixel value, thereby ensuring that the amplified features maintain the original details as much as possible while improving the resolution. The scale of is (1024, 40, 30), and the scale after bilinear interpolation upsampling is (1024, 80, 60). The scale is also (1024, 80, 60), so this layer It will not change the feature scale after upsampling. and Add together to get , The scale is (1024, 80, 60); then, The scale after bilinear interpolation upsampling is (1024, 160, 120), and The scale is (512, 160, 120), so this layer The feature scale of the upsampling result will be converted to (512, 160, 120), and Add together to get ; And so on, the final fusion feature can be obtained , whose scale is (128, 640, 480).

[0077] like Figure 6 As shown, in this embodiment, the final fusion feature Input the trained decoder to get the output fused image When , the decoder obtains the output fused image The process includes: first, using a convolution module composed of a 3×3 convolution and a LeakyReLU activation function to compress the high-dimensional features and enhance their expression ability; then using a convolution module composed of a 3×3 convolution and a Tanh activation function to map the compressed features to the three-channel output range of the target to generate the output fused image. , and the function expression of the training loss function used by the decoder during training is:

[0078] ,

[0079] ,

[0080] ,

[0081] in, is the training loss function used by the decoder during training, and is the intermediate variable, For visible light images of high, For visible light images width, To fuse the image No. Channel image, For thermal infrared images, The visible light image Channel image, is the L1 norm; is the gradient operator, Indicates taking the maximum value. Among them, the loss function The L1 norm of the output fusion image gradient is calculated according to the channel and the visible light and thermal infrared source image gradient; the loss function The L1 norm of the output fusion image gradient is calculated by channel and the gradient of the visible light and thermal infrared source images.

[0082] To test the image fusion performance of the method in this embodiment, training was performed on the MSRS dataset and testing was performed on both the MSRS and M3FD datasets. The MSRS dataset provides a dataset of thermal infrared and visible light images. It utilizes an image enhancement algorithm based on dark channel priors to optimize the contrast and signal-to-noise ratio of infrared images. It contains 1,444 pairs of high-quality aligned thermal infrared and visible light images. The M3FD dataset is a dataset of visible-infrared binocular image pairs covering a variety of scenes and object categories. It is used for multimodal image fusion, object detection, and fusion-based detection research. It contains 8,400 images (4,200 pairs) for multimodal image fusion, object detection, and fusion-based detection, and an additional 600 images (300 pairs) from independent scenes used solely for multimodal image fusion research. This embodiment uses the 1,444 pairs of thermal infrared and visible light images from the MSRS dataset and the 600 images from M3FD used for multimodal image fusion. For comparison with the method in this example, the DDFM method used is detailed in the reference: Zhao, Z., Bai, H., Zhu, Y., Zhang, J., Xu, S., Zhang, Y., Zhang, K., Meng, D., Timofte, R., & Gool, LV (2023). DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion. 2023 IEEE / CVF International Conference on Computer Vision (ICCV), 8048-8059. Furthermore, information entropy (EN), standard deviation (SD), spatial frequency (SF), mutual information (MI), visual fidelity (VIF), and gradient-based fusion performance (Qabf) are used to measure image fusion performance. Higher values indicate better image fusion performance. Information entropy (EN) is a metric used to assess the information richness of an image. A larger value indicates more detail and information in the fused image. Standard deviation (SD) reflects the degree of dispersion in the overall grayscale distribution of an image. A larger value indicates greater contrast in the fused image. Spatial frequency (SF) quantifies texture and detail in an image. A higher spatial frequency indicates better clarity and detail preservation in the fused image. Mutual information (MI) characterizes the degree of dependency and shared information between the fused image and the source image. A larger value indicates that the fused image better preserves the source image's information. Visual fidelity (VIF) compares the visual similarity between the source and fused images based on human perception. Gradient-based fusion performance (Qabf) analyzes image gradient information and edge details to comprehensively measure the clarity and detail preservation of the fused image, thereby evaluating the effectiveness of the fusion algorithm. The fusion results of this embodiment's method and the DDFM algorithm on the MSRS and M3FD datasets are shown in Tables 1 and 2.

[0083] Table 1 Comparison of fusion results between the method of this embodiment and the DDFM algorithm on the MSRS dataset

[0084]

[0085] Table 2 Comparison of fusion results between the method in this embodiment and the DDFM algorithm on the M3FD dataset

[0086]

[0087] As can be seen from Tables 1 and 2, the various metrics of the fusion results of the method of this embodiment are higher than those of the DDFM method in the prior art. The method of this embodiment has achieved improvements in both visible light and thermal infrared image fusion tasks. Figure 7 Figure 2 shows the results of an example test using the MSRS dataset using the method of this embodiment. (a)-(d) are visible light images from the MSRS dataset, (a1)-(d1) are thermal infrared images from the MSRS dataset, and (a2)-(d2) are the image fusion results of this embodiment. This demonstrates that this embodiment effectively integrates the high-resolution, clear texture details provided by the visible light sensor with the functional highlights provided by the thermal infrared sensor, fully preserving the texture details of the scene background and making salient objects in the resulting image more prominent, significantly enhancing human visual perception. This also helps improve the performance of downstream tasks such as target detection, tracking, or recognition.

[0088] In summary, the multimodal image fusion method based on joint feature uncertainty assessment in this embodiment adopts a diffusion model in the feature extraction stage. The gradual iterative mechanism of the diffusion model can continuously suppress noise, extract and improve key features of visible light and infrared images in a time-step process, and help to more carefully combine feature information from different modalities. At the same time, a fully connected layer neural network model is used to learn the latent representation of features and evaluate their potential uncertainty. This can more scientifically balance the texture details of visible light images and the salient object information of infrared images during the fusion process, avoiding the loss of details and unstable performance of the fusion results caused by noise or uncertainty. This embodiment can solve the problems of difficult feature extraction and difficult quantification of potential feature uncertainty and noise interference during image fusion, and achieve high-quality and robust fusion of multimodal images.

[0089] In addition, this embodiment also provides a multimodal image fusion system based on joint evaluation of feature uncertainty, including:

[0090] The feature potential uncertainty weight acquisition program unit is used to convert the feature potential uncertainty weight from the visible light image and and thermal infrared images The stitched image of the two The features of S different scales extracted at N time steps are used to extract the feature potential uncertainty weights of the features at different time steps using a fully connected layer neural network model; the fully connected layer neural network model is used to extract features of S different scales at N time steps and obtain the mean and standard deviation of the feature potential distribution, and calculate the feature potential uncertainty weights of the features at different time steps according to the following formula:

[0091] ,

[0092] in, is the t time step, The characteristic potential uncertainty weight of the scale feature, is the t time step The average standard deviation of the latent distribution of features at all time steps under the scale, For all time steps The average of the standard deviations of the underlying distribution of features across all time steps at the scale;

[0093] The feature fusion program unit is used to weight the features of different time steps at the same scale based on the potential uncertainty weight of the features to obtain S kinds of preliminary fusion features of different scales, and then obtain the final fusion features by upsampling layer by layer and adjusting the number of channels using the attention module. ;

[0094] Decoding program unit, used to finally fusion features The decoder is used to decode the output fusion image .

[0095] In addition, this embodiment also provides a multimodal image fusion system based on joint evaluation of feature uncertainty, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainty.

[0096] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainty through a processor.

[0097] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainties through a processor.

[0098] Those skilled in the art should understand that the technical solution provided by the present invention may be in the form of a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal image fusion method based on joint evaluation of feature uncertainty, characterized in that: including visible light images and thermal infrared images The stitched image of the two The features of S different scales extracted from N time steps are used to extract the feature potential uncertainty weights of the features of different time steps using a fully connected layer neural network model, and the features of different time steps at the same scale are weighted and concatenated based on the feature potential uncertainty weights to obtain S preliminary fusion features of different scales, and then the final fusion features are obtained by upsampling layer by layer and adjusting the number of channels using the attention module. , and use the decoder to decode and get the output fusion image The fully connected layer neural network model is used to extract features of S different scales for N time steps and obtain the mean and standard deviation of the feature potential distribution, and calculate the feature potential uncertainty weights of features at different time steps according to the following formula: ， in, is the t time step, The characteristic potential uncertainty weight of the scale feature, is the tth time step The mean of the standard deviations of the underlying distribution of features at scale, For all time steps The average of the standard deviations of the underlying distribution of features at the scale.

2. The multimodal image fusion method based on joint evaluation of feature uncertainty according to claim 1, characterized in that: The fully connected layer neural network model is used to extract features of S different scales for N time steps and obtain the mean and standard deviation of the feature potential distribution, including: first, channel compression is performed on the features of S different scales for N time steps through a channel compression module, and the channel compression module is composed of a 1×1 convolution, an instance normalization layer and a ReLU activation function connected in sequence; then, the channel-compressed features are processed through an adaptive average pooling layer, a feature extraction module, and an adaptive global average pooling layer; finally, two parallel fully connected branch networks are used to respectively obtain the mean and standard deviation of the feature potential distribution; the fully connected branch networks are composed of a linear layer, an instance normalization layer, a ReLU activation function, a random dropout layer and a linear layer connected in sequence, and one of the fully connected branch networks is used to output the mean of the feature potential distribution and the other outputs the standard deviation of the feature potential distribution.

3. The multimodal image fusion method based on feature uncertainty joint evaluation according to claim 2, characterized in that: When training the fully connected layer neural network model, the calculation function expression of the feature potential distribution adopted is: ， in, For a given feature The potential probability distribution of the features under the condition, For the The characteristics of scale, for The potential uncertainty weight of the feature, and For given features The mean and standard deviation of the underlying distribution of the features; and Represents the module that obtains the mean and standard deviation of the feature potential distribution in the fully connected layer neural network model.

4. The multimodal image fusion method based on joint evaluation of feature uncertainty according to claim 3, characterized in that: When training the fully connected layer neural network model, the function expression of the training loss function used is: ， in, is the training loss function used in training the fully connected layer neural network model. represents a specific characteristic scale, S is the total number of characteristic scales, is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions; For a given feature The potential probability distribution of the features under the condition, For a given feature The potential probability distribution of the features under the condition, is a standard normal distribution.

5. The multimodal image fusion method based on joint evaluation of feature uncertainty according to claim 1, characterized in that: The method of weighting the splicing and compression of features at different time steps at the same scale based on the feature potential uncertainty weights to obtain S kinds of preliminary fusion features at different scales includes: weighting the features of the same scale at different time steps in the channel dimension according to the feature potential uncertainty weights of the features at different time steps, and then using a convolution module formed by a combination of 1×1 convolution and ReLU activation function to perform channel compression on the spliced multi-channel features and enhance their expression ability; after completing the channel compression, using a convolution module formed by a combination of 3×3 convolution and ReLU activation function to further extract features from the channel-compressed features, thereby obtaining preliminary fusion features that are consistent with the original input in spatial size and whose number of channels is mapped to the specified output dimension. ,in is the time step The preliminary fusion features at the i-th scale.

6. The multimodal image fusion method based on joint evaluation of feature uncertainty according to claim 1, characterized in that: The final fusion feature is obtained by upsampling layer by layer and adjusting the number of channels using the attention module When , the function expression of upsampling and using the attention module to adjust the number of channels is: ， in, To integrate Scale characteristics After the fusion feature, is the attention module; For interpolation upsampling, The resolution is doubled; To integrate Scale characteristics The fusion features after Scale characteristics The fusion features after As the final fusion feature .

7. The multimodal image fusion method based on joint evaluation of feature uncertainty according to claim 1, characterized in that: The decoder is used to decode and obtain the output fusion image Including: using a convolution module formed by a combination of 3×3 convolution and LeakyReLU activation function to compress the high-dimensional features and enhance their expressiveness; using a convolution module formed by a combination of 3×3 convolution and Tanh activation function to map the compressed features to the three-channel output range of the target to generate the output fused image , and the function expression of the training loss function used by the decoder during training is: ， ， ， in, is the training loss function used by the decoder during training, and is the intermediate variable, For visible light images and of high, For visible light images width, To fuse the image No. Channel image, For thermal infrared images, The visible light image Channel image, is the L1 norm; is the gradient operator, Indicates taking the maximum value.

8. A multimodal image fusion system based on joint evaluation of feature uncertainty, characterized in that: include: The feature potential uncertainty weight acquisition program unit is used to convert the feature potential uncertainty weight into the feature potential uncertainty weight. and thermal infrared images The stitched image of the two The features of S different scales extracted at N time steps are used to extract the feature potential uncertainty weights of the features at different time steps using a fully connected layer neural network model; the fully connected layer neural network model is used to extract features of S different scales at N time steps and obtain the mean and standard deviation of the feature potential distribution, and calculate the feature potential uncertainty weights of the features at different time steps according to the following formula: ， in, is the t time step, The characteristic potential uncertainty weight of the scale feature, is the tth time step The mean of the standard deviations of the underlying distribution of features at scale, For all time steps The mean of the standard deviations of the underlying distribution of the features at the scale; The feature fusion program unit is used to weight the features of different time steps at the same scale based on the potential uncertainty weight of the features to obtain S kinds of preliminary fusion features of different scales, and then obtain the final fusion features by upsampling layer by layer and adjusting the number of channels using the attention module. ; Decoding program unit, used to finally fusion features The decoder is used to decode the output fusion image .

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainty according to any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the multimodal image fusion method based on joint evaluation of feature uncertainty according to any one of claims 1 to 7 through a processor.

Images