Multi-modal image fusion method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a multi-modal image fusion method and device, equipment, a storage medium and a program product. Extracting a cross-modal implicit joint feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capture network to obtain an implicit fusion feature vector; correspondingly modulating the visible light feature vector and the infrared feature vector according to the content association degree between the visible light feature vector and the implicit fusion feature vector and the content association degree between the infrared feature vector and the implicit fusion feature vector; determining an attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector; extracting the content of the implicit fusion feature vector based on the attention score to obtain an explicit fusion representation vector; and obtaining a fused image based on the explicit fusion representation vector. According to the multi-modal image fusion method provided by the embodiment of the invention, deep complementary information between the visible light mode and the infrared mode can be fully mined, fine-grained fusion of the images is realized, and the image fusion quality is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of image fusion, and particularly relates to a multi-modal image fusion method, device, equipment, storage medium and program product. BACKGROUND

[0002] Multi-modal image fusion refers to a process of integrating image information of the same scene from different sensors or different imaging modes to obtain a fused image with higher quality and higher information quantity. However, due to the great heterogeneity of images of different modalities in the feature space, traditional multi-modal image fusion methods, such as Laplacian pyramid, wavelet transform, nonsubsampled contourlet transform (NSCT), etc., although can achieve the goal of image fusion to a certain extent, often cannot fully exploit and utilize the complementary information between different modal images, thereby limiting the quality and availability of the fused image. SUMMARY

[0003] The embodiments of the present disclosure provide a multi-modal image fusion method, device, equipment, storage medium and program product, which can improve the fusion quality of visible light images and infrared images.

[0004] In a first aspect, the embodiments of the present disclosure provide a multi-modal image fusion method, which comprises: extracting a visible light feature vector and an infrared feature vector of a target image; extracting an implicit fusion feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capturing network, the implicit fusion feature vector being used to represent an implicit joint feature fusion feature shared by the visible light feature vector and the infrared feature vector; modulating the visible light feature vector and the infrared feature vector according to the content correlation degree of the visible light feature vector and the infrared feature vector with the implicit fusion feature vector, respectively; determining an attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector, the attention score being used to represent the correlation weight between the modulated visible light feature vector and the modulated infrared feature vector; extracting the content of the implicit fusion feature vector based on the attention score to obtain an explicit fusion representation vector, the explicit fusion representation vector being used to represent a fusion feature representation obtained based on an attention mechanism; obtaining a fused image based on the explicit fusion representation vector.

[0005] In one feasible implementation, the method further includes: extracting an implicit fusion feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capture network, including: Based on the joint implicit feature capture network, cross-modal implicit joint feature vectors of visible light feature vectors and infrared feature vectors are extracted. The cross-modal implicit joint feature vectors are used to characterize the unoptimized implicit joint features shared by visible light feature vectors and infrared feature vectors. The implicit joint eigenvector across modes is transformed nonlinearly to obtain the implicit fused eigenvector.

[0006] In one feasible implementation, the method further includes: extracting a cross-modal implicit joint feature vector of visible light feature vectors and infrared feature vectors based on a joint implicit feature capture network, including: Based on a joint implicit feature capture network, visible light feature vectors and infrared feature vectors are initially fused to obtain a preliminary fused vector. An affine transformation is performed on the initial fusion vector, followed by a nonlinear transformation, to obtain the cross-modal implicit joint feature vector.

[0007] In one feasible implementation, the method further includes: calculating the cross-modal implicit joint feature vector using the following formula: in, Represents the implicit joint feature vector across modalities; Represents the visible light feature vector; Represents the infrared feature vector; This represents the weight matrix of the neural network layers in the joint implicit feature capture network; This indicates the bias parameters of the neural network layers in the Joint Implicit Feature Capture Network. The weight matrix and bias parameters are used for affine transformation. This represents the hyperbolic tangent function, used for nonlinear transformations; This indicates a point-by-point operation, used for initial fusion.

[0008] In one feasible implementation, the method further includes: modulating the visible light feature vector and the infrared feature vector according to the degree of correlation between the visible light feature vector and the infrared feature vector and the implicit fusion feature vector, respectively, including: The first content contribution is determined based on the contribution of the visible light feature vector to the implicit fusion feature vector, and the second content contribution is determined based on the contribution of the infrared feature vector to the implicit fusion feature vector. The contribution of the first content and the contribution of the second content are normalized to obtain the first feature weight and the second feature weight. The visible light feature vector is modulated with a first feature weight, and the infrared feature vector is modulated with a second feature weight.

[0009] In one feasible implementation, the method further includes: determining a first content contribution based on the following formula 1: in, The first content contribution is used to characterize the content contribution of the visible light feature vector to the implicitly fused feature vector. This represents the i-th eigenvalue of the visible light eigenvector; This represents the i-th eigenvalue of the implicitly fused feature vector; Indicates exponentiation; This represents the length of the visible light eigenvector; The contribution of the second content is determined according to the following formula 2: in, The second content contribution is used to characterize the content contribution of the infrared feature vector to the implicitly fused feature vector. This represents the i-th eigenvalue of the infrared eigenvector; This represents the i-th eigenvalue of the implicitly fused feature vector; Indicates exponentiation; This represents the length of the infrared feature vector.

[0010] In one feasible implementation, the method further includes: determining an attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector, including: The similarity is obtained by performing a dot product operation between the modulated visible light feature vector and the modulated infrared feature vector; The similarity is adjusted by the length of the infrared feature vector to obtain the attention score.

[0011] In one feasible implementation, the method further includes: nonlinearly transforming the cross-modal implicit joint feature vector to obtain an implicit fused feature vector, including: A first nonlinear transformation is performed on the cross-modal implicit joint feature vector to obtain the probabilistic representation of the implicit fused feature vector; Based on the attention score, the implicit fusion feature vector is extracted to obtain the explicit fusion representation vector, including: A second nonlinear transformation is performed on the attention score to obtain a probability distribution representation of the attention score; The explicit fusion representation vector is obtained by multiplying the probability distribution of the attention score with the probability representation of the implicit fusion feature vector.

[0012] In one feasible implementation, the method further includes: using When the function undergoes the second nonlinear transformation, the explicit fusion representation vector is obtained by the following formula: in, This represents an explicit fusion representation vector; The function represents the second nonlinear transformation operation; This represents the modulated visible light feature vector; This represents the modulated infrared feature vector; Indicates transpose; Represents the implicitly fused feature vector; This represents the length of the infrared feature vector.

[0013] In one feasible implementation, the method further includes: obtaining a fused image based on an explicit fusion representation vector, including: Encode the explicit fusion representation vector into the initial latent representation; Add noise to the initial latent representation to obtain a noisy latent representation; predict and remove the noise from the noisy latent representation to obtain the output latent representation; The latent representation is decoded to obtain the fused image.

[0014] Secondly, embodiments of this disclosure provide a multimodal image fusion apparatus, the apparatus comprising: The vector extraction module is used to extract the visible light feature vector and infrared feature vector of the target image; The joint vector extraction module is used to extract the implicit fusion feature vector of visible light feature vector and infrared feature vector based on the joint implicit feature capture network. The implicit fusion feature vector is used to characterize the implicit joint features shared by the visible light feature vector and the infrared feature vector. The vector modulation module is used to modulate the visible light feature vector and the infrared feature vector according to the degree of correlation between the content of the visible light feature vector and the infrared feature vector and the implicit fused feature vector, respectively. An attention score determination module is used to determine an attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector; the attention score is used to characterize the correlation weight between the modulated visible light feature vector and the modulated infrared feature vector. The explicit feature fusion module is used to extract the content of the implicit fusion feature vector based on the attention score to obtain the explicit fusion representation vector; the explicit fusion representation vector is used to characterize the fusion feature representation obtained based on the attention mechanism. The image fusion module is used to obtain a fused image based on an explicit fusion representation vector.

[0015] Thirdly, this disclosure provides a multimodal image fusion apparatus, the apparatus including: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the above-described multimodal image fusion method.

[0016] Fourthly, this disclosure provides a computer storage medium storing computer program instructions, which, when executed by a processor, implement the aforementioned multimodal image fusion method.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the multimodal image fusion method described above.

[0018] The multimodal image fusion method, apparatus, device, storage medium, and program product disclosed herein extract cross-modal joint implicit features between visible light and infrared images, mapping the two modal features to a unified semantic space. Within this unified semantic space, semantic alignment of the visible light and infrared forms of the target image is performed. The implicit fusion features guide subsequent vector modulation and vector fusion at the semantic level, facilitating the full exploitation and utilization of the potential correlations between visible light and infrared images and understanding the semantics behind different modal information. Compared to explicit feature guidance, the fusion process of this disclosure relies on the data of the target image itself, eliminating the need for manually determined fusion rules, thus improving semantic understanding capabilities and offering better adaptability and robustness. Simultaneously, the entire fusion process involves at least two interactive processing steps involving visible light and infrared features: the first step involves jointly obtaining implicit fusion features from the visible light and infrared images, and then correcting the visible light and infrared feature vectors based on these implicit fusion features; the second step involves automatically guiding the model's fusion process to areas requiring focus based on the similarity between the corrected visible light and infrared feature vectors, making the fusion features more accurate and improving output precision. By combining the above processing steps, this disclosure can fully explore the deep complementary information between visible light modes and infrared modes, realize fine-grained image fusion, and improve the quality of image fusion. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments of this disclosure will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of a multimodal image fusion method provided in an embodiment of this disclosure; Figure 2This is a flowchart illustrating a method for extracting feature vectors provided in an embodiment of this disclosure; Figure 3 This is a schematic flowchart of a vector modulation method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a multimodal image fusion device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of a multimodal image fusion device provided in an embodiment of this disclosure. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of this disclosure will now be described in detail. To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, will provide a further detailed description. It should be understood that the specific embodiments described herein are intended only to explain this disclosure and not to limit it. For those skilled in the art, this disclosure can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this disclosure by illustrating examples.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0023] To better understand and explain the solutions provided in the embodiments of this disclosure, some technical terms involved in the embodiments of this disclosure will be briefly introduced below.

[0024] Neural network layers are one of the fundamental components of artificial neural networks. In neural networks, data is transmitted and processed through various types of neural network layers. Each layer typically consists of many neurons or nodes. These neurons receive the data output from the previous layer, process it, and then pass the result to the next layer. Neural network layers include fully connected layers (Dense Layer), convolutional layers, pooling layers, recurrent layers, and depthwise separable convolutional layers. In fully connected layers, each neuron is connected to all neurons in the previous layer, and they are often used to process one-dimensional vector data. Convolutional layers are mainly used to process data with a grid structure, such as images, extracting image features by learning the spatial invariance of local connections. Pooling layers are typically used to reduce the dimensionality of data while preserving the most important feature information, helping to control overfitting. Recurrent layers are suitable for processing sequential data, influencing the current input by retaining information from previous time steps. Depthwise separable convolutional layers are a variant of convolution, including independent deep convolutional layers and pointwise convolutional layers, designed to reduce the number of parameters and computational complexity while maintaining model performance. The functions of neural network layers include linear transformations: each neuron performs a linear combination of its inputs, that is, the sum of the product of the input values ​​and their corresponding weights; and nonlinear transformations: using nonlinear functions to process the results of linear combinations, with the aim of increasing the expressive power of the model so that it can fit more complex functions.

[0025] The Transformer architecture is a popular and effective deep learning model architecture in Natural Language Processing (NLP). This architecture was designed to overcome the limitations of long-standing Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) in handling long-range dependencies, particularly the slow training speed of RNNs due to their sequential processing and the limitations of CNNs in capturing long-range dependencies. The main feature of the Transformer architecture includes the self-attention mechanism, a core component that allows the model to focus on the relationships between different parts of the input when processing sequential data. Self-attention mechanisms determine which other positions should influence the output at the current position by calculating the weight distribution across all positions in the input sequence. Positional encoding is needed because self-attention mechanisms don't retain any information about the order of elements in the sequence; it provides the model with information about the relative or absolute position of each element in the input sequence. Multi-head attention allows the model to focus on different information simultaneously in different subspaces. The Transformer architecture allows self-attention to run multiple times in parallel, each time using a different linear projection, then concatenating the results of these attention heads before passing them through a linear layer. Each Transformer block also includes a fully connected feedforward network, which is identical across all positions (i.e., no shared weights). The feedforward network consists of two fully connected linear layers sandwiched between a Rectified Linear Unit (ReLU) or Gaussian Error Linear Unit (GELU) activation function.

[0026] The diffusion model is a deep learning-based probabilistic model that excels at generating images and other complex data structures. The key idea behind the diffusion model is to define a diffusion process that gradually transforms the data distribution into a simple prior distribution (typically Gaussian noise). This is usually achieved through a series of small, successive noise addition steps, each moving the data slightly away from its initial state. During this process, the data gradually loses its original structure until it becomes pure noise. The model then learns how to reverse this diffusion process—that is, how to remove noise at each step, gradually restoring the data structure through reverse diffusion, reconstructing data samples, and thus generating new instances. Taking image generation as an example, the diffusion model starts with pure noise and then gradually restores the image details through a series of denoising steps until a complete image is generated. This approach allows the model to learn the complex distribution of image data and generate new samples that conform to that distribution. Compared to other generative models, such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs), diffusion models excel in generating image quality, especially in high-resolution image generation tasks.

[0027] With the rapid development of information technology, image processing and analysis are playing an increasingly important role in many fields, such as security monitoring, medical diagnosis, and autonomous driving. In these practical applications, multimodal image fusion technology has attracted widespread attention because it can provide richer and more comprehensive information than single-modal images.

[0028] Visible light images, with their high resolution and rich color information, can clearly display the color and texture details of objects, offering a straightforward imaging method. However, under low-light conditions, the quality of visible light images deteriorates significantly, leading to information loss. Infrared images, while lacking color information and having lower spatial resolution, can capture thermal radiation information in completely dark or smoky environments, making them particularly suitable for nighttime surveillance and fire rescue scenarios. Therefore, fusing visible light and infrared images can effectively compensate for the shortcomings of single-modal images, improving the usability and reliability of image information.

[0029] Typical visible-infrared image fusion methods, such as Laplacian pyramid, wavelet transform, and non-subsampled contourlet transform, primarily rely on frequency domain analysis and mathematical transformations to complete the fusion task. However, these methods are essentially mathematical tools that use artificially designed, fixed linear transformations to cut and stitch together shallow features of images. They are often limited to specific image features (such as texture or structural features) and lack an understanding of image semantics. Furthermore, these traditional methods suffer from blind fusion processes, rigid fusion rules, limited feature representation, and high requirements for registration accuracy between different modalities. They cannot fully explore and utilize the complex and deep complementary information between different modalities and cannot cope with situations involving high image diversity and complexity, resulting in low image fusion quality.

[0030] To address the problems of existing technologies, this disclosure provides a multimodal image fusion method, apparatus, device, storage medium, and program product. This disclosure establishes at least two interaction processes between different modalities, facilitating in-depth mining of semantic relationships between modalities. Simultaneously, it guides vector modulation of different modal feature vectors based on implicit joint feature vectors between modalities, and uses the modulated vectors to guide the model to focus on learning in areas requiring more attention. Throughout the process, the model can leverage its own experience to mine implicit correlations between visible light image features and infrared image features, utilizing these implicit correlations to guide fine-grained explicit feature fusion between the two, thereby generating a fused image. This facilitates deeper image information complementarity and results in higher fusion quality.

[0031] This multimodal image fusion method can be applied to scenarios involving the fusion of infrared and visible light images, model training, validation, and use, as well as in fields such as security monitoring, medical diagnosis, autonomous driving, augmented reality (AR), and virtual reality (VR). The execution entity for this multimodal image fusion method can be a deep learning model.

[0032] The multimodal image fusion method provided in the embodiments of this disclosure will be introduced first below.

[0033] Figure 1 A flowchart illustrating a multimodal image fusion method according to an embodiment of this disclosure is shown. Figure 1 As shown, the method may include the following steps: S110. Extract the visible light feature vector and infrared feature vector of the target image; S120. Based on the joint implicit feature capture network, extract the cross-modal implicit joint feature vector of visible light feature vector and infrared feature vector. The cross-modal implicit joint feature vector is used to characterize the unoptimized implicit joint features shared by the visible light feature vector and the infrared feature vector. S130. Nonlinear transformation of the cross-modal implicit joint feature vector to obtain the implicit fusion feature vector. The implicit fusion feature vector is used to characterize the optimized implicit joint features shared by the visible light feature vector and the infrared feature vector. S140. Based on the degree of correlation between the visible light feature vector and the infrared feature vector and the implicit fusion feature vector, respectively, the visible light feature vector and the infrared feature vector are modulated accordingly. S150. Determine the attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector; S160. Extract the content of the implicit fusion feature vector based on the attention score to obtain the explicit fusion representation vector; S170. Obtain the fused image based on the explicit fusion representation vector.

[0034] In other words, in the aforementioned feature interaction fusion process of visible light and infrared feature vectors, a shared and hidden cross-modal implicit joint feature vector is first extracted from the visible light and infrared feature vectors using a joint implicit feature capture network. This extracted cross-modal implicit joint feature vector undergoes a nonlinear transformation to emphasize key information and suppress irrelevant noise, serving as the implicit fusion feature vector. Next, the visible light and infrared feature vectors are correlated with the implicit fusion feature vector to determine their importance to the final representation. This correlation is then used to weight and modulate the original visible light and infrared feature vectors, effectively emphasizing key information while suppressing irrelevant information. Finally, the modulated visible light feature vector, modulated infrared feature vector, and implicit fusion feature vector are used to guide the interaction between features, achieving fine-grained saliency fusion of cross-modal features for high-quality image fusion.

[0035] Based on this, by extracting cross-modal joint implicit features between visible light and infrared images, the two modal features are mapped to a unified semantic space. Within this unified semantic space, semantic alignment of the visible light and infrared forms of the target image is performed. The cross-modal joint implicit features guide subsequent vector modulation and vector fusion at the semantic level, facilitating the full exploitation of the potential correlation between visible light and infrared images. Simultaneously, based on the interaction between visible light and infrared feature vectors, cross-modal joint implicit features are obtained to guide the vector modulation of both visible light and infrared feature vectors. Based on the similarity between the corrected visible light and infrared feature vectors, a second interaction is performed, guiding the learning process to areas requiring focused attention. These two interactive processes and implicit feature-guided learning fully exploit the deep complementary information between the visible light and infrared modalities, achieving fine-grained image fusion and improving the quality of image fusion.

[0036] The above steps are explained in detail below: Regarding step S110, in this embodiment of the disclosure, step S110 can be implemented through the following process. Figure 2 This is a flowchart illustrating a method for extracting feature vectors according to an embodiment of this disclosure. Figure 2 As shown, step S110 includes steps S111 to S114.

[0037] First, in step S111, a target image is acquired. The target image is an image related to the target scene to be fused according to this disclosure. Next, in step S112, a visible light image and an infrared image are obtained from the target image. The visible light image and the infrared image have an overlapping area, and the overlapping area includes at least a portion of the target image. In one embodiment, the target image can be directly processed into visible light and infrared forms to obtain the visible light image and the infrared image. In another embodiment, a visible light image and an infrared image that simultaneously contain a portion of the target image can be acquired. Then, in step S113, visible light image features are extracted from the visible light image, and infrared image features are extracted from the infrared image. Finally, in step S114, the visible light image features are processed into a visible light feature vector; and the infrared image features are processed into an infrared feature vector.

[0038] In a specific example, to efficiently extract representative image features from visible light and infrared images, the visible light and infrared images are respectively input into an image feature extractor based on a deep separable convolutional neural network model to obtain visible light feature vectors and infrared feature vectors.

[0039] Here, a deep separable convolutional neural network (DSN) is a highly efficient convolutional neural network model. By decomposing standard convolution into spatial depthwise convolution and pointwise convolution, it significantly reduces the number of parameters and computational complexity while maintaining high feature extraction capabilities. Utilizing a DSN to extract independent features from each channel of visible light and infrared images, as well as combining features across channels, effectively captures key information in the images, generating visible light and infrared feature vectors, thus providing high-quality feature input for subsequent image fusion.

[0040] It should be noted that since this disclosure is based on semantic image fusion, there is no need to perform highly accurate registration between the visible light image and the infrared image before extracting image features, making it relatively simple in practice.

[0041] Regarding S120, although visible light images and infrared images have different imaging characteristics, they have a natural potential correlation because they both come from the same scene or are related to the same scene. This step is to explore this potential correlation in depth to improve the image fusion effect of visible light images and infrared images.

[0042] In this embodiment of the disclosure, step S120 can be implemented through the following process. First, based on the joint implicit feature capture network, the visible light feature vector and the infrared feature vector are initially fused to obtain an initial fused vector. Then, the initial fused vector is subjected to an affine transformation and a nonlinear transformation to obtain a cross-modal implicit joint feature vector.

[0043] Here, a joint implicit feature capture network is a neural network architecture that can automatically and end-to-end learn a unified, shared feature representation space from multi-source or multimodal data. In this space, information from different sources but with similar semantics is mapped to nearby locations. For example, the Contrastive Language-Image Pre-training (CLIP) model is a joint implicit feature capture network based on contrastive learning. Its input consists of sample pairs of images and text, which are processed by an image encoder and a text encoder, outputting a unified image-text semantic space. The model can generate text descriptions for all category names and find the text features that are closest to the features of the image to be classified. Although the CLIP model does not have a deep cross structure, its contrastive learning objective itself is a powerful implicit fusion driver. To accomplish the task of "determining whether an image and text match," the two encoders must learn to map matching image-text pairs to nearby locations in the feature space.

[0044] Traditional multimodal fusion processes involve processing different modal vectors independently and then fusing them in the final stage. This approach disrupts intermodal communication during the crucial feature learning phase, preventing the model from fully exploring and utilizing the deep, complementary semantic information between different modalities. To address this, this disclosure employs a highly efficient model architecture—a joint implicit feature capture network—to perform an initial fusion at the beginning of image fusion. The initial fusion result is then optimized to obtain more accurate cross-modal implicit joint feature vectors, fully exploring the intrinsic correlation between the visible light and infrared modalities and improving the accuracy of semantic alignment and recognition. In subsequent image fusion processes, the implicit correlation between the visible light and infrared feature vectors is used as a guide to promote explicit feature fusion, thereby achieving deeper image information complementarity.

[0045] In a specific example, the cross-modal implicit joint feature vector can be calculated using the following formula 1. Here, a nonlinear transformation is performed using the hyperbolic tangent function (tanh), combined with an affine transformation using bias parameters and a weight matrix, to initially fuse the visible light and infrared feature vectors through position-point addition. Specifically, the fused visible light and infrared feature vectors are then passed through a neural network layer based on the hyperbolic tangent function to obtain the cross-modal implicit joint feature vector.

[0046] in, Represents the implicit joint feature vector across modalities; Represents the visible light feature vector; Represents the infrared feature vector; This represents the weight matrix of the neural network layers in the joint implicit feature capture network; This indicates the bias parameters of the neural network layers in the Joint Implicit Feature Capture Network. The weight matrix and bias parameters are used for affine transformation. This represents the hyperbolic tangent function, used for nonlinear transformations; This indicates a position-based addition operation, which adds the values ​​of the visible light feature vector and the infrared feature vector in the same dimension until all dimensions have been traversed to obtain a preliminary fusion vector. The position-based addition operation is used for preliminary fusion.

[0047] Since there are various joint implicit feature capture networks, this disclosure combines... A joint implicit feature capture network with synergistic effects is designed using functions, bias parameters, weight matrices, and point addition operations. The point addition operation acts as a low-overhead mixer, and the weight matrix acts as an intelligent processor. The function acts as a stabilizer and nonlinear introducer, resulting in computationally efficient joint features with compact dimensions and high information density. No manual design of fusion rules is required; through end-to-end training, the model proactively learns how to utilize weight matrices and biases to implicitly extract the most effective combinations and associated features between infrared and visible light modes from the point-added mixed information. Furthermore, The zero-centeredness and bounded output properties of the function, combined with the flexibility of linear transformation, make the above-mentioned joint implicit feature capture network more stable during training and less prone to gradient vanishing or exploding problems.

[0048] In embodiment S130, the extracted cross-modal implicit joint feature vector is subjected to a nonlinear transformation to obtain an implicit fused feature vector through feature activation. This nonlinear transformation emphasizes key information of the cross-modal implicit joint feature vector and suppresses irrelevant noise.

[0049] In a specific example, the sigmoid function is used to perform a nonlinear transformation on the extracted cross-modal implicit joint features, and the vector after nonlinear transformation is used as the implicit fusion feature vector. Specifically, the implicit fusion feature vector is obtained through the following formula 2: in, Represents the implicitly fused feature vector; This represents the Sigmoid function.

[0050] Involving S140, feature modulation based on implicit fusion feature vectors is performed on feature vectors of different modalities.

[0051] In this embodiment of the disclosure, step S140 is implemented through the following process. Figure 3 This is a schematic flowchart illustrating a vector modulation method provided in an embodiment of this disclosure. Figure 3 As shown, step S140 modulates the visible light feature vector and the infrared feature vector according to the degree of correlation between the visible light feature vector and the infrared feature vector and the implicit fusion feature vector, respectively, including steps S141 to S143.

[0052] First, in step S141, a first content contribution degree is determined based on the content contribution of the visible light feature vector to the implicit fusion feature vector, and in step S142, a second content contribution degree is determined based on the content contribution of the infrared feature vector to the implicit fusion feature vector.

[0053] In a specific example, the dot-division vector between the visible light feature vector and the implicitly fused feature vector is calculated, and the base-2 logarithm of the absolute values ​​of each feature value in the dot-division vector is calculated to obtain the semantic relevance representation vector. Then, the dot-product vector between the semantic relevance representation vector and the visible light feature vector is calculated, and the exponential function value with base e and the sum of the feature values ​​of the dot-product vector as the exponent is calculated to obtain the first content contribution. The second content contribution is calculated similarly.

[0054] In one implementation, the aforementioned first content contribution and second content contribution can be calculated using the following formulas. The first content contribution is calculated using formula 3, and the second content contribution is calculated using formula 4.

[0055] in, The first content contribution is used to characterize the content contribution of the visible light feature vector to the implicitly fused feature vector. This represents the i-th eigenvalue of the visible light eigenvector; This represents the i-th eigenvalue of the implicitly fused feature vector; This indicates the operation of an exponential function with base e; This represents the length of the visible light feature vector, and the visible light feature vector, infrared feature vector, and implicit fusion feature vector have the same length.

[0056] in, The second content contribution is used to characterize the content contribution of the infrared feature vector to the implicitly fused feature vector. This represents the i-th eigenvalue of the infrared eigenvector; This represents the i-th eigenvalue of the implicitly fused feature vector; This indicates the operation of an exponential function with base e; This represents the length of the infrared feature vector, and the visible light feature vector, infrared feature vector, and implicitly fused feature vector have the same length.

[0057] Next, in step 143, the first content contribution and the second content contribution are normalized to obtain the first feature weight and the second feature weight. The visible light feature vector is modulated with the first feature weight, and the infrared feature vector is modulated with the second feature weight.

[0058] In the specific example mentioned above, the first feature weight and the second feature weight can be calculated using the following formulas. Furthermore, the first feature weight is calculated using Formula 5, and the second feature weight is calculated using Formula 6.

[0059] in, This represents the weight of the first feature.

[0060] in, This represents the weight of the second feature.

[0061] Meanwhile, in the aforementioned specific example, the visible light feature vector and the infrared feature vector are weighted and modulated using normalized first and second feature weights. Specifically, the modulated visible light feature vector and infrared feature vector can be calculated using Equations 7 and 8, respectively.

[0062] in, Represents the visible light feature vector; This represents the modulated visible light feature vector.

[0063] in, Represents the infrared feature vector; This represents the modulated infrared feature vector.

[0064] By evaluating the correlation between the visible light feature vector and the infrared feature vector and the implicit fusion feature vector, the importance of the visible light feature vector and the infrared feature vector in representing the final explicit fusion representation vector is determined. Based on this importance, the original visible light feature vector and infrared feature vector are weighted and modulated, thereby effectively emphasizing the key information in both while suppressing irrelevant information and improving the accuracy of the final output.

[0065] This involves S150, a fine-grained, dynamic information retrieval process. Its fundamental purpose is to leverage the similarity between visible light and infrared feature vectors to enable the model to proactively learn where and how to extract the most useful and relevant information from another modality to supplement the deficiencies of the current modality. This is not simply adding two images from different modalities together, but rather selective enhancement. In this process, the model actively understands the semantic correspondence between the visible light and infrared images and significantly enhances key regions or targets, thus correctly associating them during fusion. For example, the outline of a "wheel" in the visible light image and the "heated wheel" region in the infrared image are semantically corresponding. If the wheel is the focus of this image fusion, its corresponding feature vector will be highlighted.

[0066] In this embodiment of the disclosure, S150 can be implemented through the following process: First, the modulated visible light feature vector and the modulated infrared feature vector are subjected to a dot product operation to obtain a similarity score. Then, the similarity score is adjusted according to the length of the infrared feature vector to obtain an attention score.

[0067] For example, this process is achieved through matrix multiplication. The normalized modulated visible light feature vector is transposed and multiplied with the normalized modulated infrared feature vector to obtain the attention score matrix. Each element in the attention score matrix is ​​derived considering contextual information and represents the correlation strength between a location in the visible light image and a location in the infrared image. For example, the feature vectors of regions that are very prominent thermal targets (such as pedestrians or vehicles) in the infrared image are highlighted. When calculating the similarity between the visible light feature vector and these regions, even though the same target may appear different in the infrared and visible light images, the model can learn the true correlation between them based on context and assign a higher attention score. Similarly, even though two targets may appear identical in the infrared and visible light images, the model can learn the true correlation between them based on context and assign a lower attention score.

[0068] Attention scores are determined based on the similarity between visible light and infrared feature vectors, upgrading the fusion process from "blind mixing" to "selective enhancement." This enriches details, highlights the target, and facilitates the generation of high-quality fused images. Furthermore, the attention scores can be visualized as attention heatmaps, allowing for a direct view of the enhanced and focused information when the model makes decisions, thus overcoming the black-box problem in deep learning.

[0069] Step S160 is implemented based on step S150. Step S150 has explicitly indicated the information that should be focused on during this fusion using the attention score. The implicit fusion feature vector is obtained by fusing visible light and infrared in the aforementioned process and contains the original information that needs to be extracted. However, some of this original information is not important to the current task. Therefore, it is necessary to extract key information from the implicit fusion feature vector based on the aforementioned attention score, and then highlight and condense this key information into a new vector, namely the explicit fusion representation vector.

[0070] Based on attention scores, the implicit fusion feature vector is converted into an explicit fusion representation vector. This filters out redundant or noisy information, focusing on the parts most relevant to the current task, resulting in a highly condensed and denoised fusion vector. Furthermore, attention scores yield two different explicit representation vectors in different contexts, thus the generated explicit fusion representation vector is adapted to the target image of the current task, thereby enhancing the generalization ability and accuracy of the fusion method disclosed herein.

[0071] In a specific example, the implicit fusion feature vector is extracted based on the attention score in the form of probability distribution and independent probabilities to obtain the explicit fusion representation vector. At this point, a first nonlinear transformation needs to be performed on the cross-modal implicit joint feature vector to obtain the probabilistic representation of the implicit fusion feature vector; simultaneously, a second nonlinear transformation needs to be performed on the attention score to obtain the probability distribution representation of the attention score; multiplying the probability distribution representation of the attention score with the probability representation of the implicit fusion feature vector yields the explicit fusion representation vector.

[0072] In the above implementation, the attention score and implicit fusion vector can be directly multiplied to obtain the explicit fusion representation vector by using probability distribution and independent probability. Regardless of the distance in the calculation process, information at any position in the implicit fusion feature vector can be directly accessed and extracted, thereby effectively solving the long-range dependency problem and improving data processing efficiency.

[0073] In one alternative implementation, the first nonlinear transformation is implemented using the Sigmoid function, i.e., Equation 2; and the second nonlinear transformation uses a normalization exponent (Equation 2). Function implementation.

[0074] Accordingly, the explicit fusion representation vector in S160 is obtained through the following formula 9.

[0075] in, This represents an explicit fusion representation vector; The function represents the second nonlinear transformation operation; This represents the modulated visible light feature vector; This represents the modulated infrared feature vector; Indicates transpose; Represents the implicitly fused feature vector; This represents the length of the infrared feature vector.

[0076] In another alternative implementation, the explicit fused representation vector can be obtained directly based on the feature saliency guided interaction module of the Transformer architecture. In other words, the modulated visible light feature vector, the modulated infrared feature vector, and the implicit fused feature vector are encoded using an attention-based cross-domain interactive fusion encoding method to obtain the explicit fused representation vector. Specifically, the modulated visible light feature vector can be used as the query vector, the modulated infrared feature vector as the key vector, and the implicit fused feature vector as the value vector. Based on the matching degree between the query vector and the key vector, the content of the value vector that is closer to the key vector is searched. The Transformer architecture is used to guide the interaction between features, thereby achieving fine-grained saliency fusion of cross-modal features to generate an explicit fused representation vector containing complementary information from multimodal images.

[0077] By employing multi-level feature extraction and attention-based cross-domain interactive fusion, this method ensures that image fusion preserves detailed information while maintaining the consistency of the overall image structure. Therefore, the technical solution provided in this disclosure has significant advantages in terms of image detail restoration and naturalness.

[0078] In S170, in this embodiment of the disclosure, random noise is introduced during the final image fusion to help the system learn the true distribution of the explicit fusion representation vector. At the same time, subsequent denoising is performed step by step to recover a clear image, which facilitates the generation of a high-quality fused image.

[0079] In a specific example, the fused image is obtained through the following process: First, the explicit fusion representation vector is encoded into an initial latent representation; then, noise is added to the initial latent representation to obtain a noisy latent representation; the noise in the noisy latent representation is predicted and removed to obtain an output latent representation; finally, the output latent representation is decoded to obtain the fused image.

[0080] In one alternative implementation, generating a fused image based on an explicit fusion representation vector includes: inputting the explicit fusion representation vector into a diffusion-based fusion image generator to obtain the fused image. During the forward pass of the diffusion model, random noise is progressively added to the sharp explicit fusion representation vector until it becomes completely meaningless noise. In the backward pass, a neural network is trained to learn how to progressively recover the sharp explicit fusion representation vector from the noise, and through iterative refinement, superior detail preservation and artifact suppression are achieved, resulting in a higher-quality fused image.

[0081] By actively introducing noise, the model can learn the essential ability to denoise, gain a deep understanding of the statistical laws and intrinsic structure of clear images under noise interference, and then generate more natural and higher quality fused images based on explicit fusion representation vectors.

[0082] This disclosure focuses on mining cross-modal implicit joint features. By mining the cross-modal implicit correlations between visible light image features and infrared image features, it effectively solves the problems of low image quality and insufficient information content in existing technologies for fused images. This guides the fusion of fine-grained explicit features between visible light and infrared images to intelligently generate fused images. This fusion method fully utilizes the potential correlations between visible light and infrared images, providing clearer, more detailed, and reliable image information, thereby improving the quality and information richness of the fused image to better meet the needs of practical applications. Furthermore, in this process, the modulation effect of implicit fusion feature vectors strengthens the explicit fusion between image features, and an attention mechanism ensures the accuracy of cross-domain interactive fusion coding. This not only improves the quality of the fused image but also enhances the usability and reliability of the image information.

[0083] Figure 4 This is a schematic diagram of the structure of a multimodal image fusion device provided in an embodiment of this disclosure. Figure 4 As shown, the device 200 may include a vector extraction module 210, a joint vector extraction module 220, an implicit feature fusion module 230, a vector modulation module 240, an attention score determination module 250, an explicit feature fusion module 260, and an image fusion module 270.

[0084] Vector extraction module 210 is used to extract visible light feature vectors and infrared feature vectors of the target image; The joint vector extraction module 220 is used to extract cross-modal implicit joint feature vectors of visible light feature vectors and infrared feature vectors based on the joint implicit feature capture network. The cross-modal implicit joint feature vectors are used to characterize the unoptimized implicit joint features shared by the visible light feature vectors and the infrared feature vectors. The implicit feature fusion module 230 is used to nonlinearly transform the cross-modal implicit joint feature vector to obtain the implicit fused feature vector; the implicit fused feature vector is used to characterize the implicit joint features shared by the visible light feature vector and the infrared feature vector; The vector modulation module 240 is used to modulate the visible light feature vector and the infrared feature vector according to the degree of correlation between the content of the visible light feature vector and the infrared feature vector and the implicit fused feature vector, respectively. Attention score determination module 250 is used to determine attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector; the attention score is used to characterize the correlation weight between the modulated visible light feature vector and the modulated infrared feature vector. The explicit feature fusion module 260 is used to extract the contents of the implicit fusion feature vector based on the attention score to obtain the explicit fusion representation vector; the explicit fusion representation vector is used to characterize the fusion feature representation obtained based on the attention mechanism; Image fusion module 270 is used to obtain a fused image based on an explicit fusion representation vector.

[0085] Optionally, the joint vector extraction module 220 is also used for: Based on the joint implicit feature capture network, the visible light feature vector and the infrared feature vector are initially fused to obtain the preliminary fused vector. The preliminary fused vector is then subjected to an affine transformation and a nonlinear transformation to obtain the cross-modal implicit joint feature vector.

[0086] Optionally, the joint vector extraction module 220 is also used to implement: The cross-modal implicit joint feature vector is calculated using the following formula 1: in, Represents the implicit joint feature vector across modalities; Represents the visible light feature vector; Represents the infrared feature vector; This represents the weight matrix of the neural network layers in the joint implicit feature capture network; This indicates the bias parameters of the neural network layers in the Joint Implicit Feature Capture Network. The weight matrix and bias parameters are used for affine transformation. This represents the hyperbolic tangent function, used for nonlinear transformations; This indicates a point-by-point operation, used for initial fusion.

[0087] Optionally, the vector modulation module 240 is also used for: The first content contribution is determined based on the contribution of the visible light feature vector to the implicit fusion feature vector, and the second content contribution is determined based on the contribution of the infrared feature vector to the implicit fusion feature vector. The contribution of the first content and the contribution of the second content are normalized to obtain the first feature weight and the second feature weight. The visible light feature vector is modulated with a first feature weight, and the infrared feature vector is modulated with a second feature weight.

[0088] Optionally, the vector modulation module 240 is also used for: The contribution of the first content is determined according to the following formula 3: in, The first content contribution is used to characterize the content contribution of the visible light feature vector to the implicitly fused feature vector. This represents the i-th eigenvalue of the visible light eigenvector; This represents the i-th eigenvalue of the implicitly fused feature vector; Indicates exponentiation; This represents the number of eigenvalues ​​in the visible light eigenvector. The contribution of the second content is determined according to the following formula 4: in, The second content contribution is used to characterize the content contribution of the infrared feature vector to the implicitly fused feature vector. This represents the i-th eigenvalue of the infrared eigenvector; This represents the i-th eigenvalue of the implicitly fused feature vector; Indicates exponentiation; This represents the number of eigenvalues ​​in the infrared eigenvector.

[0089] Optionally, the attention score determination module 250 is also used for: When determining attention scores based on the similarity between modulated visible light feature vectors and modulated infrared feature vectors... The similarity is obtained by performing a dot product operation between the modulated visible light feature vector and the modulated infrared feature vector; The similarity is adjusted by the length of the infrared feature vector to obtain the attention score. Optionally, the explicit feature fusion module 260 is also used for: When implementing the implicit joint eigenvector across modalities in nonlinear transformation to obtain the implicit fused eigenvector, A first nonlinear transformation is performed on the cross-modal implicit joint feature vector to obtain the probabilistic representation of the implicit fused feature vector; When extracting the implicit fusion feature vector based on attention score to obtain the explicit fusion representation vector, A second nonlinear transformation is performed on the attention score to obtain a probability distribution representation of the attention score; The explicit fusion representation vector is obtained by multiplying the probability distribution of the attention score with the probability representation of the implicit fusion feature vector.

[0090] Optionally, the explicit feature fusion module 260 is also used for: by When the function undergoes the second nonlinear transformation, the explicit fusion representation vector is obtained by the following formula: in, This represents an explicit fusion representation vector; The function represents the second nonlinear transformation operation; This represents the modulated visible light feature vector; This represents the modulated infrared feature vector; Indicates transpose; Represents the implicitly fused feature vector; This represents the length of the infrared feature vector.

[0091] Optionally, the image fusion module 270 is also used for: When obtaining a fused image based on an explicit fusion representation vector, Encode the explicit fusion representation vector into the initial latent representation; Add noise to the initial latent representation to obtain a noisy latent representation; predict and remove the noise from the noisy latent representation to obtain the output latent representation; The latent representation is decoded to obtain the fused image.

[0092] Figure 5 A schematic diagram of the hardware structure of the multimodal image fusion device provided in an embodiment of this disclosure is shown.

[0093] The multimodal image fusion device may include a processor 301 and a memory 302 storing computer program instructions.

[0094] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this disclosure.

[0095] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0096] In one instance, memory 302 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0097] Memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0098] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figures 1 to 3 The multimodal image fusion method in the illustrated embodiment.

[0099] In one example, the multimodal image fusion device may further include a communication interface 303 and a bus 304. Wherein, as... Figure 3 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 304 and complete communication with each other.

[0100] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this disclosure.

[0101] Bus 304 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 304 may include one or more buses. Although specific buses are described and illustrated in embodiments of this disclosure, this disclosure contemplates any suitable bus or interconnect.

[0102] Furthermore, in conjunction with the multimodal image fusion methods described in the above embodiments, this disclosure can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the multimodal image fusion methods described in the above embodiments.

[0103] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements any of the multimodal image fusion methods described in the above embodiments.

[0104] It should be clarified that this disclosure is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this disclosure is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this disclosure.

[0105] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this disclosure are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0106] It should also be noted that the exemplary embodiments mentioned in this disclosure describe methods or systems based on a series of steps or apparatus. However, this disclosure is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0107] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0108] The above description is merely a specific embodiment of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the protection scope of this disclosure.

Claims

1. A multi-modal image fusion method, characterized in that, The method comprises the following steps: extracting a visible light feature vector and an infrared feature vector of a target image; extracting an implicit fusion feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capturing network, the implicit fusion feature vector being used to represent implicit joint features shared by the visible light feature vector and the infrared feature vector; modulating the visible light feature vector and the infrared feature vector according to the content correlation degree of the visible light feature vector, the infrared feature vector and the implicit fusion feature vector respectively; determining an attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector, the attention score being used to represent the correlation weight between the modulated visible light feature vector and the modulated infrared feature vector; extracting the content of the implicit fusion feature vector based on the attention score to obtain an explicit fusion representation vector, the explicit fusion representation vector being used to represent the fusion feature representation obtained based on the attention mechanism; obtaining a fusion image based on the explicit fusion representation vector.

2. The image fusion method of claim 1, wherein, The method of extracting an implicit fusion feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capturing network comprises the following steps: extracting a cross-modal implicit joint feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capturing network, the cross-modal implicit joint feature vector being used to represent unoptimized implicit joint features shared by the visible light feature vector and the infrared feature vector; nonlinearly transforming the cross-modal implicit joint feature vector to obtain the implicit fusion feature vector.

3. The image fusion method of claim 2, wherein, The method of extracting a cross-modal implicit joint feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capturing network comprises the following steps: preliminarily fusing the visible light feature vector and the infrared feature vector based on a joint implicit feature capturing network to obtain a preliminary fusion vector, and affine transforming the preliminary fusion vector and then nonlinearly transforming it to obtain a cross-modal implicit joint feature vector.

4. The image fusion method of claim 3, wherein, The cross-modal implicit joint feature vector is calculated by the following formula: wherein, represents a cross-modal implicit joint feature vector; represents a visible light feature vector; represents an infrared feature vector; represents a weight matrix of a neural network layer in the joint implicit feature capturing network; represents a bias parameter of a neural network layer in the joint implicit feature capturing network, the weight matrix and the bias parameter being used to perform the affine transformation; represents a hyperbolic tangent function, used to perform the nonlinear transformation; represents a point-wise addition operation, used to perform the preliminary fusion.

5. The image fusion method of claim 2, wherein, The method of modulating the visible light feature vector and the infrared feature vector according to the content correlation degree of the visible light feature vector, the infrared feature vector and the implicit fusion feature vector respectively comprises the following steps: determining a first content contribution degree according to the content contribution of the visible light feature vector to the implicit fusion feature vector, and determining a second content contribution degree according to the content contribution of the infrared feature vector to the implicit fusion feature vector; normalizing the first content contribution degree and the second content contribution degree to obtain a first feature weight and a second feature weight; modulating the visible light feature vector with the first feature weight, and modulating the infrared feature vector with the second feature weight.

6. The image fusion method of claim 5, wherein, The first content contribution degree is determined according to the following formula 1: wherein, denotes the first content contribution degree, and is used to represent the content contribution of the visible light feature vector to the implicit fusion feature vector; denotes the i th feature value of the visible light feature vector; denotes the i th feature value of the implicit fusion feature vector; denotes the exponential operation; denotes the length of the visible light feature vector; The second content contribution degree is determined according to the following formula 2: wherein, represents the second content contribution degree, used to represent the content contribution of the infrared feature vector to the implicit fusion feature vector; represents the i-th eigenvalue of the infrared feature vector; represents the i-th eigenvalue of the implicit fusion feature vector; represents the exponential operation; represents the length of the infrared feature vector.

7. The image fusion method of claim 2, wherein, The method of determining an attention score based on the similarity between the modulated visible light feature vector and the modulated infrared feature vector comprises the following steps: The modulated visible light feature vector and the modulated infrared feature vector are dot product operated to obtain a similarity; The similarity is adjusted by a length of the infrared feature vector to obtain the attention score.

8. The image fusion method of claim 7, wherein, The cross-modal implicit joint feature vector is nonlinearly transformed to obtain an implicit fusion feature vector, including: The cross-modal implicit joint feature vector is first nonlinearly transformed to obtain a probability representation of the implicit fusion feature vector; The content of the implicit fusion feature vector is extracted based on the attention score to obtain an explicit fusion representation vector, including: The attention score is second nonlinearly transformed to obtain a probability distribution representation of the attention score; The probability distribution representation of the attention score is multiplied by the probability representation of the implicit fusion feature vector to obtain the explicit fusion representation vector.

9. The image fusion method of claim 8, wherein, In The explicit fusion representation vector is obtained by the following equation when the second nonlinear transformation is performed by the function wherein, represents an explicit fusion representation vector; a function, representing the second non-linear transformation operation; represents a modulated visible light feature vector; represents a modulated infrared feature vector; represents a transpose; represents an implicit fusion feature vector; represents a length of the infrared feature vector.

10. The image fusion method of claim 1, wherein, The explicit fusion representation vector is used to obtain a fusion image, including: The explicit fusion representation vector is encoded into an initial latent representation; Noise is added to the initial latent representation to obtain a noisy latent representation; the noise of the noisy latent representation is predicted and removed to obtain an output latent representation; The output latent representation is decoded to obtain a fusion image.

11. A multi-modal image fusion apparatus, characterized by, The device includes: A vector extraction module is configured to extract a visible light feature vector and an infrared feature vector of a target image; A joint vector extraction module is configured to extract an implicit fusion feature vector of the visible light feature vector and the infrared feature vector based on a joint implicit feature capture network, the implicit fusion feature vector being used to represent an implicit joint feature shared by the visible light feature vector and the infrared feature vector; A vector modulation module is configured to modulate the visible light feature vector and the infrared feature vector according to a content correlation degree of the visible light feature vector, the infrared feature vector, and the implicit fusion feature vector; An attention score determination module is configured to determine an attention score based on a similarity between the modulated visible light feature vector and the modulated infrared feature vector, the attention score being used to represent a correlation weight between the modulated visible light feature vector and the modulated infrared feature vector; An explicit feature fusion module is configured to extract content of the implicit fusion feature vector based on the attention score to obtain an explicit fusion representation vector, the explicit fusion representation vector being used to represent a fusion feature representation obtained based on an attention mechanism; An image fusion module is configured to obtain a fusion image based on the explicit fusion representation vector.

12. A multi-modal image fusion device, characterized by, The device includes a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the multi-modal image fusion method of any one of claims 1-10.

13. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, which are executed by a processor to implement the multi-modal image fusion method of any one of claims 1-10.

14. A computer program product, characterised in that, The computer program is executed by a processor to implement the multi-modal image fusion method of any one of claims 1-10.

Images