A deep learning-based cross-modal image fusion method

Abstract

The application discloses a cross-modal image fusion method based on deep learning, relates to computer vision and image processing technology, and aims to solve the problems of structural distortion, texture loss and high calculation overhead in existing image fusion methods. The application first introduces a state space model, which has linear complexity when modeling long-range dependencies, greatly reduces the calculation overhead, improves the system running efficiency while ensuring the fusion quality, and is suitable for deployment and application in resource-constrained scenarios. The application also automatically extracts difference and complementary information through dynamic feature enhancement and cross-modal fusion mechanism, effectively avoiding the details omission or information bias caused by manual feature selection in traditional methods. Based on the scheme of the application, the image information from different modalities can be fully fused to automatically generate a fusion image with stronger detail retention and structural consistency, which is convenient for subsequent target detection, medical diagnosis and other tasks, and improves the image use efficiency and intelligent level.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to a cross-modal image fusion method based on deep learning. BACKGROUND

[0002] Target detection technology is one of the core tasks in the field of computer vision, and is widely used in automatic driving, intelligent security, unmanned aerial vehicle monitoring, medical image diagnosis, industrial automation detection and many other scenes. Through automatic identification and positioning of objects of interest in images or videos, environmental perception, anomaly detection and analysis decision are realized, thereby improving production efficiency, enhancing safety protection, and assisting the landing application of intelligent systems.

[0003] In 2023, Stanford University and State Space team proposed a high-efficiency sequence processing model-Mamba. Based on the principle of SSM (State Space Model), the model first extracts features from sequence data through a selective state space mechanism to capture local and global information in the sequence. Then, a gating mechanism is used to adaptively fuse state space features, effectively controlling information flow. Finally, Mamba uses a linear complexity state update method to generate an output sequence, significantly improving the computational efficiency and model performance of long sequence tasks.

[0004] The Mamba model has limited performance in cross-modal image fusion tasks. Specifically, the traditional Mamba structure is only suitable for a single input modality, and lacks an effective mechanism to capture deep feature interactions between cross-modal data, making it difficult for the model to fully fuse and utilize complementary information in heterogeneous data, limiting its application effect in complex tasks such as multi-modal fusion and cross-modal target detection. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a cross-modal image fusion method based on deep learning, which solves the problem of low utilization efficiency of feature information and loss of feature information in the process of multi-modal image fusion in the prior art.

[0006] The technical solution adopted by the present application to solve the above technical problem is:

[0007] A cross-modal image fusion method based on deep learning, comprising the following steps:

[0008] Step 1: Obtain a pair of image samples of different modalities, unify the image size and format, and input them into an image feature extraction network of a multi-layer encoding-decoding structure;

[0009] Step 2: The image feature extraction network extracts local and global features from image samples of different modalities through a dynamic visual state space module combined with dynamic local convolution and efficient channel attention mechanism.

[0010] Step 3: Through the dynamic feature fusion module, which includes a dynamic feature enhancement module and a cross-modal fusion Mamba module, dynamic interaction and difference feature enhancement between image features of different modalities are realized, and further correlation information between cross-modalities is mined to obtain fused image features. Then the decoder generates a fused image;

[0011] Step 4: Training with a joint loss function to obtain a trained image feature extraction network, which is used in image fusion tasks.

[0012] Step 2 is as follows: First, build an image feature extraction network, which is structurally inspired by the symmetric design of the encoding and decoding of the U-Net network, and each layer contains two dynamic visual state space modules;

[0013] The dynamic visual state space module includes a state space modeling unit, a dynamic local convolution unit, and a channel attention mechanism unit, which are connected in series, and the units are connected through skip connections to pass features; each unit is as follows:

[0014] The state space modeling unit uses a state space model to extract global context features, and converts the state space formula into a discrete one-dimensional convolution form:

[0015] ;

[0016] wherein is a one-dimensional convolution kernel, is a one-dimensional sequence length, represents a convolution operation, is an intermediate feature map after one-dimensional convolution; is an image sample embedding representation, is a state transition matrix, is an input projection matrix, is an output projection matrix, is an identity mapping matrix;

[0017] Before the intermediate feature map after one-dimensional convolution is processed further, it is first scanned sparsely by the sparse two-dimensional space modeling ES2D.

[0018] The dynamic local convolution unit enhances texture details through dynamic local convolution. Dynamic local convolution includes ordinary convolution and learnable convolution, which is realized by setting the relative position offset in the local neighborhood and the learnable convolution kernel parameters;

[0019] The channel attention mechanism unit uses an efficient channel attention mechanism to enhance key channel expression; first, the intermediate feature map after one-dimensional convolution After the second intermediate feature is extracted by the efficient state space modeling module, it is input into the efficient channel attention mechanism after layer normalization operation, and the obtained result is added to the result of the dynamic local convolution unit and the second intermediate feature to obtain the final output feature map.

[0020] The step 3 is specifically as follows:

[0021] Step 3.1: Enhance modal interaction through a dynamic feature fusion module, which is expressed by the following formula:

[0022] ;

[0023] ;

[0024] Wherein represents element-level addition, represents element-level multiplication, is a difference-enhanced feature of two modal image samples in the nth layer, respectively represent the feature maps input by two modal image samples in the nth layer for subsequent difference-enhanced processing, respectively are the output feature maps of two modal image samples in the nth layer after processing by the dynamic visual state space module, which includes the texture expression feature enhanced by the dynamic local convolution, is a Sigmoid function, is a global average pooling operation, represents the original image feature The coarse-grained fusion feature processed by the global average pooling and the Sigmoid function;

[0025] Step 3.2: The cross-modal fusion Mamba module generates a fusion image feature through two-stage feature fusion;

[0026] In the first stage, two difference-enhanced features respectively undergo layer normalization, full connection linear transformation, and deep convolution in sequence, and then the results obtained by the two are element-level fused to obtain a mixed fusion feature ;

[0027] In the second stage, the fusion image feature is generated based on context modeling and attention mechanism, which is expressed by the following formula:

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] in, For efficient 2D spatial scanning operations, For fully connected linear transformation operations, For high-efficiency channel attention modules, , This represents the two semantically fused representations after modal alignment. For the final fused feature map, The fused image features are the output of the nth layer;

[0033] Step 3.3: Input the fused image features into the decoding path, which includes multiple decoder modules; each decoder module corresponds to a level of fused image features; each level of fused image features first undergoes spatial dimension restoration through the Patch expansion module, and then is sequentially input into the corresponding decoder module for feature reconstruction and upsampling, finally generating a target fused image with the same size as the original image.

[0034] Furthermore, the joint loss function includes intensity loss, texture loss, structural similarity loss, perceptual loss, and edge consistency loss.

[0035] The beneficial effects of this invention are:

[0036] (1) Based on the solution of the present invention, image information from different modalities can be fully integrated to automatically generate a fused image with stronger detail preservation and structural consistency, which facilitates subsequent target detection, medical diagnosis and other tasks, and improves image utilization efficiency and intelligence level.

[0037] (2) This invention does not require manual setting of fusion rules or key regions for images. Through dynamic feature enhancement and cross-modal fusion mechanism, it can automatically extract differential and complementary information, effectively avoiding the problem of detail omission or information bias caused by manual feature selection in traditional methods.

[0038] (3) The introduced state space model has linear complexity when modeling long-distance dependencies, which greatly reduces the computational overhead compared with the Transformer structure based on self-attention mechanism. It improves the system operating efficiency while ensuring the fusion quality, and is suitable for deployment and application in resource-constrained scenarios. Attached Figure Description

[0039] Figure 1 This is a flowchart of the model training process in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the algorithm principle for model training in an embodiment of the present invention;

[0041] Figure 3 This is a flowchart of the model prediction process in an embodiment of the present invention. Detailed Implementation

[0042] This invention aims to propose a deep learning-based cross-modal image fusion method to address the problems of detail loss and structural distortion in fused images caused by limitations in local receptive fields or excessive computational complexity in existing technologies. The method includes a training process and a prediction process.

[0043] In the training process, firstly, multimodal image sample pairs are acquired and preprocessed, with image sizes unified and converted to tensor format. Next, each modal image is input into a designed multi-layer visual state space network. State space modeling is used to extract image features with long-range dependencies, and a dynamic local convolution mechanism is introduced to enhance texture representation capabilities. Simultaneously, a channel attention mechanism is used to suppress redundant features. Then, the enhanced modal features are input into a fusion module, which consists of a dynamic feature enhancement unit and a cross-modal fusion unit. This module extracts intermodal difference and shared information, improving the complementarity and consistency of the fused representation. Finally, a joint loss function including brightness loss, texture loss, and structural similarity loss is constructed. Using the similarity between the fused image and the source image as a supervision signal, gradient descent is employed to optimize network parameters, resulting in a well-trained image fusion model.

[0044] In the prediction process, the multimodal images to be fused are first preprocessed; then, the images are input into the trained model, which automatically extracts local details and global semantic features using the learned cross-modal modeling capabilities; next, the interaction and reconstruction of modal information are completed through the fusion module; finally, a fused image result with clear structure and rich details is output, which can be directly used for downstream tasks such as object detection or medical image segmentation, improving the overall task performance and application effect.

[0045] Example 1:

[0046] A deep learning-based cross-modal image fusion method includes the following steps:

[0047] A. Training process, such as Figure 2 As shown:

[0048] A1. Acquire sample data and preprocess it to obtain image sample pairs of different modalities, and standardize the image size and format;

[0049] A11. Collect multimodal image sample pairs to ensure that the images have been registered;

[0050] A12. Perform grayscale conversion and uniform size scaling on the image samples, and convert them to tensor input format to obtain the image input feature pairs. , .

[0051] A2. By designing a dynamic visual state space module, combining dynamic local convolution and efficient channel attention mechanism, local and global features are extracted from image samples of different modalities respectively; wherein, the dynamic visual state space module includes a state space modeling unit, a dynamic local convolution unit and a channel attention mechanism unit, which are connected in series.

[0052] The state-space modeling unit is used to model the long-range dependencies of image features; the dynamic local convolution unit generates learnable convolution kernels based on the input image content to enhance texture details; the channel attention mechanism unit adaptively adjusts weights based on channel responses to highlight key semantic channels. Features can be passed between multiple modules through skip connections, improving the hierarchy and consistency of the fused representation.

[0053] A21. Construct a multi-layered encoder-decoder image feature extraction network. Structurally, it borrows the symmetrical encoder-decoder design of the U-Net network. Each layer contains two dynamic visual state space modules, and the number of channels increases progressively with each layer. Specifically, the numbers are 64, 128, 256, 512, and 1024 respectively, with the number of modules per layer being... Compared to the traditional U-Net, this scheme embeds state-space modeling and dynamic convolution mechanisms in each layer to enhance long-range dependency modeling capabilities and detail preservation. The feature extraction path extracts local and global semantic information layer by layer, while the reconstruction path restores the image structure layer by layer. Both embed multiple dynamic visual state-space modules to extract deep features. Subsequent steps will elaborate on the specific role of this network structure in the feature extraction and fusion process.

[0054] A22. The state-space modeling unit uses a state-space model to extract global context features. The basic formula for state space is:

[0055]

[0056] in For the input sequence at time step The input vector is specifically the embedded representation of the image after non-overlapping image patch segmentation, that is, the image segment features obtained after converting the two-dimensional image into a one-dimensional vector sequence, which are used for subsequent state modeling. The hidden state vector. Let be the derivative of the hidden state vector with respect to time. The output for the current time step. Here is the state transition matrix. For the input projection matrix, To output the projection matrix, The identity mapping matrix is ​​optional and can be discretized into an RNN (Recurrent Neural Network) structure:

[0057]

[0058] in For discrete time step subscripts, This represents the hidden state vector at discrete time step k. To be in discrete time steps The input image embedding can be further represented as a one-dimensional convolutional form:

[0059]

[0060] in A one-dimensional convolution kernel, The length of the one-dimensional sequence. This represents the convolution operation. This is the intermediate feature map after one-dimensional convolution;

[0061] A23. Before proceeding to the next step, the input features can be sparsely scanned using the ES2D (Sparse Two-Dimensional Space Modeling) module. This module receives the intermediate feature map output from the state space modeling unit as input, adopts a sparse connection strategy to improve the modeling efficiency between long-distance pixels, and passes the modeled feature results to the dynamic local convolution unit for subsequent texture enhancement processing.

[0062] A24. The dynamic local convolution unit enhances texture details through dynamic local convolution. The definition of dynamic local convolution is:

[0063]

[0064] in, For dynamic local convolution, This is the current pixel position. For learnable control weights, For local neighborhood The relative position offset in For learnable convolution kernel parameters, For in position The input feature map values, For a learnable descriptive mask, the first term in the formula is a regular convolution, and the second term is a learnable convolution;

[0065] A25. The channel attention mechanism unit uses an efficient channel attention mechanism to enhance the representation of key channels. First, the second intermediate feature is extracted by the efficient state space modeling module. Then, after layer normalization, it is input into the efficient channel attention mechanism. The result is added to the dynamic convolution result and the intermediate feature to obtain the final output feature map. The processing of the channel attention mechanism unit in the nth layer can be represented by the following formula:

[0066]

[0067]

[0068] In the nth layer of the network, These are intermediate features extracted using an efficient state-space module. This is an efficient state space modeling module used to model long-range dependencies of image features. It includes an input projection unit, a state transition unit, and an output mapping unit. For layer-standardized operations, This is the Efficient Channel Attention module, used for allocating channel weights. This represents the intermediate feature map extracted by the state space modeling unit, which serves as the input to the subsequent efficient channel attention module and dynamic local convolution module; This is the final output feature map of the dynamic vision state space module.

[0069] A3. Through the dynamic feature fusion module, which includes a dynamic feature enhancement module and a cross-modal fusion Mamba module, dynamic interaction and differential feature enhancement between image features of different modalities are achieved, and cross-modal correlation information is further mined; specifically including:

[0070] A31. Construct a dynamic feature fusion module to enhance modal interaction. The definition of the dynamic feature enhancement module is as follows:

[0071]

[0072]

[0073] in This represents element-wise addition. Represents element-wise multiplication. To represent the difference enhancement features between two modal image samples in the nth layer, These represent the feature maps of the two modality image sample inputs in the nth layer, respectively, and are used for subsequent differential enhancement processing. These are the output feature maps of two modal image samples in the nth layer after processing by the dynamic visual state space module, including texture representation features enhanced by dynamic local convolution. It is the Sigmoid (non-linear activation) function. This is a global average pooling operation. Representing the features of the original image Coarse-grained fusion features processed by global average pooling and the sigmoid function are used to generate differential enhancement weights. The definition of the first-stage feature generation for cross-modal fusion is as follows:

[0074]

[0075]

[0076]

[0077] in, For depthwise convolution operations, For fully connected linear transformation operations, , Let be the depth features of two modal image samples in the nth layer. It is a hybrid fusion feature;

[0078] The definition of the second-stage context modeling and attention mechanism is as follows:

[0079]

[0080]

[0081]

[0082]

[0083] in For efficient 2D spatial scanning operations, , This is a semantic fusion representation after modality alignment. For the final fused feature map, The output of the nth layer is fused with image features;

[0084] A32. The fused image features are input into the decoding path, which includes multiple decoder modules; each decoder module corresponds to a level of fused feature representation. Each fused feature is first processed by the Patch expansion module for spatial dimension restoration, and then sequentially input into the corresponding decoder module for feature reconstruction and upsampling, ultimately generating a target fused image with the same size as the original image.

[0085] A4. Using the defined joint loss function (including intensity loss, texture loss, structural similarity loss, perceptual loss, and edge consistency loss), the quality of the fused image is used as supervision information. The Adam optimizer is used to optimize the network parameters through gradient descent to obtain a trained model.

[0086] Supervised training is performed using a multi-objective loss function and an adaptive loss weight learning mechanism, specifically including:

[0087] A41. The intensity loss function is defined as follows:

[0088]

[0089] in The intensity loss measures the difference in intensity between the fused image and the reference image. Given two modal image samples as the original input, To obtain the fused image, express Norm, The image height and width;

[0090] A42. Texture loss function, defined as follows:

[0091]

[0092] in For texture loss, this loss measures the fidelity of the fused image compared to the input image at the level of texture detail. This is an image gradient operation used to extract edge texture information. express Norm, used to calculate the sum of absolute errors;

[0093] A43. The structural similarity loss function is defined as follows:

[0094]

[0095] in The structural similarity loss is used to calculate the similarity between the fused image and the reference image at the perceptual semantic level. It is a structural similarity index used to measure structural fidelity;

[0096] A44. The perceptual loss function is defined as follows:

[0097]

[0098] in This is a perceptual loss, used to calculate the similarity between the fused image and the reference image at the perceptual semantic level. For a fixed-parameter perceptron network, parameters are frozen during training, with VGG-19 used by default. To indicate that input X is in the network The first in Layer feature maps are used to measure high-level semantic information. To avoid confusion with intermediate features of the backbone network, It is used only for loss calculation and does not share parameters with the backbone encoder / decoder, nor does it participate in forward feature fusion.

[0099] A45. The edge consistency loss function is defined as follows:

[0100]

[0101] in This is the edge consistency loss, which is used to constrain the consistency between the fused image and the input image in edge regions, helping to preserve structural contour information. This indicates an edge extraction operation;

[0102] A46. The total loss function using an adaptive loss weight learning mechanism is defined as follows:

[0103]

[0104] in , where i is the index of each of the above loss functions. The learnable log-variance function corresponds to each loss term. This mechanism allows the network to automatically learn the importance of each loss term during training without manually setting the loss weights.

[0105] A47. Use the Adam optimizer to optimize the network parameters using gradient descent to obtain a trained model.

[0106] B. Prediction process:

[0107] B1. Preprocess the multimodal image data to be fused, including image registration, grayscale conversion, and size unification.

[0108] B2. Input the preprocessed image data into the trained multi-layer dynamic visual state space module to extract the local and global feature representations of each modality image;

[0109] B3. Input the extracted multimodal image features into the trained dynamic feature fusion module, which includes a dynamic feature enhancement module and a cross-modal fusion module, to model the image difference information and correlation, and generate fused features;

[0110] B4. Input the fusion features into the trained reconstruction module and output the final fused image result.

[0111] Example 2:

[0112] The deep learning-based cross-modal image fusion method in this embodiment includes two parts: a training process and a prediction process. These will be described in detail below with reference to the accompanying drawings:

[0113] See Figure 1 The training process includes:

[0114] S1. Sample Data Acquisition and Preprocessing:

[0115] S11. Obtain multimodal image sample pairs, such as infrared images and visible light images, CT images and MRI images, GFP images and PC images, etc.

[0116] S12. Perform uniform preprocessing on the image samples, scaling all images to 256×256 pixels and converting them to grayscale image format; S13. Standardize the image data size, such as scaling them to 448x448 pixels.

[0117] S13. Align the processed images according to modality and construct standard training sample pairs for input into the network.

[0118] S2. Construct a multi-layer fusion neural network to extract and fuse multimodal image features, including the following processes:

[0119] (1) The modal feature extraction process is as follows:

[0120] S211. Construct a multi-layered image feature extraction network;

[0121] The network adopts a U-Net framework, with each layer containing multiple dynamic visual state space modules, and the number of channels increasing layer by layer. The number of modules per layer is ;

[0122] S212. After the image is input, non-overlapping patch segmentation is first performed. Then, the initial feature maps of each modality image are extracted by the network encoder. The state space mechanism is used to model the long-range dependency, and a dynamic local convolution mechanism is introduced to enhance local texture details.

[0123] S213. A channel attention mechanism is used to model the importance of different channels, suppress useless channels, and obtain an effective representation feature map of the multimodal image.

[0124] S214. After completing multimodal fusion at each scale, the obtained fused features are combined with shallow and deep features through skip connections to improve the integrity of semantic information, and then input into the decoder for structural reconstruction.

[0125] S215. Each layer of feature maps is normalized and arranged into a sequence, and then positional encoding is added as input for subsequent fusion processing.

[0126] (2) The modeling and fusion process of intermodal differences is as follows:

[0127] S221. Input the image features of the two modalities into the dynamic feature enhancement module respectively, perform texture information enhancement and difference feature perception, and obtain the difference enhancement feature map;

[0128] S222. Input the above results into the cross-modal fusion module. First, perform deep convolution and construct hybrid features. Then, obtain long-range dependencies through state space modeling. Finally, use the channel attention mechanism to extract key fusion features.

[0129] S223. The fusion result is residually connected to the original feature map and used as input for image reconstruction in the decoder.

[0130] S3. Construct the loss function and perform supervised training.

[0131] S31. Construct loss functions between the fused image and each modal image, including intensity loss, texture loss and structural similarity loss, which are used to measure brightness fidelity, edge detail and structural consistency, respectively.

[0132] S32. Calculate each loss function and the total loss function.

[0133] S4. Using the Adam optimizer and backpropagation algorithm, train the above neural network parameters to finally obtain the trained multimodal image fusion model.

[0134] The prediction process flow in this embodiment is shown below. Figure 3 It includes:

[0135] S1. Data Preprocessing:

[0136] In this step, the image pairs to be fused are preprocessed, including grayscale conversion, image registration, uniform scaling to 256×256 pixels, and conversion to tensor format.

[0137] S2. Image Feature Extraction:

[0138] In this step, the two modal images are input into the trained multi-layer state-space model to extract the feature representations after local enhancement and global modeling.

[0139] S3, Fusion Generation:

[0140] In this step, the aforementioned feature maps are input into the trained differential enhancement module and cross-modal fusion module to complete the fine interaction and fusion feature extraction between modalities. The fusion result is used to reconstruct the image by the decoder, and finally outputs the fused image. The fused image retains the key structural, texture and functional information in the two modal images and can be used for downstream medical image analysis or target detection tasks.

[0141] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A cross-modal image fusion method based on deep learning, characterized in that, Includes the following steps: Step 1: Obtain image sample pairs of different modalities, unify the image size and format, and input them into a multi-layer encoder-decoder structure image feature extraction network; Step 2: The image feature extraction network extracts local and global features from image samples of different modalities by using a dynamic visual state space module, combined with dynamic local convolution and an efficient channel attention mechanism. Step 3: Through the dynamic feature fusion module, which includes a dynamic feature enhancement module and a cross-modal fusion Mamba module, dynamic interaction and differential feature enhancement between image features of different modalities are realized, and cross-modal correlation information is further mined to obtain fused image features; then the decoder generates the fused image. Step 4: Train the image feature extraction network using the joint loss function to obtain the trained image feature extraction network, and then use the trained image feature extraction network in the image fusion task; Step 2 is as follows: First, build an image feature extraction network. The structure is based on the symmetrical encoding and decoding design of the U-Net network. Each layer contains two dynamic visual state space modules. The dynamic visual state space module comprises a state space modeling unit, a dynamic local convolution unit, and a channel attention mechanism unit, which are connected in series. Features are transferred between units through skip connections. The specific details of each unit are as follows: The state-space modeling unit uses a state-space model to extract global context features, transforming the state-space formula into a discretized one-dimensional convolutional form: ； in A one-dimensional convolution kernel, The length of the one-dimensional sequence. This represents the convolution operation. This is the intermediate feature map after one-dimensional convolution; For image sample embedding representation, Here is the state transition matrix. For the input projection matrix, To output the projection matrix, It is the identity mapping matrix; The dynamic local convolution unit enhances texture details through dynamic local convolution. Dynamic local convolution includes ordinary convolution and learnable convolution. Learnable convolution is achieved by setting the relative position offset in the local neighborhood and learnable convolution kernel parameters. The channel attention mechanism unit uses an efficient channel attention mechanism to enhance the representation of key channels; firstly, the intermediate feature map after one-dimensional convolution... The second intermediate feature is extracted by the efficient state space modeling module, and then after layer standardization, it is input into the efficient channel attention mechanism. The result is added to the result of the dynamic local convolution unit and the second intermediate feature to obtain the final output feature map.

2. The cross-modal image fusion method based on deep learning according to claim 1, characterized in that, Before the intermediate feature map after one-dimensional convolution is processed in the next step, it is first sparsely scanned by ES2D sparse two-dimensional spatial modeling.

3. The cross-modal image fusion method based on deep learning according to claim 2, characterized in that, Step 3 is as follows: Step 3.1: Enhance modal interaction through the dynamic feature fusion module, as shown in the following formula: ； ； in This represents element-wise addition. Represents element-wise multiplication. To represent the difference enhancement features between two modal image samples in the nth layer, These represent the feature maps of the two modality image sample inputs in the nth layer, respectively, and are used for subsequent differential enhancement processing. These are the output feature maps of two modal image samples in the nth layer after processing by the dynamic visual state space module, including texture representation features enhanced by dynamic local convolution. For the Sigmoid function, This is a global average pooling operation. Representing the features of the original image Coarse-grained fusion features processed by global average pooling and the Sigmoid function; Step 3.2: Cross-modal fusion. The Mamba module generates fused image features through two-stage feature fusion. In the first stage, two differential enhancement features The results are sequentially processed through layer normalization, fully connected linear transformation, and depthwise convolution. Then, the results from both are fused element-wise to obtain the hybrid fused feature. ; In the second stage, fused image features are generated based on context modeling and attention mechanisms, as expressed by the following formula: ； ； ； ； in, For efficient 2D spatial scanning operations, For fully connected linear transformation operations, For high-efficiency channel attention modules, , This represents the two semantically fused representations after modal alignment. For the final fused feature map, The fused image features are the output of the nth layer. Standardized operations for layers; Step 3.3: Input the fused image features into the decoding path, which includes multiple decoder modules; each decoder module corresponds to a level of fused image features; The fused image features at each level are first restored to their spatial dimensions by the Patch extension module, and then sequentially input into the corresponding decoder modules for feature reconstruction and upsampling, ultimately generating a target fused image with the same size as the original image.

4. The cross-modal image fusion method based on deep learning according to claim 3, characterized in that, The joint loss function includes intensity loss, texture loss, structural similarity loss, perceptual loss, and edge consistency loss.

5. The cross-modal image fusion method based on deep learning according to claim 4, characterized in that, The formula for calculating the strength loss is as follows: ； in For strength loss, Given two modal image samples as the original input, To obtain the fused image, express Norm, The image height and width; The formula for calculating the texture loss is as follows: ； in For texture loss, This is an image gradient operation used to extract edge texture information. express Norm, The formula for calculating the structural similarity loss is as follows: ； in For structural similarity loss, It is a structural similarity index; The formula for calculating the perceived loss is as follows: ； in In order to perceive loss, For a fixed-value perceptron network, the parameters are frozen during training. To represent the input X in the perceptual network The first in Layer feature map, Used only for this perceptual loss calculation; The formula for calculating the edge consistency loss is as follows: ； in For edge consistency loss, This indicates an edge extraction operation; Finally, an adaptive loss weight learning mechanism is used to construct the total loss function: ； in , where i is the index of each of the above loss functions. The learnable log-variance function corresponding to each loss term.

Images