Multi-modal image fusion method based on Mama and attention mechanism

Abstract

The invention relates to the technical field of image processing, and discloses a multi-modal image fusion method based on Mama and an attention mechanism. According to the method, through cooperative work of multiple modules, the multi-modal image fusion effect is greatly improved, the LDC-pyramid adaptive feature fusion module is combined with the SURF algorithm, multi-scale features are comprehensively and stably extracted, the adaptability to complex scenes is enhanced, the SCIEAM module weighs the features from channel and space dimensions, key features of pathogens are highlighted, and the robustness of the multi-modal image fusion is improved. The VSS block effectively captures local and global features and accurately fuses information of different regions and scales, the multi-modal feature fusion and refining module deeply optimizes features by using an SS2D mechanism and an MLP and excavates potential relations among multi-modal features, the introduced VSS block is based on the advantage of Mama, the calculation cost is reduced, the reasoning speed is increased, the whole fusion process is more efficient, and the fusion efficiency is improved. A fusion image obtained through processing of the multi-scale residual error feature reconstruction module can clearly present pathogen information, and the livestock and poultry disease prevention and control level is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal image fusion method based on Mamba and attention mechanism. Background Art

[0002] Multimodal image fusion aims to combine the strengths of different modalities to generate more informative images, which is of great significance in the field of livestock and poultry pathogen detection. Currently, single-modality detection has limitations. For example, images from fluorescent quantitative digital PCR instruments can accurately quantify nucleic acids but cannot reveal pathogen morphology, while images from serpentine microfluidic synthetic imaging devices can reveal microscopic details but lack nucleic acid quantification. Fusion of these two methods will help improve the diagnosis and prevention of livestock and poultry diseases.

[0003] In recent years, numerous multimodal image fusion methods have emerged, including CNNs, automatic neural networks (AEs), generative adversarial networks (GANs), diffusion models, and Transformer-based methods. However, these methods struggle to achieve both efficiency and effectiveness. CNNs are limited by their receptive field, and AEs have difficulty designing encoders. Generative model-based methods generate high-quality images, but suffer from inefficiencies such as unstable training and long training times. Transformer-based methods are computationally expensive, and even improvements to them fail to address fundamental issues.

[0004] Mamba's selective mechanism and efficient hardware-aware design provide new solutions to the above challenges, and Mamba has been proven to outperform Transformer in tasks that require long-term dependency modeling. This is due to its input adaptation and global information modeling capabilities, while maintaining linear complexity, reducing computational costs and improving inference speed. VSSblock extends the concept of Mamba to the field of visual data or multimodal data processing. VSS block solves the challenges brought by the two-dimensional characteristics of visual data by introducing bidirectional scanning mechanisms and cross-scanning mechanisms, enabling the model to capture dependency information in different directions of the image. This is an important expansion of the application scenarios of the Mamba module. However, the role of Mamba in multimodal image fusion tasks has not been fully explored because it lacks a design similar to cross-attention. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a multimodal image fusion method based on Mamba and attention mechanism, which has the advantages of accurately obtaining livestock and poultry pathogen information and solving the defects of single modality data, thus solving the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal image fusion method based on Mamba and attention mechanism, comprising the following steps:

[0007] S1: Input original images, including fluorescence images generated by fluorescence quantitative digital PCR instrument and microscopic images generated by serpentine microfluidic synthetic imaging device;

[0008] S2: The initial features of the original image are input through the multimodal image feature extraction module, and the enhanced features F1 and F2 are output;

[0009] The multimodal image feature initial extraction module includes an LDC-pyramid adaptive feature fusion module, a SURF module and a SCIEAM module;

[0010] The LDC-pyramid adaptive feature fusion module extracts multi-scale features in parallel through LDC convolution layers with different dilation rates and global average pooling;

[0011] The SURF module detects interest points and generates feature descriptors by constructing the Hessian matrix and scale space;

[0012] The SCIEAM module is used to receive the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module, perform weighted fusion after splicing them in the channel dimension, and output enhanced features;

[0013] S3: Through multimodal feature interaction and deepening module, two channel exchanges are performed, and two VSS blocks are set between the two channel exchanges to deepen the features F1 and F2 of the two feature maps in S2 and obtain the deepened features;

[0014] S4: Input the deepened features obtained in S3 into the multimodal feature fusion and refinement module, and use the SS2D selective scanning mechanism and MLP in the multimodal feature fusion and refinement module to perform feature fusion and optimization to obtain optimized features;

[0015] S5: The optimized features obtained in S4 are input into the multi-scale residual feature reconstruction module, and the final fused image is generated through multi-scale convolution extraction, SS2D enhancement and residual connection.

[0016] As a preferred technical solution of the present invention, the LDC-pyramid adaptive feature fusion module is provided with four LDC convolution layers with different dilation rates and a global average pooling layer to process the original image in parallel, and the features output by the four LDC convolution layers with different dilation rates and the global average pooling layer are spliced ​​in the channel dimension, and then the multi-scale fusion features are generated by 1×1 convolution dimensionality reduction;

[0017] The specific processing steps of the SURF module are:

[0018] Detect candidate interest points through the Hessian matrix and perform non-maximum suppression in the scale space;

[0019] Determine the main direction of the interest point through Haar wavelet response and generate a feature descriptor with scale and rotation invariance;

[0020] The specific steps of the SCIEAM module include: parallel calculation of the channel descriptors and spatial descriptors of the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module after splicing in the channel dimension, decomposing the channel descriptors and spatial descriptors into new channel and spatial weights through MLP after splicing, activating the weights with the Sigmoid function, and then performing weighted fusion on the features in the new channel and spatial weights respectively.

[0021] As a preferred technical solution of the present invention, the multimodal feature interaction and deepening module performs the following operations based on the enhanced features of the output of S2: performing a first channel exchange operation on the enhanced features, and then inputting them into a VSSblock for processing. After the processing is completed, a second channel exchange operation is performed, and the features are input into another VSS block for processing, and the two VSS blocks have the same structure.

[0022] As a preferred technical solution of the present invention, the multimodal feature fusion and refinement module has the following specific steps:

[0023] S401: Perform layer normalization on the deepening features output in S3;

[0024] S402: Use LDC convolution to process the output of S401, and concatenate the outputs of the two LDC convolutions in the channel dimension. Then use SS2D to process the concatenated features, and use a multi-layer perceptron to process the output of SS2D.

[0025] S403: Fusing the result of the layer normalization operation in S401 with the output of S402 through pixel-by-pixel multiplication;

[0026] S404: Add the processing result of S403 and the output of S402 pixel by pixel to obtain the optimized feature.

[0027] As a preferred technical solution of the present invention, the specific steps of the multimodal feature fusion and refinement module are as follows:

[0028] S501: The optimized features obtained in S4 are fed into three parallel convolutional layers respectively;

[0029] S502: concatenate and integrate the outputs of the three parallel convolutional layers in the channel dimension to generate multi-scale fusion features;

[0030] S503: Input the multi-scale fusion feature into SS2D for feature enhancement, and perform residual connection between the output of SS2D and the output of S802;

[0031] S504: upsample the outputs of S503 and S4, add them element by element, and perform dimension reduction on the result of the addition before outputting it.

[0032] Compared with the existing technology, the present invention provides a multimodal image fusion method based on Mamba and attention mechanism, which has the following beneficial effects:

[0033] The present invention greatly improves the multimodal image fusion effect through the collaborative work of multiple modules. The LDC-pyramid adaptive feature fusion module combines with the SURF algorithm to comprehensively and stably extract multi-scale features, enhancing adaptability to complex scenes. The SCIEAM module weights these features from the channel and spatial dimensions to highlight the key features of pathogens. The VSSblock effectively captures local and global features and accurately integrates information from different regions and scales. The multimodal feature fusion and refinement module uses the SS2D mechanism and MLP deep optimization features to explore the potential connections between multimodal features. The introduced VSS block is based on the advantages of Mamba, reduces computing costs, improves inference speed, and makes the entire fusion process more efficient. Finally, the fused image obtained by processing the multi-scale residual feature reconstruction module can clearly present pathogen information and greatly improve the level of livestock and poultry disease prevention and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flowchart of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the multimodal image feature initial extraction module;

[0036] Figure 3 Schematic diagram of the LDC-pyramid adaptive feature fusion module structure;

[0037] Figure 4 Schematic diagram of the SCIEAM module structure;

[0038] Figure 5 Schematic diagram of the multimodal feature interaction and deepening module structure;

[0039] Figure 6 This is a schematic diagram of the VSS block structure;

[0040] Figure 7 This is a schematic diagram of the module structure of the multimodal feature fusion and refinement module;

[0041] Figure 8 Schematic diagram of the feature reconstruction module structure. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] See also Figure 1-8 , a multimodal image fusion method based on Mamba and attention mechanism, including the following steps:

[0044] S1: Input original images, including fluorescence images generated by a fluorescence quantitative digital PCR instrument and microscopic images generated by a serpentine microfluidic synthetic imaging device, with 3 channels and an initial scale of 256×256 pixels;

[0045] S2: The initial features of the original image are input through the multimodal image feature extraction module, and the enhanced features F1 and F2 are output;

[0046] The multimodal image feature initial extraction module includes an LDC-pyramid adaptive feature fusion module, a SURF module and a SCIEAM module;

[0047] The LDC-pyramid adaptive feature fusion module extracts multi-scale features in parallel through LDC convolution layers with different dilation rates and global average pooling;

[0048] The SURF module detects interest points and generates feature descriptors by constructing the Hessian matrix and scale space;

[0049] The SCIEAM module is used to receive the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module, perform weighted fusion after splicing them in the channel dimension, and output enhanced features;

[0050] S201 , extracting features from the original image using both the input LDC-pyramid adaptive feature fusion module and the SURF module.

[0051] By introducing pyramid pooling ASPP with LDC as convolution, the multi-scale features of the image can be extracted more comprehensively. When fusing multimodal images, the multi-scale information about livestock and poultry pathogens in images of different modalities can be better integrated, which is expected to improve the quality of the fused image, thereby providing more accurate and richer information for pathogen detection, and improving the accuracy and reliability of detection.

[0052] The SURF algorithm can quickly and stably extract salient feature points and generate robust feature descriptors. These features complement the output of the LDC-pyramid adaptive feature fusion module. Its scale and rotation invariance, as well as its robustness to illumination changes, ensure accurate feature extraction from livestock and poultry pathogen images acquired under diverse conditions, improving the fusion model's adaptability in complex scenarios.

[0053] The structure of the LDC-pyramid adaptive feature fusion module is shown in the figure below. Figure 3 As shown in the figure, the specific processing steps are as follows: in the LDC-pyramid adaptive feature fusion module, four LDC convolution layers with different dilation rates and a global average pooling are set to process the image in parallel. The convolution kernel size of the first LDC convolution layer is 1×1 and the dilation rate is 1; the second convolution kernel size is 3×3 and the dilation rate is 12; the third convolution kernel size is 3×3 and the dilation rate is 24; the third convolution kernel size is 3×3 and the dilation rate is 36. In this way, convolution layers with different dilation rates can capture different scale features of the image. For example, when the dilation rate is 1, local details are focused, and when the dilation rate is 3, a wider range of context information is obtained. Each set of feature maps obtained by LDC convolution and global average pooling with different dilation rates is spliced ​​in the channel dimension. The number of channels of the obtained feature map becomes 5×256, and the scale remains 256×256 pixels. Finally, a 1×1 convolution is used to reduce the dimension of the spliced ​​feature map so that the number of channels of the output image is reduced to 256.

[0054] The specific steps of S2 are as follows:

[0055] S201: The original image enters the LDC-pyramid adaptive feature fusion module for processing, specifically:

[0056] First, four LDC convolutional layers with different dilation rates are used in parallel with a global average pooling layer. The dilated convolution outputs with different dilation rates and the global average pooling output are concatenated in the channel dimension. A 1×1 convolution is then used to perform feature aggregation and dimensionality reduction on the concatenated outputs of the dilated convolution outputs with different dilation rates and the global average pooling output in the channel dimension, forming a comprehensive feature representation containing rich multi-scale information.

[0057] S202: The original image is processed into the SURF module to obtain 320-dimensional features, and then the spliced ​​features are input into the SCIEAM module for processing;

[0058] The specific steps of SURF module processing are as follows:

[0059] S202(1): Construct Hessian matrix and generate scale space: For each pixel in the image, construct the Hessian matrix to detect candidate interest points. The Hessian matrix is ​​used to describe the local second-order derivative information of the image. The matrix elements are calculated by convolving the Gaussian second-order derivative with the image. The Hessian matrix is ​​calculated at different scales to generate the scale space to achieve scale invariance. The expression of the Hessian matrix H is:

[0060]

[0061] Among them, L xx (x,y,δ),L xy (x,y,δ),L yy (x, y, δ) and are the values ​​of the Gaussian second-order differential approximated by the box filter at the point (x, y) at scale δ;

[0062] S202(2): Interest point detection: In the scale space, perform non-maximum suppression on each pixel in its 8 neighborhoods and 26 neighborhoods of adjacent scales to find the local maximum points of the Hessian matrix determinant value. These points are candidate interest points.

[0063] S202(3): Interest point positioning and screening: Perform sub-pixel precision positioning of candidate interest points, and determine their precise locations by fitting a three-dimensional quadratic function. At the same time, remove interest points with low contrast and those located on the edge to improve the stability and robustness of feature points. Assuming that the value of the Hessian matrix determinant at point X = (x, y, δ) is D(X), its quadratic Taylor expansion is:

[0064]

[0065] Among them, D(X0) is the gradient of the function at X0, H is the Hessian matrix, and by solving More accurate key point locations can be obtained;

[0066] S202(4): Determine the main direction of the point of interest: Calculate the Haar wavelet response within a certain size neighborhood centered on the point of interest. Calculate the response values ​​in the horizontal and vertical directions, and determine the main direction by summing the responses within the fan-shaped area, so that the feature description is rotationally invariant.

[0067] For a pixel (X0), the horizontal Haar wavelet response d x and the vertical Haar wavelet response d y They are:

[0068] d x =I(x+s,y)-I(xs,y)

[0069] d y =I(x,y+s)-I(x,ys)

[0070] Where S is the size of the wavelet template;

[0071] S202(5): Generate feature descriptors: With the point of interest as the center, rotate the coordinate axis to the main direction, select a square area around it, and divide it into several sub-areas. Calculate the Haar wavelet response in each sub-area, and obtain the horizontal and vertical response values ​​and the sum of their absolute values, and finally combine them into a 64-dimensional SURF feature descriptor;

[0072] S203: The features extracted by the LDC-pyramid adaptive feature fusion module and SURF are concatenated in the channel dimension, and the concatenated features are input into the SCIEAM (Spatial-Channel Interaction Enhanced Attention Module) for processing;

[0073] S204: Input the output feature map of S203 into the SCIEAM module for processing, first calculate the channel feature map and the spatial feature map of the input image in parallel, and then splice and fuse the channel feature map and the spatial feature map in the channel dimension;

[0074] The specific steps of SCIEAM module processing are:

[0075] S204(1): First, the input image is subjected to global average pooling and global maximum pooling operations in the spatial dimension, and the two vectors obtained by average pooling and maximum pooling are input into a shared multi-layer perceptron (MLP) respectively. Finally, the two results after MLP processing are added together to obtain a channel descriptor;

[0076] S204(2): Calculate the spatial descriptor. First, perform average pooling and maximum pooling operations on the input image along the channel dimension, and then input the two vectors obtained by average pooling and maximum pooling into a shared multi-layer perceptron (MLP). Finally, the two results after MLP processing are added together to obtain the channel descriptor.

[0077] S204(3): reshape the spatial descriptor into a dimension matching the channel descriptor, concatenate the two, and then input them into a shared multi-layer perceptron (MLP);

[0078] S204(4): The reshape function is again used to decompose the new channel description and spatial descriptor from the output of S204(3).

[0079] S204(5): After activating the new channel description and spatial descriptor respectively using the Sigmoid function, the channel description and spatial descriptor are multiplied by the image originally input to the SCIEAM module to obtain a channel feature weighted map and a spatial feature weighted map.

[0080] S204(6): The channel feature weighted map and spatial feature weighted map obtained in S204(5) are input to the linear layer, and finally a 1×1 convolution is used to further extract and integrate the linearly transformed output;

[0081] S3: Through the multimodal feature interaction and deepening module, two channel exchanges are performed, and two VSS blocks are set between the two channel exchanges to deepen the features of the two feature map features F1 and F2 in S2, and obtain the deepened features. The structure diagram of the multimodal feature interaction and deepening module is shown in Figure 5 The specific steps are as follows:

[0082] S301: First, the output of S2 undergoes the first channel swap operation and then is input into a VSS block for processing;

[0083] The two feature maps obtained from the previous processing are features and features, respectively. Their number of channels is C, height is H, and width is W. The specific operation of exchanging channels is as follows:

[0084] Divide F1 and F2 into four equal parts along the channel dimension.

[0085]

[0086] Among them, [:,:,:,:] represents the slicing operation on different channel dimensions of the data. Respectively represent the values ​​from 0 to to to Slices to the C channel, Respectively represent the values ​​from 0 to to to Slices to C channels, where C is the total number of channels;

[0087] Will Sequentially concatenate in the channel dimension to generate a new feature map F new1 ;Will Sequentially concatenate in the channel dimension to generate a new feature map Fnew2 ;

[0088] The formula for the newly generated feature map is as follows:

[0089]

[0090] in, Indicates splicing in the channel dimension;

[0091] S302, repeat the operation of S301 again;

[0092] The VSS block structure is as follows Figure 6 As shown, the specific processing steps are as follows:

[0093] S301(1): Normalize the image after channel exchange (LayerNorm).

[0094] S301(2): The linear layer processed data from S301(1) is input into the depthwise separable convolution, with the depthwise convolution kernel size of 3×3 and the pointwise convolution kernel size of 1×1.

[0095] S301(3): The output of S301(2) is activated and input into the 2D-selective scanning (SS2D) module to perform selective scanning on the feature map. After 2D-selective scanning, the feature map scale remains 256×256 pixels and the number of channels remains 320.

[0096] S301: (4) Multiply the output of S301(1) after linear layer processing and activation by the output of S301(3) after normalization element by element.

[0097] S301(5): The output of S301(4) is processed by the linear layer and then element-wise added to the image initially input to the VSS block to obtain the final output;

[0098] S4: Input the deepened features obtained in S3 into the multimodal feature fusion and refinement module, and use the SS2D selective scanning mechanism and MLP in the multimodal feature fusion and refinement module to fuse and optimize the features to obtain optimized features, and further deeply fuse and refine the features. Its structure diagram is as follows Figure 7 As shown;

[0099] S401: Perform layer normalization on the deepening features output in S3;

[0100] S402: The output of S401 is processed using a 3×3 LDC convolution kernel, and the outputs of the two LDC convolutions are spliced ​​in the channel dimension. Then, the spliced ​​features are processed using SS2D, and the output of SS2D is processed using a multi-layer perceptron (MLP);

[0101] S403: Fusing the result of the layer normalization operation in S401 with the output of S402 through pixel-by-pixel multiplication;

[0102] S404: Add the processing result of S403 and the output of S402 pixel by pixel to obtain the optimized feature;

[0103] S5: Input the optimized features obtained in S4 into the multi-scale residual feature reconstruction module, and generate the final fused image through multi-scale convolution extraction, SS2D enhancement and residual connection;

[0104] S501: The output of S4 is fed into three parallel convolutional layers: one is a 1×1 convolution with a dilation rate of 1, and the other two are 3×3 convolutions with dilation rates of 2 and 4, respectively. This allows features to be extracted from different scales.

[0105] S502: The outputs of the three parallel convolutional layers are concatenated in the channel dimension, and then 1×1 convolution is used to integrate features of different scales to generate multi-scale fusion features;

[0106] S503: Input the multi-scale fusion features output by S502 into SS2D for feature enhancement, and add the output of SS2D and the output of S502 element by element;

[0107] S504: S504, upsample the outputs of S503 and S4, and add them element by element. Finally, use a 3×3 convolution to reduce the dimension of the result of the addition to 3 to obtain the final output.

[0108] Therefore, the present invention adopts the aforementioned multimodal image fusion method based on Mamba and the attention mechanism. It utilizes multiple modules to achieve efficient multimodal image fusion. The LDC-pyramid adaptive feature fusion module, in conjunction with the SURF algorithm, comprehensively and stably extracts multiscale features and enhances scene adaptability. The SCIEAM module weightedly optimizes features to highlight key pathogen information. The VSS block captures local and global features for precise fusion. The multimodal feature fusion and refinement module deeply optimizes features. Finally, the VSS block improves computational efficiency. The resulting fused image clearly displays pathogen information, significantly improving livestock and poultry disease prevention and control.

[0109] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal image fusion method based on Mamba and attention mechanism, characterized by: The following steps are involved: S1: Input original images, including fluorescence images generated by a fluorescence quantitative digital PCR instrument and microscopic images generated by a serpentine microfluidic synthetic imaging device; S2: The initial features of the original image are input through the multimodal image feature extraction module, and the enhanced features F1 and F2 are output; The multimodal image feature initial extraction module includes an LDC-pyramid adaptive feature fusion module, a SURF module and a SCIEAM module; The LDC-pyramid adaptive feature fusion module extracts multi-scale features in parallel through LDC convolution layers with different dilation rates and global average pooling; The SURF module detects interest points and generates feature descriptors by constructing the Hessian matrix and scale space; The SCIEAM module is used to receive the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module, perform weighted fusion after splicing them in the channel dimension, and output enhanced features; S3: Through multimodal feature interaction and deepening module, two channel exchanges are performed, and two VSS blocks are set between the two channel exchanges to deepen the features F1 and F2 of the two feature maps in S2 and obtain the deepened features; S4: Input the deepened features obtained in S3 into the multimodal feature fusion and refinement module, and use the SS2D selective scanning mechanism and MLP in the multimodal feature fusion and refinement module to perform feature fusion and optimization to obtain optimized features; S5: The optimized features obtained in S4 are input into the multi-scale residual feature reconstruction module, and the final fused image is generated through multi-scale convolution extraction, SS2D enhancement and residual connection.

2. The multimodal image fusion method based on Mamba and attention mechanism according to claim 1, characterized in that: The LDC-pyramid adaptive feature fusion module sets four LDC convolution layers with different dilation rates and a global average pooling layer to process the original image in parallel, concatenates the features output by the four LDC convolution layers with different dilation rates and the global average pooling layer in the channel dimension, and generates multi-scale fusion features through 1×1 convolution dimension reduction; The specific processing steps of the SURF module are: Detect candidate interest points through the Hessian matrix and perform non-maximum suppression in the scale space; Determine the main direction of the interest point through Haar wavelet response and generate a feature descriptor with scale and rotation invariance; The specific steps of the SCIEAM module include: parallel calculation of the channel descriptors and spatial descriptors of the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module after splicing in the channel dimension, decomposing the channel descriptors and spatial descriptors into new channel and spatial weights through MLP after splicing, activating the weights with the Sigmoid function, and then performing weighted fusion on the features in the new channel and spatial weights respectively.

3. The multimodal image fusion method based on Mamba and attention mechanism according to claim 2, characterized in that: The multimodal feature interaction and deepening module performs the following operations based on the enhanced features of the output of S2: performing a first channel swap operation on the enhanced features, then inputting them into a VSS block for processing, performing a second channel swap operation after the processing is completed, and inputting them into another VSS block for processing, and the two VSS blocks have the same structure.

4. The multimodal image fusion method based on Mamba and attention mechanism according to claim 1, characterized in that: The specific steps of the multimodal feature fusion and refinement module are as follows: S401: Perform layer normalization on the deepening features output in S3; S402: Use LDC convolution to process the output of S401, and concatenate the outputs of the two LDC convolutions in the channel dimension. Then use SS2D to process the concatenated features, and use a multi-layer perceptron to process the output of SS2D. S403: Fusing the result of the layer normalization operation in S401 with the output of S402 through pixel-by-pixel multiplication; S404: Add the processing result of S403 and the output of S402 pixel by pixel to obtain the optimized feature.

5. The multimodal image fusion method based on Mamba and attention mechanism according to claim 1, characterized in that: The specific steps of the multimodal feature fusion and refinement module are as follows: S501: The optimized features obtained in S4 are fed into three parallel convolutional layers respectively; S502: concatenate and integrate the outputs of the three parallel convolutional layers in the channel dimension to generate multi-scale fusion features; S503: Input the multi-scale fusion feature into SS2D for feature enhancement, and add the output of SS2D and the output of S502 element by element; S504: upsample the outputs of S503 and S4, add them element by element, and perform dimension reduction on the result of the addition before outputting it.

Images