A multimodal image fusion method based on Mamba and attention mechanism

Abstract

This invention relates to the field of image processing technology and discloses a multimodal image fusion method based on Mamba and attention mechanisms. This method significantly improves the multimodal image fusion effect through the collaborative work of multiple modules. The LDC-pyramid adaptive feature fusion module, combined with the SURF algorithm, comprehensively and stably extracts multi-scale features, enhancing adaptability to complex scenes. The SCIEAM module weights these features from channel and spatial dimensions, highlighting key pathogen features. The VSS block effectively captures local and global features, accurately fusing information from different regions and scales. The multimodal feature fusion and refinement module utilizes the SS2D mechanism and MLP deep feature optimization to mine potential connections between multimodal features. The introduced VSS block, based on the advantages of Mamba, reduces computational costs and improves inference speed, making the entire fusion process more efficient. The fused image processed by the multi-scale residual feature reconstruction module clearly presents pathogen information, significantly improving the level of livestock and poultry disease prevention and control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal image fusion method based on Mamba and attention mechanism. BACKGROUND

[0002] Multi-modal image fusion aims to combine the advantages of different modal images to generate images with more information, which is of great significance in the field of livestock pathogen detection. Current single modal detection has limitations, such as fluorescent quantitative digital PCR instrument images can accurately quantify nucleic acids but cannot present pathogen morphology, and snake-shaped microfluidic synthesis imaging device images can display microscopic details but lack nucleic acid quantification, and the fusion of the two can help improve the level of livestock disease diagnosis and prevention and control;

[0003] In recent years, there are many multi-modal image fusion methods, including CNNs, AEs, GANs, diffusion models and Transformer-based methods, but they are difficult to balance efficiency and effectiveness. CNNs are limited by the receptive field, and AEs are difficult to design encoders; the image quality generated by the generation model-based method is high, but there are efficiency problems such as unstable training and long time; the calculation cost of the Transformer-based method is too high, and even if it is improved, the fundamental problem has not been solved;

[0004] Mamba provides a new solution to the above challenges with its selective mechanism and efficient hardware-aware design, and Mamba has been proven to be superior to Transformer in tasks that require long-term reliance on modeling. This is due to its input adaptation and global information modeling capabilities, while maintaining linear complexity, reducing computational cost and improving inference speed. VSS block is an extension of Mamba's concept to the field of visual data or multi-modal data processing. VSS block solves the challenges brought by the two-dimensional nature of visual data by introducing a bidirectional scanning mechanism and a cross-scanning mechanism, enabling the model to capture dependent information in different directions of the image. This is an important expansion of the application scenario of Mamba modules, however, the role of Mamba in multi-modal image fusion tasks has not been fully explored because it lacks a cross-attention-like design. SUMMARY

[0005] In view of the shortcomings of the prior art, the present application provides a multi-modal image fusion method based on Mamba and attention mechanism, which has the advantages of accurately obtaining livestock pathogen information and solving the defects of single modal data, and solves the above technical problems.

[0006] To achieve the above purpose, the present application provides the following technical scheme: a multi-modal image fusion method based on Mamba and attention mechanism, comprising the following steps:

[0007] S1: input the original image, specifically including the fluorescent image generated by the fluorescent quantitative digital PCR instrument and the microscopic image generated by the serpentine microfluidic synthesis imaging device;

[0008] S2: output the enhanced features F1 and F2 through the preliminary features of the original image input by the multi-modal image feature preliminary extraction module;

[0009] The multi-modal image feature preliminary extraction module comprises an LDC-pyramid adaptive feature fusion module, a SURF module and a SCIEAM module;

[0010] The LDC-pyramid adaptive feature fusion module extracts multi-scale features in parallel through LDC convolution layers with different hole rates and global average pooling;

[0011] The SURF module detects interest points in the scale space and generates feature descriptors by constructing a Hessian matrix;

[0012] The SCIEAM module is used to receive the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module after being spliced in the channel dimension, and perform weighted fusion, outputting enhanced features;

[0013] S3: two channel exchanges are performed through the multi-modal feature interaction and deepening module, and two VSS blocks are set between the two channel exchanges to deepen the features of the two feature maps F1 and F2 in S2, and the deepened features are obtained;

[0014] S4: input the deepened features in S3 into the multi-modal feature fusion and refining module, and perform feature fusion and optimization using the SS2D selective scanning mechanism and MLP in the multi-modal feature fusion and refining module to obtain optimized features;

[0015] S5: input the optimized features in S4 into the multi-scale residual feature reconstruction module, and generate the final fused image through multi-scale convolution extraction, SS2D enhancement and residual connection.

[0016] As a preferred technical scheme of the present application, four LDC convolution layers with different hole rates and one global average pooling layer are arranged in parallel in the LDC-pyramid adaptive feature fusion module to process the original image, and after splicing the features output by the four LDC convolution layers with different hole rates and the global average pooling layer in the channel dimension, multi-scale fusion features are generated through 1x1 convolution dimension reduction;

[0017] The specific processing steps of the SURF module are as follows:

[0018] Detect candidate interest points through the Hessian matrix, and perform non-maximum suppression in the scale space;

[0019] The main direction of the interest point is determined by a Haar wavelet response, and a feature descriptor with scale and rotation invariance is generated;

[0020] The specific steps of the SCIEAM module include: parallel computing of channel descriptors and spatial descriptors of features after channel dimension splicing of features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module, splicing of the channel descriptors and the spatial descriptors, decomposition of the spliced channel descriptors and spatial descriptors into new channel and spatial weights through the MLP, weighting fusion of features in the new channel and spatial weights after activation of the weights using the Sigmoid function.

[0021] As a preferred technical solution of the application, the multi-modal feature interaction and deepening module performs the following operations on the enhanced features based on the output of S2: first channel exchange operation on the enhanced features, then input to a VSS block processing, second channel exchange operation after the processing is completed, and input to another VSS block processing, and the two VSS block structures are consistent.

[0022] As a preferred technical solution of the application, the multi-modal feature fusion and refining module has the following specific steps:

[0023] S401: performing layer normalization operation on the deepening features output in S3;

[0024] S402: processing the output of S401 using LDC convolution, splicing the outputs of the two LDC convolutions in the channel dimension, then processing the spliced features using SS2D, processing the output of SS2D using a multi-layer perceptron;

[0025] S403: fusing the result after the layer normalization operation of S401 and the output of S402 through pixel-by-pixel multiplication;

[0026] S404: pixel-by-pixel addition of the processing result after S403 processing and the output of S402 to obtain optimized features.

[0027] As a preferred technical solution of the application, the multi-modal feature fusion and refining module has the following specific steps:

[0028] S501: the optimized features obtained in S4 are respectively input into three parallel convolution layers;

[0029] S502: splicing and integrating the outputs of the three parallel convolution layers in the channel dimension to generate multi-scale fusion features;

[0030] S503: input the multi-scale fusion feature into SS2D for feature enhancement, and perform residual connection on the output of SS2D and the output of S802;

[0031] S504: upsample the outputs of S503 and S4, add them element by element, and output the result after dimension reduction.

[0032] Compared with the prior art, the application provides a multi-modal image fusion method based on Mamba and attention mechanism, which has the following beneficial effects:

[0033] The application greatly improves the multi-modal image fusion effect through the cooperative work of multiple modules, the LDC-pyramid adaptive feature fusion module combined with the SURF algorithm comprehensively and stably extracts multi-scale features, enhances the adaptability to complex scenes, the SCIEAM module weights these features from the channel and spatial dimensions, highlights the key features of pathogens, the VSS block effectively captures local and global features, accurately fuses different regional and scale information, the multi-modal feature fusion and refining module optimizes the features using the SS2D mechanism and MLP, excavates the potential relationship between multi-modal features, introduces the VSS block based on the advantages of Mamba, reduces the calculation cost, improves the inference speed, makes the whole fusion process more efficient, and finally, the fusion image obtained through the multi-scale residual feature reconstruction module can clearly present the pathogen information, greatly improves the level of livestock and poultry disease prevention and control. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the application;

[0035] Figure 2 is a structural diagram of the multi-modal image feature initial extraction module;

[0036] Figure 3 is a structural diagram of the LDC-pyramid adaptive feature fusion module;

[0037] Figure 4 is a structural diagram of the SCIEAM module;

[0038] Figure 5 is a structural diagram of the multi-modal feature interaction and deepening module;

[0039] Figure 6 is a structural diagram of the VSS block;

[0040] Figure 7 is a structural diagram of the multi-modal feature fusion and refining module;

[0041] Figure 8 is a structural diagram of the feature reconstruction module. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0043] Please refer to Figures 1-8 A multi-modal image fusion method based on Mamba and attention mechanism, comprising the following steps:

[0044] S1: input original images, specifically, fluorescent images generated by a fluorescent quantitative digital PCR instrument and microscopic images generated by a snake-shaped microfluidic synthesis imaging device, the channels are 3, and the initial scales are all 256*256 pixels;

[0045] S2: output enhanced features F1 and F2 through the preliminary features of the original images input by a multi-modal image feature preliminary extraction module;

[0046] The multi-modal image feature preliminary extraction module comprises an LDC-pyramid adaptive feature fusion module, a SURF module and a SCIEAM module;

[0047] The LDC-pyramid adaptive feature fusion module extracts multi-scale features in parallel through LDC convolution layers with different hole rates and global average pooling;

[0048] The SURF module detects interest points and generates feature descriptors by constructing a Hessian matrix and a scale space;

[0049] The SCIEAM module is used for receiving the features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module after channel dimension splicing, performing weighted fusion, and outputting enhanced features;

[0050] S201: simultaneously use the input LDC-pyramid adaptive feature fusion module and the SURF module to extract features from the original images.

[0051] By introducing the pyramid pooling ASPP with LDC convolution, multi-scale features of the images can be more comprehensively extracted, and in multi-modal image fusion, multi-scale information about livestock and poultry pathogens in different modal images can be better integrated, which is expected to improve the quality of the fused images, thereby providing more accurate and rich information for pathogen detection and improving the precision and reliability of the detection.

[0052] The SURF algorithm can quickly and stably extract significant feature points and generate robust feature descriptors, which are complementary to the output of the LDC-pyramid adaptive feature fusion module. The scale and rotation invariance and the robustness to light changes of the LDC-pyramid adaptive feature fusion module can ensure that the features of the livestock and poultry pathogen images obtained under different conditions can be accurately extracted, and the adaptability of the fusion model in complex scenes can be improved.

[0053] The structure diagram of the LDC-pyramid adaptive feature fusion module is shown in Figure 3 The specific processing steps are as follows: in the LDC-pyramid adaptive feature fusion module, four LDC convolution layers with different hole rates and one global average pooling are set in parallel to process the image. The first LDC convolution layer has a convolution kernel size of 1x1 and a hole rate of 1; the second convolution kernel size is 3x3, and the hole rate is 12; the third convolution kernel size is 3x3, and the hole rate is 24; the third convolution kernel size is 3x3, and the hole rate is 36. In this way, the convolution layers with different hole rates can capture different scale features of the image, for example, the hole rate of 1 focuses on local details, and the hole rate of 3 obtains more context information in a larger range. Each group of feature maps obtained by the LDC convolution with different hole rates and the global average pooling is spliced in the channel dimension, and the number of channels of the obtained feature maps becomes 5x256, and the scale remains 256x256 pixels. Finally, a 1x1 convolution is used to reduce the dimension of the spliced feature maps, so that the number of channels of the output image is reduced to 256.

[0054] The specific steps of S2 are as follows:

[0055] S201: The original image enters the LDC-pyramid adaptive feature fusion module for processing, specifically:

[0056] First, four LDC convolution layers with different hole rates and one global average pooling are used in parallel to process the image. The outputs of the convolution layers with different hole rates and the global average pooling are spliced in the channel dimension. A 1x1 convolution is used to aggregate and reduce the dimension of the output after splicing the outputs of the convolution layers with different hole rates and the global average pooling in the channel dimension, to form a comprehensive feature representation containing rich multi-scale information.

[0057] S202: The original image enters the SURF module for processing to obtain 320-dimensional features, and then the spliced features are input into the SCIEAM module for processing.

[0058] The specific steps of the SURF module processing are as follows:

[0059] S202(1): Constructing Hessian matrix and generating scale space: for each pixel point in the image, a Hessian matrix is constructed to detect candidate interest points, the Hessian matrix is used to describe the second derivative information of the local image, the matrix elements are calculated by convolving the Gaussian second derivative with the image, the Hessian matrix is calculated at different scales, and the scale space is generated to realize the scale invariance, and the expression of the Hessian matrix H is:

[0060]

[0061] Wherein, L xx (x,y,δ), L xy (x,y,δ), L yy (x,y,δ) and respectively are the values of the Gaussian second derivative at the point (x,y) scale δ approximated by the box filter;

[0062] S202(2): Interest point detection: in the scale space, non-maximum suppression is performed on each pixel point in its 8-neighborhood and 26-neighborhood of adjacent scales, and local maximum points of the Hessian matrix determinant value are found, which are candidate interest points;

[0063] S202(3): Interest point positioning and screening: the candidate interest points are accurately positioned at sub-pixel level, the accurate position of the interest point is determined by fitting a three-dimensional quadratic function, at the same time, the interest points with low contrast and located on the edge are removed, the stability and robustness of the feature points are improved, and it is assumed that the value of the Hessian matrix determinant at the point X=(x,y,δ) is D(X), and the quadratic Taylor expansion of D(X) is:

[0064]

[0065] Wherein, D(X0) is the gradient of the function at X0, H is the Hessian matrix, and by solving A more accurate key point position can be obtained;

[0066] S202(4): Determining the main direction of the interest point: taking the interest point as the center, the Haar wavelet response in a certain size neighborhood is calculated. The response value is calculated in the horizontal and vertical directions, and the main direction is determined by counting the total response sum in the sector region, so that the feature description has rotation invariance.

[0067] For a pixel point (X0), the Haar wavelet response d x and the Haar wavelet response d y in the vertical direction are respectively:

[0068] d x =I(x+s,y)-I(x-s,y)

[0069] d y = I(x, y + s) - I(x, y - s)

[0070] where S is the size of the wavelet template;

[0071] S202(5): Generating feature descriptor: taking the interest point as the center, rotating the coordinate axis to the principal direction, selecting a square region around it, and dividing it into several sub-regions. Calculate the Haar wavelet response in each sub-region to obtain the response values in the horizontal and vertical directions and the sum of their absolute values, and finally combine them into a 64-dimensional SURF feature descriptor;

[0072] S203: Splice the LDC-pyramid adaptive feature fusion module and the features extracted by SURF in the channel dimension, and input the spliced features into the SCIEAM (space-channel interactive enhanced attention module) processing;

[0073] S204, input the output feature map of S203 into the SCIEAM module processing, first calculate the channel feature map and the spatial feature map of the input image in parallel, and then splice and fuse the channel feature map and the spatial feature map in the channel dimension;

[0074] The specific steps of the SCIEAM module processing are:

[0075] S204(1): First, after performing global average pooling and global maximum pooling operations on the input image in the spatial dimension, respectively, input the two vectors obtained by the average pooling and the maximum pooling into a shared multi-layer perceptron (MLP), and finally add the two results processed by the MLP to obtain the channel descriptor;

[0076] S204(2): Calculate the spatial descriptor. First, perform average pooling and maximum pooling operations along the channel dimension on the input image, respectively, and then input the two vectors obtained by the average pooling and the maximum pooling into a shared multi-layer perceptron (MLP), and finally add the two results processed by the MLP to obtain the channel descriptor;

[0077] S204(3): Reshape the spatial descriptor to match the dimensions of the channel descriptor, splice the two, and then input them into a shared multi-layer perceptron (MLP);

[0078] S204(4): Again, use the reshape function to decompose the new channel description and spatial descriptor from the output of S204(3).

[0079] S204(5): After the new channel description and spatial description are activated respectively using the Sigmoid function, the channel description and spatial description are multiplied with the image input into the SCIEAM module initially to obtain a channel feature weighted map and a spatial feature weighted map.

[0080] S204(6): The channel feature weighted map and the spatial feature weighted map obtained in S204(5) are input into a linear layer (Linear layer), and a 1x1 convolution is used to further extract and integrate the output after linear transformation;

[0081] S3: Two channel exchanges are performed through the multi-modal feature interaction and deepening module, and two VSS blocks are arranged between the two channel exchanges to perform feature deepening on the two feature maps F1 and F2 in S2, and deepened features are obtained. The structure diagram of the multi-modal feature interaction and deepening module is shown in Figure 5 , and the specific steps are as follows:

[0082] S301: First, the output of S2 is subjected to a first channel exchange operation, and then input into a VSS block for processing;

[0083] The two feature maps obtained from the previous processing are feature F1 and feature F2, the number of channels is C, the height is H, and the width is W. The specific operation of exchanging channels is as follows:

[0084] F1 and F2 are divided into four equal parts along the channel dimension.

[0085]

[0086] where [:, :, :, :] represents a slicing operation on different channel dimensions of data, representing taking 0 to C-1 channels from the F1 map, representing taking 0 to C-1 channels from the F2 map,

[0087] are sequentially spliced in the channel dimension to generate a new feature map F new1 ; and are sequentially spliced in the channel dimension to generate a new feature map F​​​​​​​new2 ;

[0088] The newly generated feature map is as follows:

[0089]

[0090] wherein, represents concatenation in the channel dimension;

[0091] S302, repeat the operation of S301 again;

[0092] The VSS block structure is as shown in Figure 6 , and the specific processing steps are as follows:

[0093] S301(1): The image after channel exchange is first normalized (LayerNorm).

[0094] S301(2): After being processed by a linear layer, the output of S301(1) is input into a depth separable convolution, and the depth convolution kernel size is 3x3, and the pointwise convolution kernel size is 1x1.

[0095] S301(3): The output of S301(2) is activated and input into a 2D-selective scanning (SS2D) module to perform selective scanning on the feature map. After 2D-selective scanning, the feature map size remains 256x256 pixels, and the channel number is still 320.

[0096] S301(4): The output of S301(1) after being processed by a linear layer and activated is multiplied element by element with the output of S301(3) after being normalized.

[0097] S301(5): The output of S301(4) is processed by a linear layer, and then added element by element with the image input into the VSS block to obtain the final output;

[0098] S4: The deepened features obtained in S3 are input into a multi-modal feature fusion and refinement module, and the SS2D selective scanning mechanism and MLP in the multi-modal feature fusion and refinement module are used to perform feature fusion and optimization to obtain optimized features. The structure diagram is as shown in Figure 7 ;

[0099] S401: Layer normalization operation is performed on the deepened features output in S3;

[0100] S402: The output of S401 is processed using a 3x3 LDC convolution kernel, and the outputs of the two LDC convolutions are concatenated in the channel dimension. Then, the concatenated features are processed using SS2D, and the output of SS2D is processed using a multi-layer perceptron (MLP).

[0101] S403: The result after the layer normalization operation of S401 is fused with the output of S402 through pixel-by-pixel multiplication;

[0102] S404: The processing result after S403 is added to the output of S402 pixel by pixel to obtain an optimized feature;

[0103] S5: The optimized feature obtained in S4 is input into a multi-scale residual feature reconstruction module to generate a final fused image through multi-scale convolution extraction, SS2D enhancement and residual connection;

[0104] S501: The output of S4 is respectively input into three parallel convolution layers: one is a 1x1 convolution with a dilation rate (Rate) of 1, and the other two are 3x3 convolutions with dilation rates of 2 and 4, respectively. In this way, features can be extracted from different scales;

[0105] S502: The outputs of the three parallel convolution layers are spliced in the channel dimension, and then a 1x1 convolution is used to integrate features of different scales together to generate a multi-scale fused feature;

[0106] S503: The multi-scale fused feature output by S502 is input into SS2D for feature enhancement, and the output of SS2D is added to the output of S502 element by element;

[0107] S504: The outputs of S504, S503 and S4 are all upsampled, and then added element by element, and finally a 3x3 convolution is used to reduce the dimension of the added result to 3 to obtain the final output.

[0108] Therefore, the present application adopts the above-mentioned method for multi-modal image fusion based on Mamba and attention mechanism, and the present application realizes efficient fusion of multi-modal images by using various modules, comprehensively and stably extracts multi-scale features through cooperation of the LDC-pyramid adaptive feature fusion module and the SURF algorithm to enhance scene adaptability, the SCIEAM module optimizes features with weighting to highlight key information of pathogens, the VSS block captures local and global features for accurate fusion, the multi-modal feature fusion and refining module optimally optimizes features, and the VSS block improves calculation efficiency. The finally generated fused image can clearly present pathogen information, and effectively improves the level of livestock and poultry disease prevention and control.

[0109] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-modal image fusion method based on Mamba and attention mechanism, characterized in that: The method comprises the following steps: S1: inputting an original image, specifically including a fluorescent image generated by a fluorescent quantitative digital PCR instrument and a microscopic image generated by a serpentine microfluidic synthesis imaging device; S2: outputting enhanced features F1 and F2 through preliminary features of the original image input by a multi-modal image feature preliminary extraction module; The multi-modal image feature preliminary extraction module comprises an LDC-pyramid adaptive feature fusion module, a SURF module and a SCIEAM module; The LDC-pyramid adaptive feature fusion module extracts multi-scale features in parallel through LDC convolution layers with different hole rates and global average pooling; The SURF module detects interest points in a scale space and generates feature descriptors by constructing a Hessian matrix; The SCIEAM module is used for receiving features extracted by the LDC-pyramid adaptive feature fusion module and the SURF module after channel dimension splicing, performing weighted fusion, and outputting enhanced features; S3: performing twice channel exchange through a multi-modal feature interaction and deepening module, and interspersing two VSS blocks between the twice channel exchange to perform feature deepening on the two feature maps of features F1 and F2 in S2, and obtaining deepened features; S4: inputting the deepened features in S3 into a multi-modal feature fusion and refining module, performing feature fusion and optimization through an SS2D selective scanning mechanism and an MLP in the multi-modal feature fusion and refining module, and obtaining optimized features; S5: inputting the optimized features in S4 into a multi-scale residual feature reconstruction module, generating a final fusion image through multi-scale convolution extraction, SS2D enhancement and residual connection.

2. The multi-modal image fusion method based on Mamba and attention mechanism according to claim 1, characterized in that: The LDC-pyramid adaptive feature fusion module is provided with four LDC convolution layers with different hole rates and one global average pooling layer in parallel to process the original image, and after splicing the features output by the four LDC convolution layers with different hole rates and the global average pooling layer in the channel dimension, multi-scale fusion features are generated through 1×1 convolution dimension reduction; The specific processing steps of the SURF module are as follows: detecting candidate interest points through a Hessian matrix, and performing non-maximum suppression in a scale space; determining the main direction of the interest points through a Haar wavelet response, and generating feature descriptors with scale and rotation invariance; The specific steps of the SCIEAM module include: parallelly calculating the channel descriptors and spatial descriptors of the features spliced in the channel dimension and extracted by the LDC-pyramid adaptive feature fusion module and the SURF module, splicing the channel descriptors and spatial descriptors, decomposing the new channel and spatial weights through an MLP after splicing, activating the weights through a Sigmoid function, and respectively performing weighted fusion on the features in the new channel and spatial weights.

3. The multi-modal image fusion method based on Mamba and attention mechanism according to claim 2, characterized in that: The multi-modal feature interaction and deepening module performs the following operations on the enhanced features based on the output of S2: first channel exchange operation is performed on the enhanced features, then the enhanced features are input into a VSS block for processing, second channel exchange operation is performed after the processing is completed, and the processed features are input into another VSS block for processing, and the two VSS blocks have consistent structures.

4. The multi-modal image fusion method based on Mamba and attention mechanism according to claim 1, characterized in that: The multi-modal feature fusion and refining module has the following specific steps: S401: performing layer normalization operation on the deepening features output in S3; S402: using LDC convolution to process the output of S401, splicing the outputs of the two LDC convolutions in the channel dimension, then using SS2D to process the spliced features, and using a multi-layer perceptron to process the output of SS2D; S403: fusing the result of the layer normalization operation on S401 and the output of S402 through pixel-by-pixel multiplication; S404: performing pixel-by-pixel addition on the processing result of S403 and the output of S402 to obtain optimized features.

5. The multi-modal image fusion method based on Mamba and attention mechanism according to claim 1, characterized in that: The multi-modal feature fusion and refining module has the following specific steps: S501: inputting the optimized features obtained in S4 into three parallel convolution layers respectively; S502: integrating the outputs of the three parallel convolution layers after splicing in the channel dimension to generate multi-scale fusion features; S503: inputting the multi-scale fusion features into SS2D for feature enhancement, and performing element-by-element addition on the output of SS2D and the output of S502; S504: upsampling the outputs of S503 and S4, performing element-by-element addition on the two outputs, and outputting the result after dimension reduction.

Images