A multi-modal medical image fusion method and model with variable modal input

Abstract

The application belongs to the technical field of image processing, and discloses a multi-modal medical image fusion method and model with variable modal input, which gradually performs denoising processing on a fusion image through a conditional diffusion fusion model based on Mamba, and corrects output features according to input source images and the number of modalities of the input source images by using an expectation maximization module, so that flexible fusion of medical images with different numbers of modalities is realized, the quality of the fusion image is improved, and the advantages of being able to adapt to the medical image fusion requirements of different numbers of modalities and improving the quality of the fusion image are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal medical image fusion method and model with variable modal input. BACKGROUND

[0002] Medical imaging technology plays a crucial role in modern medical diagnosis. Common medical imaging modalities include CT (Computed Tomography), MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), and SPECT (Single Photon Emission Computed Tomography), etc. These different imaging modalities can provide unique anatomical, functional, or metabolic information. However, due to differences in imaging principles and objectives, single-modality medical images often cannot fully reflect the characteristics of biological tissues. For example, CT excels in bone imaging, MRI has advantages in soft tissue imaging, while PET and SPECT can provide important functional data, but their spatial resolution is relatively low.

[0003] To overcome the limitations of single-modality images, multi-modal medical image fusion (MMIF) technology has emerged. MMIF technology aims to integrate the advantages of different modalities of images to generate a fused image containing more comprehensive information, thereby providing more reliable basis for clinical diagnosis and treatment decision-making. Currently, MMIF technology mainly includes two categories: two-modality image fusion methods and three-modality image fusion methods.

[0004] In two-modality image fusion methods, common techniques include methods based on autoencoders, convolutional neural networks, Transformers, GANs (Generative Adversarial Networks), or Diffusion models. Among them, GAN models have shown excellent performance in image fusion tasks, but the instability problem in their training process may affect the quality of the final fused image. Diffusion models, while solving the instability problem of GAN to some extent, have the characteristics of large computational overhead and long fusion time, making it difficult to meet the real-time requirements of clinical applications.

[0005] Three-modality image fusion methods include traditional sparse representation and multi-scale transformation methods, as well as deep learning-based Diffusion and GAN methods. Traditional methods, although fast in computation, often lack sufficient robustness and are difficult to adapt to complex and variable medical images. While deep learning methods have superior performance, they require high computational resources, which may impose additional hardware investment burdens on medical institutions.

[0006] The existing MMIF methods usually only support image fusion of a fixed number of modalities, which brings a series of problems. First, each acquisition of medical images requires the patient to pay the corresponding cost, although the fusion of more modalities of medical images can obtain more comprehensive information, but at the same time, it will increase the economic burden of the patient. Second, the fusion method of a fixed number of modalities lacks flexibility and cannot be adjusted according to actual clinical needs and available resources. Finally, image fusion of different numbers of modalities often requires the use of different models, which increases the complexity of model development and maintenance, and may also lead to inconsistency between the fusion results of different numbers of modalities.

[0007] In addition, the existing fusion methods still have room for improvement in image quality. For example, how to effectively suppress noise and artifacts while preserving the unique information of each modality image, and improve the contrast and clarity of the fused image, is still a problem to be solved.

[0008] In view of the above problems, the existing technology needs to be improved. SUMMARY

[0009] The purpose of the present application is to provide a multi-modality medical image fusion method and model with variable modality input, which can adapt to the medical image fusion needs of different numbers of modalities and improve the quality of the fused image.

[0010] In a first aspect, the present application provides a multi-modality medical image fusion model with variable modality input, comprising a Mamba-based conditional diffusion fusion model and an expectation maximization module.

[0011] The Mamba-based conditional diffusion fusion model is used to perform denoising processing on the fused image step by step with a preset pure noise image as the initialized fused image.

[0012] The expectation maximization module is used to correct the output feature of each step of denoising processing of the Mamba-based conditional diffusion fusion model according to the input source image and its number of modalities by adjusting the likelihood, to obtain a corrected output feature; before the Mamba-based conditional diffusion fusion model completes the denoising processing of a preset number of steps, the corrected output feature is taken as the fused image input into the Mamba-based conditional diffusion fusion model for the next step of denoising processing; when the Mamba-based conditional diffusion fusion model completes the denoising processing of a preset number of steps, the corrected output feature is taken as the final fused image for output; in this way, the final fused image is matched with the input source image and its number of modalities; the number of modalities of the input source image is 2 or 3.

[0013] The model gradually denoises the fused image through the Mamba-based conditional diffusion fusion model, and corrects the output feature according to the input source image and the number of modalities thereof by using an expectation maximization module, so as to realize flexible fusion of medical images with different numbers of modalities, improve the quality of the fused image, and have the advantages of being able to adapt to the fusion requirements of medical images with different numbers of modalities and improving the quality of the fused image.

[0014] Preferably, when the expectation maximization module corrects the output feature after each step of denoising processing of the Mamba-based conditional diffusion fusion model according to the input source image and the number of modalities thereof by adjusting the likelihood, to obtain a corrected output feature, the expectation maximization module performs:

[0015] The corrected first reference feature is iteratively calculated using the coordinate descent method according to the following formula:

[0016] ;

[0017] ;

[0018] wherein k and u are constraint variables, is the corrected first reference feature obtained at the current iteration step, is the corrected first reference feature obtained at the previous iteration step, the initial value of is the first reference feature x corresponding to , is the output feature of the Mamba-based conditional diffusion fusion model after denoising processing at the current step, and y is a second reference feature, is an inverse fast Fourier transform function, is an inverse fast Fourier transform function, is a gradient operator, is the gradient of k, denotes the conjugate complex of denotes element-wise multiplication, and are constant parameters, and denote two modalities of source images in the input source image, M and N are latent variable matrices, and satisfy:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] in, is the element in row i and column j in M, is the element in row i and column j in N, To correspond to The intermediate amount, To correspond to The intermediate amount, is the element in row i and column j in m, is the element in row i and column j in n, m and n are the latent variable matrices of the conditional expectation of the log-likelihood function of the image fusion probability, is the element in row i and column j of y, is the element in row i and column j in x, is the source image of the third modality in the input source image. If the input source image only contains source images of two modalities, then is a zero matrix, express The conditional expectation of express The conditional expectation of express expectations, express expectations, for The expected mean of for The expected mean value of , h is the height pixel size of the source image, w is the width pixel size of the source image;

[0027] The corrected output feature is calculated using the following formula based on the corrected first reference feature:

[0028] ;

[0029] in, Features are output for the correction.

[0030] The advantage of this correction method is its adaptability to input source images with varying numbers of modalities, enhancing the model's flexibility and versatility. By performing correction after each denoising step, the fusion process is continuously optimized, improving the quality of the final fused image. Compared to existing techniques, this method offers greater adaptability and higher fusion quality when fusion multimodal medical images.

[0031] Preferably, the Mamba-based conditional diffusion fusion model comprises a first input layer, a patching block, an encoding block, a plurality of DFM blocks connected in series, a first regularization layer, a linear layer and a first output layer; the patching block is connected between the first input layer and the first DFM block, the encoding block is used to encode the time steps and input each DFM block, the last DFM block, the first regularization layer and the linear layer are connected in series, and the output of the first input layer and the output of the linear layer are subtracted and input into the first output layer.

[0032] The patching block is used to patch the input image to generate a plurality of label sequences.

[0033] The DFM block is used to encode the input label sequence.

[0034] The linear layer is used to decrypt the label sequence to obtain and output the predicted noise.

[0035] This structure design enables the model to effectively process and fuse multi-modal medical images, and solves the structure design problem of the Mamba-based conditional diffusion fusion model. By reasonably arranging various functional modules, the model can gradually denoise and extract features from the input image, and finally obtain a high-quality fused image.

[0036] Preferably, the DFM block comprises a second input layer, a second regularization layer, a scaling and translation layer, a DF-Mamba block, a scaling layer, a first full connection layer and a second output layer; the second input layer, the second regularization layer, the scaling and translation layer, the DF-Mamba block and the scaling layer are connected in series; the first full connection layer is used to dimensionally align and transform the features of the encoded time steps output by the encoding block and input the scaling layer; the output of the scaling layer and the output of the second input layer are added and input into the second output layer.

[0037] Preferably, the DF-Mamba block comprises a third input layer, a flattening and linear projection block, a diffusion fusion Mamba encoder, a second full connection layer and a third output layer connected in series.

[0038] Preferably, the diffusion fusion Mamba encoder comprises a fourth input layer, a third regularization layer, a forward propagation conv1d block, a forward propagation SSM block, an activation layer and a fourth output layer; an input end of the third regularization layer is connected with an output end of the fourth input layer, an output end of the third regularization layer is connected with an input end of the forward propagation conv1d block and an input end of the activation layer respectively, an input end of the forward propagation SSM block is connected with an output end of the forward propagation conv1d block, and an output of the forward propagation SSM block and an output of the activation layer are multiplied and then added with an output of the fourth input layer, and a result of the addition is input into the fourth output layer.

[0039] In a second aspect, the present application provides a multi-modal medical image fusion method with variable modal input, comprising the steps of:

[0040] A1. constructing an initial multi-modal medical image fusion model with variable modal input; the initial multi-modal medical image fusion model with variable modal input is the multi-modal medical image fusion model with variable modal input described above;

[0041] A2. training the Mamba-based conditional diffusion fusion model in the initial multi-modal medical image fusion model with variable modal input to obtain a trained multi-modal medical image fusion model with variable modal input;

[0042] A3. using the trained multi-modal medical image fusion model with variable modal input to perform image fusion on a to-be-fused source image; the to-be-fused source image comprises a source image of two modalities or a source image of three modalities.

[0043] Preferably, step A2 comprises:

[0044] A201. obtaining a medical image set, the medical image set comprising medical images of multiple different modalities;

[0045] A202. dividing the medical image set into a training set and a validation set;

[0046] A203. training the Mamba-based conditional diffusion fusion model using the training set;

[0047] A204. verifying the trained Mamba-based conditional diffusion fusion model using the validation set.

[0048] Preferably, step A203 comprises:

[0049] B1. randomly selecting a medical image from the training set as a training image;

[0050] B2. adding random noise in steps in the training image according to a preset step number;

[0051] B3. In each time step, the following steps are performed:

[0052] inputting the training image with random noise added in the current time step into the Mamba-based conditional diffusion fusion model to obtain a predicted image of the training image without random noise added in the current time step output by the Mamba-based conditional diffusion fusion model;

[0053] calculating a loss function according to the predicted image and the training image without random noise added in the current time step;

[0054] updating the model parameters of the Mamba-based conditional diffusion fusion model by back propagation according to the loss function;

[0055] B4. Steps B1-B3 are repeatedly performed until a preset ending condition is met, and then the training is stopped.

[0056] Preferably, step A3 comprises:

[0057] inputting the expected maximum module in the trained modality-variable multi-modal medical image fusion model with the to-be-fused source image and inputting the Mamba-based conditional diffusion fusion model in the trained modality-variable multi-modal medical image fusion model with a preset pure noise image to obtain a fusion image output by the trained modality-variable multi-modal medical image fusion model.

[0058] Beneficial effects: The modality-variable multi-modal medical image fusion method and model provided by the present application gradually denoises the fusion image through the Mamba-based conditional diffusion fusion model, and corrects the output features according to the input source image and the number of modalities thereof by using the expected maximum module, thereby realizing flexible fusion of medical images with different numbers of modalities, improving the quality of the fusion image, and having the advantages of being able to adapt to the medical image fusion requirements of different numbers of modalities and improving the quality of the fusion image. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The principle diagram of the modality-variable multi-modal medical image fusion model provided by the embodiments of the present application.

[0060] Figure 2 The structural schematic diagram of the Mamba-based conditional diffusion fusion model.

[0061] Figure 3 The structural schematic diagram of the MFD block.

[0062] Figure 4 The structural schematic diagram of the DF-Mamba block.

[0063] Figure 5 Structural schematic diagram of diffusion fusion Mamba encoder.

[0064] Figure 6 Flowchart of the multi-modal medical image fusion method with variable modal input provided by the embodiments of the present application.

[0065] Label explanation: 1, first input layer; 2, patching block; 3, encoding block; 4, DFM block; 5, first regularization layer; 6, linear layer; 7, first output layer; 8, second input layer; 9, second regularization layer; 10, scaling and translation layer; 11, DF-Mamba block; 12, scaling layer; 13, first full connection layer; 14, second output layer; 15, third input layer; 16, flattening and linear projection block; 17, diffusion fusion Mamba encoder; 18, second full connection layer; 19, third output layer; 20, fourth input layer; 21, third regularization layer; 22, forward propagation conv1d block; 23, forward propagation SSM block; 24, activation layer; 25, fourth output layer; 100, Mamba-based conditional diffusion fusion model; 200, expectation maximization module. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0067] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms “first”, “second” and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0068] Multi-modal medical image fusion technology plays an important role in clinical diagnosis and treatment. By integrating the advantages of different imaging modalities such as CT, MRI, PET, and SPECT, a fusion image containing more comprehensive information can be generated. However, existing fusion methods usually only support image fusion with a fixed number of modalities, which limits their flexibility in practical applications. When faced with input images of different modalities, these methods are difficult to adapt, which may result in poor fusion results or inability to perform fusion. In addition, existing methods often have low computational efficiency and unstable fusion quality when dealing with high-dimensional and complex medical images. These technical limitations seriously affect the widespread application and effectiveness of multi-modal medical image fusion technology in clinical practice.

[0069] If the problem of multi-modal medical image fusion with variable modal input cannot be effectively solved, it will have a negative impact on the medical diagnosis and treatment process. First, medical institutions may need to maintain multiple fusion systems for different modality combinations, which not only increases hardware and software costs but also increases system maintenance complexity. Second, doctors may need to compare and weigh different fusion results when making a diagnosis, which increases the complexity of diagnosis and the risk of misdiagnosis. From a technical point of view, fusion methods that cannot adapt to different numbers of modalities may result in insufficient information utilization in some cases, while introducing unnecessary noise or artifacts in other cases. This inconsistency may affect the accuracy and reliability of diagnosis. In addition, in the field of clinical research and medical image analysis, the lack of flexible fusion methods may limit the exploration and research of different modality combinations, hindering the further development of medical imaging technology. Therefore, it is particularly important to develop a multi-modal medical image fusion method that can adapt to different numbers of modalities, maintain high-quality fusion results, and have good computational efficiency.

[0070] To this end, please refer to Figures 1-5 The application provides a multi-modal medical image fusion model with variable modal input, comprising a Mamba-based conditional diffusion fusion model 100 and an expectation maximization module 200.

[0071] The Mamba-based conditional diffusion fusion model 100 is used to denoise the fusion image step by step with a preset pure noise image as the initialized fusion image.

[0072] The expectation maximization module 200 is used to correct the output feature of each step of the Mamba-based conditional diffusion fusion model 100 after denoising processing by adjusting the likelihood according to the input source image and the number of its modalities, to obtain a corrected output feature; before the Mamba-based conditional diffusion fusion model 100 completes the preset number of denoising processes, the corrected output feature is taken as the input of the fusion image of the Mamba-based conditional diffusion fusion model 100 for the next step of denoising processing; when the Mamba-based conditional diffusion fusion model 100 completes the preset number of denoising processes, the corrected output feature is taken as the final fusion image for output (for example Figure 1 In the middle, is the output feature of each step of the Mamba-based conditional diffusion fusion model 100 after denoising processing, is the corrected output feature of each time step, is the final fusion image, ); in this way, the final fusion image matches the input source image and the number of its modalities; the number of modalities of the input source image is 2 or 3.

[0073] The model gradually denoises the fusion image through the Mamba-based conditional diffusion fusion model 100, and corrects the output feature according to the input source image and the number of its modalities through the expectation maximization module 200, thereby realizing flexible fusion of medical images of different modalities, improving the quality of the fusion image, and having the advantages of being able to adapt to the fusion needs of medical images of different modalities and improving the quality of the fusion image.

[0074] wherein the preset pure noise image is a random noise image used as an initialization fusion image, which can be generated by using Gaussian white noise or other types of random noise.

[0075] wherein the preset number of steps can be adjusted according to actual needs.

[0076] wherein the input source image can include two or three of CT images, MR-T1 images, MR-T2 images, MR-Gad images, PET images and SPECT images.

[0077] The principle of the model is as follows:

[0078] In each step of denoising processing, the Mamba-based conditional diffusion fusion model 100 uses the sequence modeling capability of Mamba to process the input image feature. The core of the Mamba model is its efficient state space model structure, which can capture long-distance dependencies, which is particularly important for complex structures and detail preservation in medical images.

[0079] The expectation maximization module 200 receives two inputs: the output features of the Mamba-based conditional diffusion fusion model 100, and the original input source images and their modality number information. The expectation maximization module 200 corrects the output features by adjusting the likelihood. This process takes into account the input source images and their modality numbers, ensuring that the corrected features adequately reflect the information of all input modalities.

[0080] The corrected output features are then sent back to the Mamba-based conditional diffusion fusion model 100 as input for the next step of denoising processing. This forms an iterative process, with each step optimizing the quality of the fused image.

[0081] When the preset number of denoising processes is completed, the last corrected output features are considered as the final fused image. This fused image contains key information of all input modalities, and its features match the modality numbers of the input source images.

[0082] The reason for choosing to combine the Mamba model and the conditional diffusion process is that the Mamba model exhibits better performance than the Transformer in sequence modeling tasks, while also having higher computational efficiency. This allows the model to better handle complex structures and long-range dependencies in medical images. The conditional diffusion process provides a gradual image generation method, which helps to generate high-quality fused images. The introduction of the expectation maximization module 200 is to solve the problem of variable modality numbers. By dynamically adjusting the likelihood, this module can optimize the fusion process according to the actual input modality numbers, ensuring that both two-modality and three-modality inputs are properly processed and fused.

[0083] Preferably, when the expectation maximization module 200 corrects the output features of the Mamba-based conditional diffusion fusion model 100 after each step of denoising processing according to the input source images and their modality numbers by adjusting the likelihood, the corrected output features are obtained, the expectation maximization module 200 performs:

[0084] The first reference feature of the correction is calculated according to the following formula:

[0085] (1);

[0086] (2);

[0087] where k and u are constraint variables, is the corrected first reference feature obtained in the current iteration step, is the corrected first reference feature obtained in the previous iteration step, the initial value of is the first reference feature x corresponding to the first reference feature x of the correction, is the output feature of the Mamba-based conditional diffusion fusion model after denoising at the current step, y is the second reference feature, is the inverse fast Fourier transform function, is the inverse fast Fourier transform function, is the gradient operator, is the gradient of k, express The complex conjugate of represents element-wise multiplication, and is a constant parameter, and Represent the source images of the two modalities in the input source image, M and N are latent variable matrices, and satisfy:

[0088] (3);

[0089] (4);

[0090] (5);

[0091] (6);

[0092] (7);

[0093] (8);

[0094] (9);

[0095] in, is the element in row i and column j in M, is the element in row i and column j in N, To correspond to The intermediate amount, To correspond to The intermediate amount, is the element in row i and column j in m, is the element in row i and column j in n, m and n are the latent variable matrices of the conditional expectation of the log-likelihood function of the image fusion probability, is the element in row i and column j of y, is the element in row i and column j in x, is the source image of the third modality in the input source image. If the input source image only contains source images of two modalities, then is a zero matrix, express The conditional expectation of express the conditional expectation of denotes the expectation of denotes the expectation of is the expected mean of is the expected mean of h is the height pixel size of the source image, and w is the width pixel size of the source image;

[0096] According to the corrected first reference feature, the corrected output feature is calculated using the following formula:

[0097] (10);

[0098] wherein, is the corrected output feature.

[0099] The above correction method has the advantages of being able to adapt to different modal numbers of input source images, improving the flexibility and universality of the model. By performing correction after each denoising process, the fusion process can be continuously optimized, and the quality of the final fused image can be improved. Compared with the prior art, the method has stronger adaptability and higher fusion quality when processing multi-modal medical image fusion.

[0100] The formula derivation process of the above correction method is as follows:

[0101] First, the loss function of the free switching fusion task can be taken as:

[0102] (11);

[0103] wherein, is the weight (if the input source image only contains two modal source images, then is a zero matrix), denotes the 1-norm.

[0104] Let the first reference feature be , and the second reference feature be (thereby according to the definition, formulas (2), (9) and (10) can be obtained), and the variable substitution is performed on formula (11), and the loss function of the free switching fusion task is converted to:

[0105] (12);

[0106] Obviously, y is a known quantity, and x is an unknown quantity. The optimization problem of formula (12) can be converted into a maximum likelihood inference problem through probability analysis. The log-likelihood function of the probability inference problem of image fusion is:

[0107] (13);

[0108] where, is the log-likelihood function, is the gradient of x, is the total variation penalty term, denotes the 2-norm.

[0109] To solve this maximum likelihood inference problem, it can be treated as a latent variable optimization problem, and the expectation-maximization algorithm is used to obtain the optimal x.

[0110] The expectation value of the log-likelihood function with respect to the probability distribution of the source image can be expressed as:

[0111] (14);

[0112] where, is the expectation value of the log-likelihood function with respect to the probability distribution of the source image, a and b are algebraic, is the log-likelihood function corresponding to .

[0113] The conditional expectation of the latent variable and in formula (14) is:

[0114] (15);

[0115] (16);

[0116] (17);

[0117] (18);

[0118] Let two intermediate matrices and , where, the element in the i-th row and j-th column of is the intermediate variable , and , the element in the i-th row and j-th column of is the intermediate variable , and , and let two latent variable matrices M and N, where the element in the i-th row and j-th column of M is , and , and the element in the i-th row and j-th column of N is , and ; thus, formula (3) - formula (8) can be obtained by combining formula (15) - formula (18). Combined with formula (13), formula (14) can be converted to:

[0119] (19);

[0120] wherein, is the gradient of is the element in the i-th row and j-th column of

[0121] According to formula (19), the semi-quadratic splitting algorithm is used to minimize the negative function of :

[0122] (20);

[0123] Then the solution of formula (20) is converted into an unconstrained optimization problem:

[0124] (21);

[0125] The solving process of formula (21) can be converted into: the following formula is solved by iteration through the coordinate descent method:

[0126] ;

[0127] That is, formula (1) is obtained, and it needs to be explained that in the first step of iteration, x is calculated by formula (9), and the x is taken as the initial value of is substituted into the above formula to start the coordinate descent iteration process.

[0128] Specifically, see Figure 2 , the conditional diffusion fusion model 100 based on Mamba includes a first input layer 1, a patching block 2, an encoding block 3, a plurality of DFM blocks 4 connected in series, a first regularization layer 5, a linear layer 6 and a first output layer 7; the patching block 2 is connected between the first input layer 1 and the first DFM block 4, and the encoding block 3 is used to encode the time step and input each DFM block 4 (t in the figure represents the time step information input into the encoding block 3, and t1 represents the encoded time step information); the last DFM block 4, the first regularization layer 5 and the linear layer 6 are connected in series; the output of the first input layer 1 (i.e. the image input into the conditional diffusion fusion model 100 based on Mamba) and the output of the linear layer 6 are subtracted and input into the first output layer 7;

[0129] The patching block 2 is used to patch the input image to generate a plurality of marker sequences;

[0130] The DFM block 4 is used to encode the input marker sequence;

[0131] The linear layer 6 is used to decrypt the mark sequence to obtain and output the predicted noise (thus, the output image of the first output layer 7 is the image obtained by subtracting the predicted noise from the image input into the Mamba-based conditional diffusion fusion model 100).

[0132] This structural design enables the model to effectively process and fuse multi-modal medical images, solving the structural design problem of the Mamba-based conditional diffusion fusion model 100. By reasonably arranging various functional modules, the model can gradually denoise and extract features from the input image, and finally obtain a high-quality fused image.

[0133] Specifically, the first input layer 1 receives the input image to provide a data source for subsequent processing. The patching block 2 divides the input image into p x p patches (p is a hyperparameter, which can be set according to actual needs), and converts the input image into a plurality of mark sequences with a dimension of d (the specific number is determined by the hyperparameter p, and d is set according to actual needs) by performing linear embedding (such as sine-cosine position embedding) on each patch, which helps the model better capture local features. The encoding block 3 encodes the time step to provide time sequence information for the DFM block 4, which helps the model understand the process of image changes. The DFM block 4 is the core processing unit of the model, and through a plurality of serially connected DFM blocks 4, the model can gradually extract and fuse image features. The first regularization layer 5 and the linear layer 6 further process the output of the DFM block 4, which helps to improve the generalization ability and prediction accuracy of the model. The first output layer 7 obtains the final prediction result (i.e., the single-step denoising result) by subtracting the predicted noise output by the linear layer 6 from the output of the first input layer 1 (i.e., the input image received by the first input layer 1).

[0134] Among them, the linear layer 6 can adopt a standard linear decoder. Specifically, the linear layer 6 applies adaLN normalization (i.e., adaptive layer normalization) to decode each mark sequence into a p x p x 2C tensor, where C represents the number of channels of the spatial input to the DFM block 4, and then rearranges the decoded mark sequence back to its original spatial layout, thereby obtaining the predicted noise, wherein the predicted noise output is used to be subtracted from the output of the first input layer 1.

[0135] Further, see Figure 3, the DFM block 4 includes a second input layer 8, a second regularization layer 9, a scaling and shifting layer 10, a DF-Mamba block 11, a scaling layer 12, a first fully connected layer 13, and a second output layer 14; the second input layer 8, the second regularization layer 9, the scaling and shifting layer 10, the DF-Mamba block 11, and the scaling layer 12 are connected in sequence; the first fully connected layer 13 is used to input the scaling layer 12 after dimension alignment and feature transformation of the time step (i.e., t1 in the figure) output by the encoding block 3; the output of the scaling layer 12 and the output of the second input layer 8 (i.e., the label sequence input into the DFM block 4) are added and input into the second output layer 14.

[0136] Specifically, the second input layer 8, the second regularization layer 9, the scaling and shifting layer 10, the DF-Mamba block 11, and the scaling layer 12 are connected in sequence, constituting the main processing path of the DFM block 4. The first fully connected layer 13 is used to perform dimension alignment and feature transformation on the encoded time step output by the encoding block 3, and then input the processed result into the scaling layer 12. Finally, the output of the scaling layer 12 and the output of the second input layer 8 are added to form a residual connection structure, and the result is input into the second output layer 14.

[0137] Each component of this structure design has its specific function and role. The second input layer 8 receives the input label sequence, providing raw data for subsequent processing. The second regularization layer 9 performs standardization processing on the input, which helps to improve the stability and convergence speed of the model. The scaling and shifting layer 10 performs linear transformation on the features, enhancing the expression ability of the model. The DF-Mamba block 11, as the core processing unit, is responsible for deep feature extraction and encoding of the label sequence. The scaling layer 12 adjusts the output of the DF-Mamba block 11 to adapt to the subsequent processing requirements.

[0138] The introduction of the first fully connected layer 13 is to realize the effective fusion of time information and spatial features. It processes the time step output by the encoding block 3, realizes dimension alignment and feature transformation, and matches the feature dimension of the backbone network. This design allows effective fusion of time information and spatial features, improving the model's processing capability for time series data. Among them, the first fully connected layer 13 can be implemented by two layers of MLP (Multi-Layer Perceptron).

[0139] The output of the scaling layer 12 and the output of the second input layer 8 are added to form a residual connection. This structure can alleviate the gradient vanishing problem in deep networks, help information transmission without loss, and thus improve the training efficiency and performance of the model.

[0140] Further, see Figure 4 , the DF-Mamba block 11 includes a third input layer 15, a flattening and linear projection block 16, a diffusion fusion Mamba encoder 17, a second fully connected layer 18, and a third output layer 19 connected in sequence.

[0141] Specifically, the third input layer 15 receives input data from the previous layer. This layer serves as the entrance of the DF-Mamba block 11, responsible for receiving and preliminary processing the inputted token sequence. The flatten and linear projection block 16 performs a flatten operation on the input data and adjusts the data dimension through linear projection, preparing for subsequent processing. The flatten operation can convert multi-dimensional data into a one-dimensional vector, facilitating subsequent processing. Linear projection can adjust the dimension of the data to adapt to the input requirements of the diffusion fusion Mamba encoder 17. The diffusion fusion Mamba encoder 17 serves as the core processing unit, using the Mamba structure for feature extraction and fusion. The Mamba structure is a new type of sequence modeling architecture, which has higher computational efficiency and better long sequence modeling ability compared to traditional Transformer structure. In medical image processing, this feature is particularly important because medical images usually have high resolution and complex spatial structure. The second fully connected layer 18 further transforms and adjusts the dimension of the encoded features. The second fully connected layer 18 can capture the complex relationships between features and enhance the model's expression ability. The third output layer 19 outputs the processed results as the final output of the DF-Mamba block 11.

[0142] This design achieves effective processing of input data by reasonably arranging the order and function of each component. The flatten and linear projection block 16 first adjusts the data format to prepare for the processing of the diffusion fusion Mamba encoder 17. The diffusion fusion Mamba encoder 17, as the core, uses its powerful sequence modeling ability to extract and fuse features. Finally, the second fully connected layer 18 and the third output layer 19 complete the final transformation and output of the features. This structural design helps to improve the model's ability to extract and fuse features of multi-modal medical images, thereby improving the quality of the final fused image. Due to the use of Mamba structure, this design may have higher computational efficiency and better long sequence modeling ability, which is particularly advantageous for processing high-resolution medical images.

[0143] Further, see Figure 5, the diffusion fusion Mamba encoder 17 comprises a fourth input layer 20, a third regularization layer 21, a forward propagation conv1d block 22, a forward propagation SSM block 23, an activation layer 24 and a fourth output layer 25; the input end of the third regularization layer 21 is connected with the output end of the fourth input layer 20, the output end of the third regularization layer 21 is connected with the input end of the forward propagation conv1d block 22 and the input end of the activation layer 24 respectively, the input end of the forward propagation SSM block 23 is connected with the output end of the forward propagation conv1d block 22, the output of the forward propagation SSM block 23 is multiplied with the output of the activation layer 24, and then added with the output (i.e. the input feature of the diffusion fusion Mamba encoder 17) of the fourth input layer 20, and the addition result is input into the fourth output layer 25.

[0144] Specifically, the third regularization layer 21 performs standardization processing on the input and divides it into two outputs, one output is sequentially subjected to feature extraction processing of the forward propagation conv1d block 22 and long sequence modeling processing of the forward propagation SSM block 23, and the other output is subjected to processing of the activation layer 24 and then used for gating the output of the forward propagation SSM block 23, and then the gating result and the input feature of the diffusion fusion Mamba encoder 17 are added and output.

[0145] The diffusion fusion Mamba encoder 17 design effectively solves the information transmission and interaction problem in multi-modal medical image fusion, and provides a new solution for improving the fusion image quality and model efficiency.

[0146] Reference Figure 6 The application also provides a multi-modal medical image fusion method with variable modal input, comprising the steps of:

[0147] A1. Constructing an initial multi-modal medical image fusion model with variable modal input; the initial multi-modal medical image fusion model with variable modal input is the multi-modal medical image fusion model with variable modal input described above;

[0148] A2. Training the Mamba-based conditional diffusion fusion model 100 in the initial multi-modal medical image fusion model with variable modal input to obtain a trained multi-modal medical image fusion model with variable modal input;

[0149] A3. Using the trained multi-modal medical image fusion model with variable modal input to perform image fusion on the source images to be fused; the source images to be fused include source images of two modalities or source images of three modalities.

[0150] In step A2, when training the model, the Mamba-based conditional diffusion fusion model 100 in the initial modality-input-variable multi-modal medical image fusion model is trained alone (the expectation maximization module 200 does not participate in the training process), and the trained Mamba-based conditional diffusion fusion model 100 and the expectation maximization module 200 constitute the trained modality-input-variable multi-modal medical image fusion model.

[0151] Specifically, step A2 includes:

[0152] A201. Obtain a medical image set, the medical image set including medical images of multiple different modalities;

[0153] A202. Divide the medical image set into a training set and a validation set;

[0154] A203. Train the Mamba-based conditional diffusion fusion model 100 using the training set;

[0155] A204. Verify the trained Mamba-based conditional diffusion fusion model 100 using the validation set.

[0156] The medical image set can include at least three of CT images, MR-T1 images, MR-T2 images, MR-Gad images, PET images, and SPECT images, each including a plurality of images.

[0157] When dividing the training set and the validation set, an 8:2 or 7:3 ratio can be used for division.

[0158] Step A203 includes:

[0159] B1. Randomly select a medical image from the training set as a training image;

[0160] B2. Add random noise to the training image in steps according to a preset number of steps;

[0161] B3. In each time step, perform:

[0162] Input the training image after adding random noise at the current time step into the Mamba-based conditional diffusion fusion model 100 to obtain a predicted image of the training image before adding random noise at the current time step output by the Mamba-based conditional diffusion fusion model 100 (i.e., an image obtained after single-step denoising of the training image after adding random noise at the current time step);

[0163] Calculate a loss function according to the predicted image and the training image before adding random noise at the current time step;

[0164] According to the loss function, the model parameters of the Mamba-based conditional diffusion fusion model 100 are updated by back propagation;

[0165] B4. Repeat steps B1-B3 until a preset end condition is met, and then stop training.

[0166] Specifically, in step B1, a random number generator can be used to select the image index in the training set, or the entire training set can be randomly sorted using a shuffle algorithm and then selected in turn. This random selection method ensures the diversity of the training data, which helps to improve the generalization ability of the model.

[0167] In step B2, the preset number of steps can be set according to actual needs, and the random noise added step by step can be Gaussian noise, Poisson noise, or salt and pepper noise, etc.

[0168] In step B3, the loss function can use mean square error (MSE), structural similarity index (SSIM), or perceptual loss, etc. For example, using the calculation method of mean square error (MSE) as follows:

[0169] ;

[0170] wherein, is the loss function of the current time step, is the predicted image of the training image before adding random noise at the current time step output by the Mamba-based conditional diffusion fusion model 100, is the training image before adding random noise at the current time step (since random noise is added step by step in the training image, is known).

[0171] The process of updating the model parameters by back propagation can use different optimizers, such as stochastic gradient descent (SGD), Adam, or RMSprop, etc.

[0172] In step B4, the preset end condition can be based on multiple indicators, such as the number of training rounds, the performance indicators on the validation set, or the convergence degree of the loss function, etc. For example, a threshold value can be set, and when the average peak signal-to-noise ratio (PSNR) on the validation set reaches a certain value, the training is stopped.

[0173] Specifically, step A3 includes:

[0174] The expected maximum module 200 in the trained modality input variable multi-modal medical image fusion model is input with the to-be-fused source image, and the Mamba-based conditional diffusion fusion model 100 in the trained modality input variable multi-modal medical image fusion model is input with the preset pure noise image, to obtain a fusion image output by the trained modality input variable multi-modal medical image fusion model.

[0175] Below, specific examples are used to experimentally verify the present invention. Source images are obtained from the whole-brain atlas database of Harvard Medical School and combined. The three-modality fusion combination includes five combinations: MR-T2 image + MR-Gad image + PET image, CT image + MR-T2 image + SPECT image, MR-T1 image + MR-T2 image + PET image, MR-T2 image + MR-Gad image + SPECT image, and MR-T1 image + MR-T2 image + SPECT image. The two-modality fusion combination includes three combinations: MRI image + CT image, MRI image + PET image, and MRI image + SPECT image.

[0176] Comparison models for trimodal fusion include the BitonicX model, the Cognitive Trading System (CTSR) model, and the Graph Convolutional Network (GCN) model. Comparison models for bimodal fusion include the Unified Unsupervised Image Fusion Network (U2fusion), the Correlation-Driven Dual-Branch Feature Decomposition Fusion (CDDFuse), the Denoising Diffusion Model for Multi-Modality Image Fusion (DDFM), the Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (Swinfusion), the Synergistic High-Order Interaction (SHIP), the Attention-based Layered Memory Fusion Network (ALMFnet), the Dual-Discriminator Conditional Generative Adversarial Network (DDcGAN), the End-to-End Multimodal Model for Autonomous Driving (EMMA), and the Multi-modality Image Fusion via Diffusion Model with Fusion Knowledge Prio (Diff-IF). The source images of the obtained three-modal fusion combination are fused using the comparison model of the present application and the three-modal fusion, and the comparison of the evaluation indicators of the fusion results is shown in Table 1; the source images of the obtained two-modal fusion combination are fused using the comparison model of the present application and the two-modal fusion, and the comparison of the evaluation indicators of the fusion results is shown in Table 2.

[0177] Table 1

[0178] Model VIF SSIM PSNR FSIM MI Qncie Qs Qcv BitonicX 9.2453 0.6099 48.228 0.8860 0.7146 0.8104 0.8985 1603.137 CTSR 6.2016 0.6056 48.298 0.8860 0.6437 0.8091 0.8985 1603.006 GCN 4.1390 0.6151 47.505 0.8872 0.6408 0.8087 0.8978 1948.991 The present application 9.4243 0.6474 48.345 0.8909 0.6969 0.8191 0.8986 1596.948

[0179] Table 2

[0180] Model VIF SSIM PSNR FSIM MI Qncie Qs Qcv U2fusion 0.2758 0.2536 16.777 0.760 0.636 0.8053 0.3531 937.048 CDDFuse 0.3697 0.7484 15.366 0.796 0.884 0.8104 0.8990 281.723 DDFM 0.3236 0.7328 16.491 0.787 0.936 0.8097 0.8240 883.323 Swinfusion 0.2287 0.4112 15.340 0.767 0.657 0.8059 0.5601 462.521 SHIP 0.2557 0.2837 14.852 0.766 0.623 0.8067 0.4550 260.863 ALMFnet 0.2947 0.7461 15.483 0.795 0.740 0.8069 0.8762 262.876 DDcGAN 0.2196 0.7166 14.871 0.768 0.645 0.8061 0.8351 404.063 EMMA 0.3277 0.7419 15.390 0.799 0.894 0.8098 0.8911 270.354 Diff-IF 0.0414 0.1487 11.261 0.646 0.468 0.8043 0.1972 4344.589 The present application 0.3914 0.7493 16.839 0.795 0.921 0.8187 0.9087 261.561

[0181] In the table, VIF is visual information fidelity, SSIM is structural similarity, PSNR is peak signal-to-noise ratio, FSIM is feature similarity, MI is mutual information, Qncie (Quality Indicators of NVSS Checked by Image Experts) is an index for evaluating the quality of image fusion, Qs is an objective non-reference quality evaluation index of image fusion, and Qcv (Chen-Varshney metric) is an image fusion quality evaluation index based on human perception.

[0182] As can be seen from the table, in the three-modal fusion experiment, the objective evaluation indexes of the present application and the existing models are compared, and the results show that the present application performs excellently in multiple key indexes, fully demonstrating its advantages in the three-modal medical image fusion task. Overall, whether compared with a special three-modal fusion model or a two-modal fusion model, the present application performs outstandingly, and the advantages are particularly significant in detail preservation, noise suppression, information fusion, and subjective visual quality.

[0183] In the embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, and electrical, mechanical or other forms.

[0184] In addition, the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, and can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiments.

[0185] Furthermore, the function modules in each of the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0186] In this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations.

[0187] The above merely provides an example of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multimodal medical image fusion model with variable modality input, characterized in that: Including Mamba-based conditional diffusion fusion model and expectation maximization module; The Mamba-based conditional diffusion fusion model is used to use a preset pure noise image as an initialized fusion image and gradually perform denoising on the fusion image; The expectation maximization module is used to correct the output features of the Mamba-based conditional diffusion fusion model after each denoising process by adjusting the likelihood according to the input source image and the number of its modalities, to obtain corrected output features; before the Mamba-based conditional diffusion fusion model completes the denoising process for a preset number of steps, the corrected output features are used as input to the fused image of the Mamba-based conditional diffusion fusion model for the next denoising process; when the Mamba-based conditional diffusion fusion model completes the denoising process for the preset number of steps, the corrected output features are output as the final fused image; thereby, the final fused image matches the input source image and the number of its modalities; the number of modalities of the input source image is 2 or 3; The expectation maximization module corrects the output features of each step of the denoising process of the Mamba-based conditional diffusion fusion model by adjusting the likelihood according to the input source image and the number of its modalities, and when obtaining the corrected output features, executes: The loss function of the free switching fusion task can be taken as: ; in, is the weight, is the output feature of the Mamba-based conditional diffusion fusion model after denoising at the current step, represents the source image of the first modality in the input source image, represents the source image of the second modality in the input source image, is the source image of the third modality in the input source image. If the input source image only contains source images of two modalities, then is a zero matrix; Let the first reference feature be , the second reference feature is ; The first reference feature of the correction is iteratively calculated using the coordinate descent method according to the following formula: ； Among them, k and u are constraint variables, is the first reference feature of the correction obtained in the current iteration step, is the first reference feature of the correction obtained in the previous iteration step, The initial value corresponds to The first reference feature x, is the inverse fast Fourier transform function, is the fast Fourier transform function, is the gradient operator, is the gradient of k, express The complex conjugate of represents element-wise multiplication, and is a constant parameter, M and N are latent variable matrices, The corrected output feature is calculated using the following formula based on the corrected first reference feature: ； in, Features are output for the correction.

2. The multimodal medical image fusion model with variable modality input according to claim 1, characterized in that: The Mamba-based conditional diffusion fusion model includes a first input layer, a patching block, an encoding block, a plurality of sequentially connected DFM blocks, a first regularization layer, a linear layer, and a first output layer; the patching block is connected between the first input layer and the first DFM block, the encoding block is used to encode the time step and then input it into each DFM block; the last DFM block, the first regularization layer, and the linear layer are connected in sequence; the output of the first input layer and the output of the linear layer are subtracted and then input into the first output layer; The patching block is used to patch the input image to generate multiple tag sequences; The DFM block is used to encode the input tag sequence; The linear layer is used to perform decryption processing on the tag sequence to obtain and output prediction noise.

3. The multimodal medical image fusion model with variable modality input according to claim 2, characterized in that: The DFM block includes a second input layer, a second regularization layer, a scaling and translation layer, a DF-Mamba block, a scaling layer, a first fully connected layer, and a second output layer; the second input layer, the second regularization layer, the scaling and translation layer, the DF-Mamba block, and the scaling layer are connected in sequence; the first fully connected layer is used to perform dimension alignment and feature transformation on the encoded time step output by the encoding block and then input it into the scaling layer; The output of the scaling layer and the output of the second input layer are added together and then input into the second output layer.

4. The multimodal medical image fusion model with variable modality input according to claim 3, characterized in that: The DF-Mamba block includes a third input layer, a flattening and linear projection block, a diffusion fusion Mamba encoder, a second fully connected layer, and a third output layer connected in sequence.

5. The multimodal medical image fusion model with variable modality input according to claim 4, characterized in that: The diffusion fusion Mamba encoder includes a fourth input layer, a third regularization layer, a forward propagation conv1d block, a forward propagation SSM block, an activation layer and a fourth output layer; the input end of the third regularization layer is connected to the output end of the fourth input layer, the output end of the third regularization layer is respectively connected to the input end of the forward propagation conv1d block and the input end of the activation layer, the input end of the forward propagation SSM block is connected to the output end of the forward propagation conv1d block, the output of the forward propagation SSM block and the output of the activation layer are multiplied and added to the output of the fourth input layer, and the addition result is input into the fourth output layer.

6. A multimodal medical image fusion method with variable modality input, characterized in that: Including steps: A1. Constructing an initial multimodal medical image fusion model with variable modal input; the initial multimodal medical image fusion model with variable modal input is the multimodal medical image fusion model with variable modal input according to any one of claims 1-5; A2. Training the Mamba-based conditional diffusion fusion model in the initial multimodal medical image fusion model with variable modal input to obtain a trained multimodal medical image fusion model with variable modal input; A3. Perform image fusion on source images to be fused using the trained multimodal medical image fusion model with variable modality input; the source images to be fused include source images of two modalities or source images of three modalities.

7. The multimodal medical image fusion method with variable modality input according to claim 6, characterized in that: Step A2 includes: A201. Acquire a medical image set, wherein the medical image set includes medical images of multiple different modalities; A202. Dividing the medical image set into a training set and a validation set; A203. Using the training set, the Mamba-based conditional diffusion fusion model is trained; A204. Use the validation set to validate the trained Mamba-based conditional diffusion fusion model.

8. The multimodal medical image fusion method with variable modality input according to claim 7, characterized in that: Step A203 includes: B1. Randomly select a medical image from the training set as a training image; B2. Add random noise in the training image step by step according to a preset number of steps; B3. Execute at each time step: Inputting the training image after adding random noise at the current time step into the Mamba-based conditional diffusion fusion model to obtain a predicted image of the training image before adding random noise at the current time step output by the Mamba-based conditional diffusion fusion model; Calculating a loss function based on the predicted image and the training image before adding random noise at the current time step; According to the loss function, updating the model parameters of the Mamba-based conditional diffusion fusion model through back propagation; B4. Repeat steps B1 to B3 until the preset end condition is met, then stop training.

9. The multimodal medical image fusion method with variable modality input according to claim 6, characterized in that: Step A3 includes: The source image to be fused is input into the expectation maximization module of the trained multimodal medical image fusion model with variable modal input, and the preset pure noise image is input into the Mamba-based conditional diffusion fusion model of the trained multimodal medical image fusion model with variable modal input to obtain a fused image output by the trained multimodal medical image fusion model with variable modal input.

Images