Multi-modal medical image fusion method and model with variable modal input

Abstract

The invention belongs to the technical field of image processing, and discloses a modal-input-variable multi-modal medical image fusion method and model, denoising processing is performed on a fused image step by step through a Mama-based conditional diffusion fusion model, and an expectation maximization module is utilized to correct output features according to an input source image and the modal number thereof, so that the modal-input-variable multi-modal medical image fusion method and model are obtained. Therefore, flexible fusion of the medical images with different modal numbers is realized, the quality of the fused image is improved, and the method has the advantages of being capable of adapting to the fusion requirements of the medical images with different modal numbers and improving the quality of the fused image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-modal medical image fusion method and model with variable modality input. Background Art

[0002] Medical imaging technology plays a vital role in modern medical diagnosis. Common medical imaging modalities include CT (computed tomography), MRI (magnetic resonance imaging), PET (positron emission tomography) and SPECT (single photon emission computed tomography). These different imaging modes can provide unique anatomical, functional or metabolic information. However, due to differences in imaging principles and objectives, single-modality medical images often cannot fully reflect the characteristics of biological tissues. For example, CT performs well in bone imaging, MRI has advantages in soft tissue imaging, and PET and SPECT can provide important functional data, but their spatial resolution is relatively low.

[0003] In order to overcome the limitations of single-modality images, multimodal medical image fusion (MMIF) technology has emerged. MMIF technology aims to integrate the advantages of different modal images to generate a fused image containing more comprehensive information, thereby providing a more reliable basis for clinical diagnosis and treatment decisions. At present, MMIF technology is mainly divided into two categories: two-modality image fusion method and three-modality image fusion method.

[0004] Among the two-modal image fusion methods, common techniques include methods based on autoencoders, convolutional neural networks, transformers, generative adversarial networks (GANs), or diffusion models. Among them, the GAN model has shown excellent performance in image fusion tasks, but its instability during training may affect the quality of the final fused image. Although the Diffusion model has solved the instability problem of GAN to a certain extent, its high computational overhead and long fusion time make it difficult to meet the real-time requirements of clinical applications.

[0005] Trimodal image fusion methods include traditional sparse representation and multi-scale transformation methods, as well as Diffusion and GAN methods based on deep learning. Although traditional methods have fast calculation speeds, they often lack sufficient robustness and are difficult to adapt to complex and changeable medical images. Although deep learning methods have superior performance, they have high requirements for computing resources and may bring additional hardware investment burdens to medical institutions.

[0006] Existing MMIF methods usually only support image fusion with a fixed number of modalities. This limitation brings a series of problems. First, the acquisition of each medical image requires patients to pay a corresponding fee. Although fusing medical images of more modalities can obtain more comprehensive information, it will also increase the financial burden on patients. Secondly, the fusion method with a fixed number of modalities lacks flexibility and cannot be adjusted according to actual clinical needs and available resources. Finally, the fusion of images with different numbers of modalities often requires the use of different models, which increases the complexity of model development and maintenance, and may also lead to inconsistencies between the fusion results of different numbers of modalities.

[0007] In addition, the existing fusion methods still have room for improvement in image quality. For example, how to effectively suppress noise and artifacts while retaining the unique information of each modality image and improving the contrast and clarity of the fused image is still an urgent problem to be solved.

[0008] In view of the above problems, the existing technology needs to be improved urgently. Summary of the invention

[0009] The purpose of this application is to provide a multi-modal medical image fusion method and model with variable modality input, which can adapt to the medical image fusion needs of different numbers of modalities and improve the quality of fused images.

[0010] In a first aspect, the present application provides a multimodal medical image fusion model with variable modality input, including a Mamba-based conditional diffusion fusion model and an expectation maximization module; The Mamba-based conditional diffusion fusion model is used to use a preset pure noise image as an initialized fusion image and gradually perform denoising on the fusion image; The expectation maximization module is used to correct the output features of the Mamba-based conditional diffusion fusion model after each step of denoising processing by adjusting the likelihood according to the input source image and the number of its modalities, so as to obtain the corrected output features; before the Mamba-based conditional diffusion fusion model completes the denoising processing of a preset number of steps, the corrected output features are used as the input of the fused image of the Mamba-based conditional diffusion fusion model for the next step of denoising processing; when the Mamba-based conditional diffusion fusion model completes the denoising processing of the preset number of steps, the corrected output features are output as the final fused image; in this way, the final fused image matches the input source image and the number of its modalities; the number of modalities of the input source image is 2 or 3.

[0011] The model gradually denoises the fused image through the conditional diffusion fusion model based on Mamba, and uses the expectation maximization module to correct the output features according to the input source image and its modality number, thereby realizing the flexible fusion of medical images with different modality numbers and improving the quality of the fused image. It has the advantages of being able to adapt to the fusion needs of medical images with different modality numbers and improving the quality of the fused image.

[0012] Preferably, when the expectation maximization module corrects the output features of each step of the denoising process of the Mamba-based conditional diffusion fusion model by adjusting the likelihood according to the input source image and the number of its modalities to obtain the corrected output features, the following steps are executed: The first reference feature of the correction is iteratively calculated using the coordinate descent method according to the following formula: ; ; Among them, k and u are constraint variables. is the first reference feature of the correction obtained in the current iteration step, is the first reference feature of the correction obtained in the previous iteration step, The initial value corresponds to The first reference feature x, is the output feature of the Mamba-based conditional diffusion fusion model after denoising at the current step, y is the second reference feature, is the inverse fast Fourier transform function, is the inverse fast Fourier transform function, is the gradient operator, is the gradient of k, express The complex conjugate of represents element-wise multiplication, and is a constant parameter, and Respectively represent the source images of the two modes in the input source image, M and N are latent variable matrices, and satisfy: ; ; ; ; ; ; ; in, is the element in the i-th row and j-th column of M, is the element in row i and column j in N, For the corresponding The intermediate amount, For the corresponding The intermediate amount, is the element in row i and column j in m, is the element in the i-th row and j-th column of n, m and n are the conditional expectation latent variable matrices of the image fusion probability log-likelihood function, is the element in row i and column j in y, is the element in row i and column j in x, is the source image of the third mode in the input source image. If the input source image only contains source images of two modes, then is a zero matrix, express The conditional expectation of express The conditional expectation of express expectations, express expectations, for The expected mean of for The expected mean value of , h is the height pixel size of the source image, w is the width pixel size of the source image; According to the corrected first reference feature, the corrected output feature is calculated using the following formula: ; in, Features are output for the correction.

[0013] The advantage of the above correction method is that it can adapt to input source images of different modalities, which improves the flexibility and versatility of the model. By performing correction after each denoising process, the fusion process can be continuously optimized and the quality of the final fused image can be improved. Compared with the existing technology, this method has stronger adaptability and higher fusion quality when processing multimodal medical image fusion.

[0014] Preferably, the Mamba-based conditional diffusion fusion model includes a first input layer, a patching block, a coding block, a plurality of sequentially connected DFM blocks, a first regularization layer, a linear layer and a first output layer; the patching block is connected between the first input layer and the first DFM block, the coding block is used to encode the time step and then input it into each DFM block; the last DFM block, the first regularization layer and the linear layer are sequentially connected; the output of the first input layer and the output of the linear layer are subtracted and then input into the first output layer; The patching block is used to patch the input image to generate multiple tag sequences; The DFM block is used to encode the input tag sequence; The linear layer is used to decrypt the label sequence to obtain and output the predicted noise.

[0015] This structural design enables the model to effectively process and fuse multimodal medical images, solving the structural design problem of the conditional diffusion fusion model based on Mamba. By rationally arranging various functional modules, the model can gradually denoise and extract features from the input image, and finally obtain a high-quality fused image.

[0016] Preferably, the DFM block includes a second input layer, a second regularization layer, a scaling and translation layer, a DF-Mamba block, a scaling layer, a first fully connected layer and a second output layer; the second input layer, the second regularization layer, the scaling and translation layer, the DF-Mamba block and the scaling layer are connected in sequence; the first fully connected layer is used to perform dimension alignment and feature transformation on the encoded time step output by the encoding block and then input it into the scaling layer; the output of the scaling layer and the output of the second input layer are added and input into the second output layer.

[0017] Preferably, the DF-Mamba block comprises a third input layer, a flattening and linear projection block, a diffusion fusion Mamba encoder, a second fully connected layer and a third output layer connected in sequence.

[0018] Preferably, the diffusion fusion Mamba encoder includes a fourth input layer, a third regularization layer, a forward propagation conv1d block, a forward propagation SSM block, an activation layer and a fourth output layer; the input end of the third regularization layer is connected to the output end of the fourth input layer, the output end of the third regularization layer is respectively connected to the input end of the forward propagation conv1d block and the input end of the activation layer, the input end of the forward propagation SSM block is connected to the output end of the forward propagation conv1d block, the output of the forward propagation SSM block is multiplied by the output of the activation layer and then added to the output of the fourth input layer, and the addition result is input into the fourth output layer.

[0019] In a second aspect, the present application provides a multi-modal medical image fusion method with variable modality input, comprising the steps of: A1. Constructing an initial multimodal medical image fusion model with variable modality input; the initial multimodal medical image fusion model with variable modality input is the multimodal medical image fusion model with variable modality input described above; A2. training the Mamba-based conditional diffusion fusion model in the initial multimodal medical image fusion model with variable modal input to obtain a trained multimodal medical image fusion model with variable modal input; A3. Using the trained multimodal medical image fusion model with variable modality input to perform image fusion on source images to be fused; the source images to be fused include source images of two modalities or source images of three modalities.

[0020] Preferably, step A2 comprises: A201. Acquire a medical image set, wherein the medical image set includes medical images of multiple different modalities; A202. Dividing the medical image set into a training set and a validation set; A203. Using the training set to train the Mamba-based conditional diffusion fusion model; A204. Use the validation set to validate the trained Mamba-based conditional diffusion fusion model.

[0021] Preferably, step A203 includes: B1. randomly selecting a medical image from the training set as a training image; B2. adding random noise to the training image step by step according to a preset number of steps; B3. Perform at each time step: Inputting the training image after adding random noise at the current time step into the Mamba-based conditional diffusion fusion model to obtain a predicted image of the training image before adding random noise at the current time step output by the Mamba-based conditional diffusion fusion model; Calculating a loss function based on the predicted image and the training image before adding random noise at the current time step; According to the loss function, updating the model parameters of the Mamba-based conditional diffusion fusion model through back propagation; B4. Repeat steps B1 to B3 until the preset end condition is met, then stop training.

[0022] Preferably, step A3 comprises: The source image to be fused is input into the expectation maximization module of the trained multimodal medical image fusion model with variable modal input, and the preset pure noise image is input into the Mamba-based conditional diffusion fusion model of the trained multimodal medical image fusion model with variable modal input to obtain a fused image output by the trained multimodal medical image fusion model with variable modal input.

[0023] Beneficial effects: The multimodal medical image fusion method and model with variable modality input provided in the present application gradually denoises the fused image through a conditional diffusion fusion model based on Mamba, and uses an expectation maximization module to correct the output features according to the input source image and its modality number, thereby realizing flexible fusion of medical images with different modality numbers and improving the quality of the fused image. It has the advantages of being able to adapt to the fusion needs of medical images with different modality numbers and improving the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of a multimodal medical image fusion model with variable modality input provided in an embodiment of the present application.

[0025] Figure 2 This is a structural diagram of the conditional diffusion fusion model based on Mamba.

[0026] Figure 3 This is a schematic diagram of the structure of the MFD block.

[0027] Figure 4 Schematic diagram of the structure of the DF-Mamba block.

[0028] Figure 5 Schematic diagram of the structure of the diffusion fusion Mamba encoder.

[0029] Figure 6 A flowchart of a multi-modal medical image fusion method with variable modality input provided in an embodiment of the present application.

[0030] Legend of the numbers: 1. First input layer; 2. Patching block; 3. Encoding block; 4. DFM block; 5. First regularization layer; 6. Linear layer; 7. First output layer; 8. Second input layer; 9. Second regularization layer; 10. Scaling and translation layer; 11. DF-Mamba block; 12. Scaling layer; 13. First fully connected layer; 14. Second output layer; 15. Third input layer; 16. Flattening and linear projection block; 17. Diffusion fusion Mamba encoder; 18. Second fully connected layer; 19. Third output layer; 20. Fourth input layer; 21. Third regularization layer; 22. Forward propagation conv1d block; 23. Forward propagation SSM block; 24. Activation layer; 25. Fourth output layer; 100. Conditional diffusion fusion model based on Mamba; 200. Expectation maximization module. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Multimodal medical image fusion technology plays an important role in clinical diagnosis and treatment. By integrating the advantages of different imaging modalities, such as CT, MRI, PET, and SPECT, fused images containing more comprehensive information can be generated. However, existing fusion methods usually only support image fusion with a fixed number of modalities, which limits their flexibility in practical applications. When faced with input images with different numbers of modalities, these methods are difficult to adapt, which may result in poor fusion effects or inability to perform fusion. In addition, existing methods often suffer from problems such as low computational efficiency and unstable fusion quality when dealing with high-dimensional and complex-structured medical images. These technical limitations have seriously affected the widespread application and effectiveness of multimodal medical image fusion technology in clinical practice.

[0034] If the problem of multimodal medical image fusion with variable modality input cannot be effectively solved, it will have many negative impacts on the medical diagnosis and treatment process. First, medical institutions may need to maintain multiple fusion systems for different modality combinations at the same time, which not only increases hardware and software costs, but also increases the complexity of system maintenance. Second, doctors may need to compare and weigh different fusion results when making a diagnosis, which increases the complexity of diagnosis and the potential risk of misdiagnosis. From a technical point of view, fusion methods that cannot adapt to different numbers of modalities may lead to insufficient information utilization in some cases, while introducing unnecessary noise or artifacts in other cases. This inconsistency may affect the accuracy and reliability of diagnosis. In addition, in the field of clinical research and medical image analysis, the lack of flexible fusion methods may limit the exploration and research of different modality combinations, thereby hindering the further development of medical imaging technology. Therefore, it is particularly important to develop a multimodal medical image fusion method that can adapt to different numbers of modalities, maintain high-quality fusion effects, and have good computational efficiency.

[0035] To do this, please refer to Figure 1-Figure 5 , the present application provides a multi-modal medical image fusion model with variable modality input, including a Mamba-based conditional diffusion fusion model 100 and an expectation maximization module 200; The Mamba-based conditional diffusion fusion model 100 is used to use a preset pure noise image as an initialized fusion image and gradually perform denoising on the fusion image; The expectation maximization module 200 is used to correct the output features of the Mamba-based conditional diffusion fusion model 100 after each step of denoising by adjusting the likelihood according to the input source image and the number of its modalities, so as to obtain the corrected output features; before the Mamba-based conditional diffusion fusion model 100 completes the denoising process for a preset number of steps, the corrected output features are used as the input fusion image of the Mamba-based conditional diffusion fusion model 100 for the next step of denoising; when the Mamba-based conditional diffusion fusion model 100 completes the denoising process for a preset number of steps, the corrected output features are output as the final fusion image (for example Figure 1 middle, is the output feature after denoising at each step of the conditional diffusion fusion model 100 based on Mamba, is the corrected output feature at each time step, For the final fused image, ); in this way, the final fused image matches the input source image and its number of modalities; the number of modalities of the input source image is 2 or 3.

[0036] The model gradually denoises the fused image through a Mamba-based conditional diffusion fusion model 100, and uses an expectation maximization module 200 to correct the output features according to the input source image and its modality number, thereby achieving flexible fusion of medical images with different modality numbers and improving the quality of the fused image. It has the advantages of being able to adapt to the fusion needs of medical images with different modality numbers and improving the quality of the fused image.

[0037] The preset pure noise image refers to a random noise image used as an initialization fusion image, and can be specifically generated by using Gaussian white noise or other types of random noise.

[0038] Among them, the preset number of steps can be adjusted according to actual needs.

[0039] The input source images may include two or three of a CT image, an MR-T1 image, an MR-T2 image, an MR-Gad image, a PET image, and a SPECT image.

[0040] The model works as follows: In each denoising step, the Mamba-based conditional diffusion fusion model 100 uses Mamba’s sequence modeling capabilities to process the input image features. The core of the Mamba model lies in its efficient state-space model structure that can capture long-range dependencies, which is particularly important for complex structures and detail preservation in medical images.

[0041] The expectation maximization module 200 receives two inputs: the output features of the Mamba-based conditional diffusion fusion model 100, and the original input source image and its modality number information. The expectation maximization module 200 corrects the output features by adjusting the likelihood. This process takes into account the input source image and its modality number, ensuring that the corrected features can fully reflect the information of all input modalities.

[0042] The corrected output features are then fed back to the Mamba-based conditional diffusion fusion model 100 as input for the next denoising process. This forms an iterative process, with each step optimizing the quality of the fused image.

[0043] After the preset number of steps of denoising is completed, the output features of the last correction are regarded as the final fused image. This fused image contains the key information of all input modalities, and its features match the number of modalities of the input source image.

[0044] The reason why this application chooses to combine the Mamba model and the conditional diffusion process is that the Mamba model shows better performance than the Transformer in sequence modeling tasks, while having higher computational efficiency. This enables the model to better handle complex structures and long-distance dependencies in medical images. The conditional diffusion process provides a progressive image generation method that helps to generate high-quality fused images. The introduction of the expectation maximization module 200 is to solve the problem of the variable number of modalities. By dynamically adjusting the likelihood, the module can optimize the fusion process according to the actual number of input modalities, ensuring that both bimodal and trimodal inputs can be properly processed and fused.

[0045] Preferably, when the expectation maximization module 200 corrects the output features of each step of the denoising process of the Mamba-based conditional diffusion fusion model 100 by adjusting the likelihood according to the input source image and the number of its modalities to obtain the corrected output features, the following is performed: The corrected first reference characteristic is calculated according to the following formula: (1); (2); Among them, k and u are constraint variables. is the first reference feature of the correction obtained in the current iteration step, is the first reference feature of the correction obtained in the previous iteration step, The initial value corresponds to The first reference feature x, is the output feature of the Mamba-based conditional diffusion fusion model after denoising at the current step, y is the second reference feature, is the inverse fast Fourier transform function, is the inverse fast Fourier transform function, is the gradient operator, is the gradient of k, express The complex conjugate of represents element-wise multiplication, and is a constant parameter, and Respectively represent the source images of the two modes in the input source image, M and N are latent variable matrices, and satisfy: (3); (4); (5); (6); (7); (8); (9); in, is the element in the i-th row and j-th column of M, is the element in row i and column j in N, For the corresponding The intermediate amount, For the corresponding The intermediate amount, is the element in row i and column j in m, is the element in the i-th row and j-th column of n, m and n are the conditional expectation latent variable matrices of the image fusion probability log-likelihood function, is the element in row i and column j in y, is the element in row i and column j in x, is the source image of the third mode in the input source image. If the input source image only contains source images of two modes, then is a zero matrix, express The conditional expectation of express The conditional expectation of express expectations, express expectations, for The expected mean of for The expected mean value of , h is the height pixel size of the source image, w is the width pixel size of the source image; According to the corrected first reference feature, the corrected output feature is calculated using the following formula: (10); in, To correct the output features.

[0046] The advantage of the above correction method is that it can adapt to input source images of different modalities, which improves the flexibility and versatility of the model. By performing correction after each denoising process, the fusion process can be continuously optimized and the quality of the final fused image can be improved. Compared with the existing technology, this method has stronger adaptability and higher fusion quality when processing multimodal medical image fusion.

[0047] The formula derivation process of the above correction method is as follows: First, the loss function of the free switching fusion task can be taken as: (11); in, is the weight (if the input source image only contains source images of two modalities, then is a zero matrix), Represents the 1-norm.

[0048] Let the first reference feature be , the second reference feature is (According to this definition, we can get formula (2), formula (9) and formula (10)). By replacing the variables in formula (11), the loss function of the free switching fusion task is converted to: (12); Obviously, y is a known quantity and x is an unknown quantity. The optimization problem of formula (12) can be transformed into a maximum likelihood inference problem through probability analysis. The log-likelihood function for the probability inference problem of image fusion is: (13); in, is the log-likelihood function, is the gradient of x, is the total variation penalty term, represents the 2-norm.

[0049] To solve this maximum likelihood inference problem, we can view it as a latent variable optimization problem and use the expectation maximization algorithm to obtain the optimal x.

[0050] The expected value of the log-likelihood function with respect to the probability distribution of the source image can be expressed as: (14); in, is the expected value of the log-likelihood function of the probability distribution relative to the source image, a and b are algebraic, For the corresponding The log-likelihood function of .

[0051] The latent variables in formula (14) and The conditional expectation of is: (15); (16); (17); (18); Assume two intermediate matrices and ,in, The element in the i-th row and j-th column is the intermediate quantity ,and , The element in the i-th row and j-th column is the intermediate quantity ,and , let's assume two latent variable matrices M and N, where the element in the i-th row and j-th column of M is ,and , the element in row i and column j in N is ,and ; Combining formula (15)-formula (18) we can get formula (3)-formula (8). Combining formula (13), formula (14) can be converted to: (19); in, for The gradient of for The element at row i and column j in .

[0052] According to formula (19), the semi-quadratic splitting algorithm is used to minimize Negative function: (20); Then the solution of formula (20) is transformed into an unconstrained optimization problem: (twenty one); The solution process of formula (21) can be transformed into: iteratively solving the following formula by coordinate descent method: ; That is, we get formula (1). It should be noted that in the first iteration of the iterative solution, x is calculated by formula (9) and used as Substitute the initial value of into the above formula to start the coordinate descent method iteration process.

[0053] Specifically, see Figure 2, the Mamba-based conditional diffusion fusion model 100 includes a first input layer 1, a patching block 2, a coding block 3, a plurality of sequentially connected DFM blocks 4, a first regularization layer 5, a linear layer 6 and a first output layer 7; the patching block 2 is connected between the first input layer 1 and the first DFM block 4, the coding block 3 is used to encode the time step and then input it into each DFM block 4 (t in the figure represents the time step information of the input coding block 3, and t1 represents the encoded time step information); the last DFM block 4, the first regularization layer 5 and the linear layer 6 are connected in sequence; the output of the first input layer 1 (i.e., the image input into the Mamba-based conditional diffusion fusion model 100) is subtracted from the output of the linear layer 6 and then input into the first output layer 7; The patching block 2 is used to patch the input image to generate multiple tag sequences; DFM block 4 is used to encode the input tag sequence; The linear layer 6 is used to decrypt the tag sequence to obtain and output the predicted noise (thereby, the output image of the first output layer 7 is the image obtained by subtracting the predicted noise from the image input to the Mamba-based conditional diffusion fusion model 100).

[0054] This structural design enables the model to effectively process and fuse multimodal medical images, solving the structural design problem of the Mamba-based conditional diffusion fusion model 100. By rationally arranging various functional modules, the model can gradually denoise and extract features from the input image, and finally obtain a high-quality fused image.

[0055] Specifically, the first input layer 1 receives the input image and provides a data source for subsequent processing. The patching block 2 divides the input image into p×p patches (p is a hyperparameter and can be set according to actual needs), and converts the input image into a number of labeled sequences of dimension d (the specific number is determined by the hyperparameter p, and d is set according to actual needs) by linearly embedding each patch (for example, sine-cosine position embedding), which helps the model better capture local features. The encoding block 3 encodes the time step and provides timing information for the DFM block 4, which helps the model understand the process of image change. The DFM block 4 is the core processing unit of the model. Through multiple series-connected DFM blocks 4, the model can gradually extract and fuse image features. The first regularization layer 5 and the linear layer 6 further process the output of the DFM block 4, which helps to improve the generalization ability and prediction accuracy of the model. The first output layer 7 obtains the final prediction result (i.e., the single-step denoising result) by subtracting the output of the first input layer 1 (i.e., the input image received by the first input layer 1) from the prediction noise output by the linear layer 6.

[0056] Among them, the linear layer 6 can adopt a standard linear decoder. Specifically, the linear layer 6 applies adaLN normalization (i.e., adaptive layer normalization) to decode each tag sequence into a p×p×2C tensor, where C represents the number of channels of the spatial input input to the DFM block 4, and then rearranges the decoded tag sequences back to their respective original spatial layouts to obtain prediction noise, where the prediction noise output is used to make a difference with the output of the first input layer 1.

[0057] Further, see Figure 3 , the DFM block 4 includes a second input layer 8, a second regularization layer 9, a scaling and translation layer 10, a DF-Mamba block 11, a scaling layer 12, a first fully connected layer 13 and a second output layer 14; the second input layer 8, the second regularization layer 9, the scaling and translation layer 10, the DF-Mamba block 11 and the scaling layer 12 are connected in sequence; the first fully connected layer 13 is used to perform dimension alignment and feature transformation on the time step output by the encoding block 3 (i.e., t1 in the figure) and then input it into the scaling layer 12; the output of the scaling layer 12 and the output of the second input layer 8 (i.e., the label sequence input to the DFM block 4) are added and then input into the second output layer 14.

[0058] Specifically, the second input layer 8, the second regularization layer 9, the scaling and translation layer 10, the DF-Mamba block 11 and the scaling layer 12 are connected in sequence to form the main processing path of the DFM block 4. The first fully connected layer 13 is used to align the dimensions and transform the features of the encoded time steps output by the encoding block 3, and then input the processed results to the scaling layer 12. Finally, the output of the scaling layer 12 is added to the output of the second input layer 8 to form a residual connection structure, and the result is input to the second output layer 14.

[0059] Each component of this structural design has its specific function and role. The second input layer 8 receives the input tag sequence and provides raw data for subsequent processing. The second regularization layer 9 standardizes the input, which helps to improve the stability and convergence speed of the model. The scaling and translation layer 10 performs linear transformation on the features to enhance the expressiveness of the model. The DF-Mamba block 11, as the core processing unit, is responsible for deep feature extraction and encoding of the tag sequence. The scaling layer 12 adjusts the output of the DF-Mamba block 11 to meet the needs of subsequent processing.

[0060] The first fully connected layer 13 is introduced to achieve effective fusion of temporal information and spatial features. It processes the time step output by the encoding block 3 to achieve dimension alignment and feature transformation to match the feature dimension of the backbone network. This design allows effective fusion of temporal information and spatial features, improving the model's ability to process time series data. Among them, the first fully connected layer 13 can be implemented with two layers of MLP (multi-layer perceptron).

[0061] The output of the scaling layer 12 is added to the output of the second input layer 8 to form a residual connection. This structure can alleviate the gradient vanishing problem in deep networks and facilitate lossless transmission of information, thereby improving the training efficiency and performance of the model.

[0062] Further, see Figure 4 , the DF-Mamba block 11 includes a third input layer 15, a flattening and linear projection block 16, a diffusion fusion Mamba encoder 17, a second fully connected layer 18 and a third output layer 19 which are connected in sequence.

[0063] Specifically, the third input layer 15 receives input data from the previous layer. This layer serves as the entrance to the DF-Mamba block 11 and is responsible for receiving and preliminarily processing the input labeled sequence. The flattening and linear projection block 16 flattens the input data and adjusts the data dimension through linear projection in preparation for subsequent processing. The flattening operation can convert multidimensional data into a one-dimensional vector for subsequent processing. The linear projection can adjust the dimension of the data to meet the input requirements of the diffusion fusion Mamba encoder 17. The diffusion fusion Mamba encoder 17, as the core processing unit, uses the Mamba structure for feature extraction and fusion. The Mamba structure is a new type of sequence modeling architecture that has higher computational efficiency and better long sequence modeling capabilities than the traditional Transformer structure. This feature is particularly important in medical image processing because medical images usually have high resolution and complex spatial structures. The second fully connected layer 18 performs further nonlinear transformation and dimensionality adjustment on the encoded features. The second fully connected layer 18 can capture the complex relationship between features and enhance the expression ability of the model. The third output layer 19 outputs the processed results as the final output of the DF-Mamba block 11.

[0064] This design achieves effective processing of input data by reasonably arranging the order and functions of each component. The flattening and linear projection block 16 first adjusts the data format to prepare for the processing of the diffusion fusion Mamba encoder 17. As the core, the diffusion fusion Mamba encoder 17 uses its powerful sequence modeling ability to deeply extract and fuse features. Finally, the second fully connected layer 18 and the third output layer 19 complete the final transformation and output of the features. This structural design helps to improve the model's ability to extract and fuse multimodal medical image features, thereby improving the quality of the final fused image. Due to the use of the Mamba structure, the design may have higher computational efficiency and better long sequence modeling capabilities, which is particularly beneficial for processing high-resolution medical images.

[0065] Further, see Figure 5The diffusion fusion Mamba encoder 17 includes a fourth input layer 20, a third regularization layer 21, a forward propagation conv1d block 22, a forward propagation SSM block 23, an activation layer 24 and a fourth output layer 25; the input end of the third regularization layer 21 is connected to the output end of the fourth input layer 20, the output end of the third regularization layer 21 is respectively connected to the input end of the forward propagation conv1d block 22 and the input end of the activation layer 24, the input end of the forward propagation SSM block 23 is connected to the output end of the forward propagation conv1d block 22, the output of the forward propagation SSM block 23 and the output of the activation layer 24 are multiplied and then added to the output of the fourth input layer 20 (i.e., the input feature of the diffusion fusion Mamba encoder 17), and the addition result is input to the fourth output layer 25.

[0066] Specifically, the third regularization layer 21 standardizes the input and divides it into two outputs. One output is processed by the feature extraction of the forward propagation conv1d block 22 and the long sequence modeling of the forward propagation SSM block 23 in turn, and the other output is processed by the activation layer 24 and used to gate the output of the forward propagation SSM block 23, and then the gated result is added to the input feature of the diffusion fusion Mamba encoder 17 for output.

[0067] This diffusion fusion Mamba encoder 17 design effectively solves the information transfer and interaction problems in multimodal medical image fusion, and provides a new solution for improving the quality of fused images and model efficiency.

[0068] refer to Figure 6 The present application also provides a multi-modal medical image fusion method with variable modality input, comprising the steps of: A1. Constructing an initial multimodal medical image fusion model with variable modality input; the initial multimodal medical image fusion model with variable modality input is the multimodal medical image fusion model with variable modality input mentioned above; A2. Training the Mamba-based conditional diffusion fusion model 100 in the initial multimodal medical image fusion model with variable modal input to obtain a trained multimodal medical image fusion model with variable modal input; A3. Perform image fusion on source images to be fused using a trained multimodal medical image fusion model with variable modality input; the source images to be fused include source images of two modalities or source images of three modalities.

[0069] Among them, in step A2, when performing model training, the Mamba-based conditional diffusion fusion model 100 in the initial multimodal medical image fusion model with variable modal input is trained separately (the expectation maximization module 200 does not participate in the training process), and the trained Mamba-based conditional diffusion fusion model 100 and the expectation maximization module 200 constitute the trained multimodal medical image fusion model with variable modal input.

[0070] Specifically, step A2 includes: A201. Obtain a medical image set, where the medical image set includes medical images of multiple different modalities; A202. Divide the medical image set into a training set and a validation set; A203. Train the Mamba-based conditional diffusion fusion model 100 using the training set; A204. Use the validation set to verify the trained Mamba-based conditional diffusion fusion model 100.

[0071] The medical image set may include at least three of CT images, MR-T1 images, MR-T2 images, MR-Gad images, PET images, and SPECT images, and each medical image set includes multiple images.

[0072] Among them, when dividing the training set and the validation set, the ratio of 8:2 or 7:3 can be used.

[0073] Wherein, step A203 includes: B1. Randomly select a medical image from the training set as a training image; B2. Add random noise to the training image step by step according to the preset number of steps; B3. Perform at each time step: The training image after random noise is added at the current time step is input into the Mamba-based conditional diffusion fusion model 100, and a predicted image of the training image before random noise is added at the current time step output by the Mamba-based conditional diffusion fusion model 100 is obtained (i.e., an image obtained after single-step denoising of the training image after random noise is added at the current time step); Calculate the loss function based on the predicted image and the training image before adding random noise at the current time step; According to the loss function, the model parameters of the Mamba-based conditional diffusion fusion model 100 are updated through back propagation; B4. Repeat steps B1 to B3 until the preset end condition is met, then stop training.

[0074] Specifically, in step B1, a random number generator can be used to select the image index in the training set, or a shuffle algorithm can be used to randomly sort the entire training set and then select them in sequence. This random selection method ensures the diversity of training data and helps to improve the generalization ability of the model.

[0075] In step B2, the preset number of steps can be set according to actual needs, and the random noise added step by step can be Gaussian noise, Poisson noise, salt and pepper noise, etc.

[0076] In step B3, the loss function may use mean square error (MSE), structural similarity index (SSIM) or perceptual loss, etc. For example, the calculation method of using mean square error (MSE) is as follows: ; in, is the loss function of the current time step, The predicted image of the training image before adding random noise to the current time step output by the Mamba-based conditional diffusion fusion model 100, The training image before adding random noise to the current time step (since random noise is added step by step in the training image, is known).

[0077] The process of backpropagation to update model parameters can use different optimizers, such as stochastic gradient descent (SGD), Adam, or RMSprop.

[0078] In step B4, the preset end condition can be based on multiple indicators, such as the number of training rounds, the performance indicators on the validation set, or the convergence degree of the loss function, etc. For example, a threshold can be set to stop training when the average peak signal-to-noise ratio (PSNR) on the validation set reaches a certain value.

[0079] Specifically, step A3 includes: The source image to be fused is input into the expectation maximization module 200 of the trained multimodal medical image fusion model with variable modal input, and the preset pure noise image is input into the Mamba-based conditional diffusion fusion model 100 of the trained multimodal medical image fusion model with variable modal input, so as to obtain a fused image output by the trained multimodal medical image fusion model with variable modal input.

[0080] In the following, the present invention is experimentally verified by using specific examples. Source images are obtained from the whole brain atlas database of Harvard Medical School and combined. The three-modality fusion combination includes five combinations: MR-T2 image + MR-Gad image + PET image, CT image + MR-T2 image + SPECT image, MR-T1 image + MR-T2 image + PET image, MR-T2 image + MR-Gad image + SPECT image, and MR-T1 image + MR-T2 image + SPECT image. The two-modality fusion combination includes three combinations: MRI image + CT image, MRI image + PET image, and MRI image + SPECT image.

[0081] The comparison models of trimodal fusion include BitonicX model, CTSR (Cognitive Trading System) model and GCN (Graph Convolutional Network) model. The comparison models of bimodal fusion include U2fusion (Unified Unsupervised Image Fusion Network) model, CDDFuse (Correlation-Driven Dual-Branch Feature Decomposition Fusion) model, DDFM (Denoising Diffusion Modelfor Multi-Modality Image Fusion) model, Swinfusion (Cross-domain Long-range Learning for General Image Fusion via Swin Transformer) model, SHIP (‌Synergistic High-Order Interaction) model, ALMFnet (Attention-based Layered Memory Fusion Network) model, DDcGAN (Dual-Discriminator Conditional Generative Adversarial Network) model, EMMA (End-to-End Multimodal Model for Autonomous Driving) model and Diff-IF (Multi-modality image fusion via diffusion model with fusion knowledgeprio) model. The source images of the obtained three-modal fusion combination are fused using the comparison model of the present application and the three-modal fusion, and the comparison of the evaluation indicators of the fusion results is shown in Table 1; the source images of the obtained two-modal fusion combination are fused using the comparison model of the present application and the two-modal fusion, and the comparison of the evaluation indicators of the fusion results is shown in Table 2.

[0082] Table 1 Model VIF SSIM PSNR FSIM MI QnI Q QUR BitonicX 9.2453 0.6099 48.228 0.8860 0.7146 0.8104 0.8985 1603.137 CTSR 6.2016 0.6056 48.298 0.8860 0.6437 0.8091 0.8985 1603.006 GCN 4.1390 0.6151 47.505 0.8872 0.6408 0.8087 0.8978 1948.991 This application 9.4243 0.6474 48.345 0.8909 0.6969 0.8191 0.8986 1596.948 Table 2 Model VIF SSIM PSNR FSIM MI QnI Q QUR U2fusion 0.2758 0.2536 16.777 0.760 0.636 0.8053 0.3531 937.048 CDDFuse 0.3697 0.7484 15.366 0.796 0.884 0.8104 0.8990 281.723 DDFM 0.3236 0.7328 16.491 0.787 0.936 0.8097 0.8240 883.323 Swinfusion 0.2287 0.4112 15.340 0.767 0.657 0.8059 0.5601 462.521 SHIP 0.2557 0.2837 14.852 0.766 0.623 0.8067 0.4550 260.863 ALMFnet 0.2947 0.7461 15.483 0.795 0.740 0.8069 0.8762 262.876 DDcGAN 0.2196 0.7166 14.871 0.768 0.645 0.8061 0.8351 404.063 EMMA 0.3277 0.7419 15.390 0.799 0.894 0.8098 0.8911 270.354 Diff-IF 0.0414 0.1487 11.261 0.646 0.468 0.8043 0.1972 4344.589 This application 0.3914 0.7493 16.839 0.795 0.921 0.8187 0.9087 261.561 In the table, VIF is visual information fidelity, SSIM is structural similarity, PSNR is peak signal-to-noise ratio, FSIM is feature similarity, MI is mutual information, Qncie (Quality Indicators of NVSS Checked by Image Experts) is an indicator used to evaluate image fusion quality, Qs is an objective non-reference quality evaluation indicator of image fusion, and Qcv (Chen-Varshney metric) is an image fusion quality evaluation indicator based on human perception.

[0083] It can be seen from the table that in the trimodal fusion experiment, the objective evaluation indicators of this application were compared with the existing models. The results show that this application performs well in multiple key indicators, fully demonstrating its advantages in the trimodal medical image fusion task. Overall, whether compared with a dedicated trimodal fusion model or a two-modal fusion model, this application performs well, especially in detail preservation, noise suppression, information fusion and subjective visual quality. ‌ In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0084] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0086] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0087] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal medical image fusion model with variable modality input, characterized in that: Includes the Mamba-based conditional diffusion fusion model and expectation maximization module; The Mamba-based conditional diffusion fusion model is used to use a preset pure noise image as an initialized fusion image and gradually perform denoising on the fusion image; The expectation maximization module is used to correct the output features of the Mamba-based conditional diffusion fusion model after each step of denoising processing by adjusting the likelihood according to the input source image and the number of its modalities, so as to obtain the corrected output features; before the Mamba-based conditional diffusion fusion model completes the denoising processing of a preset number of steps, the corrected output features are used as the input of the fused image of the Mamba-based conditional diffusion fusion model for the next step of denoising processing; when the Mamba-based conditional diffusion fusion model completes the denoising processing of the preset number of steps, the corrected output features are output as the final fused image; in this way, the final fused image matches the input source image and the number of its modalities; the number of modalities of the input source image is 2 or 3.

2. The multi-modal medical image fusion model with variable modality input according to claim 1, characterized in that: The expectation maximization module corrects the output features of each step of the denoising process of the Mamba-based conditional diffusion fusion model by adjusting the likelihood according to the input source image and the number of its modalities, and when obtaining the corrected output features, executes: The first reference feature of the correction is iteratively calculated using the coordinate descent method according to the following formula: ； ； Among them, k and u are constraint variables. is the first reference feature of the correction obtained in the current iteration step, is the first reference feature of the correction obtained in the previous iteration step, The initial value corresponds to The first reference feature x, is the output feature of the Mamba-based conditional diffusion fusion model after denoising at the current step, y is the second reference feature, is the inverse fast Fourier transform function, is the inverse fast Fourier transform function, is the gradient operator, is the gradient of k, express The complex conjugate of represents element-wise multiplication, and is a constant parameter, and Respectively represent the source images of the two modes in the input source image, M and N are latent variable matrices, and satisfy: ； ； ； ； ； ； ； in, is the element in the i-th row and j-th column of M, is the element in row i and column j in N, For the corresponding The intermediate amount, For the corresponding The intermediate amount, is the element in row i and column j in m, is the element in the i-th row and j-th column of n, m and n are the conditional expectation latent variable matrices of the image fusion probability log-likelihood function, is the element in row i and column j in y, is the element in row i and column j in x, is the source image of the third mode in the input source image. If the input source image only contains source images of two modes, then is a zero matrix, express The conditional expectation of express The conditional expectation of express expectations, express expectations, for The expected mean of for The expected mean value of , h is the height pixel size of the source image, w is the width pixel size of the source image; According to the corrected first reference feature, the corrected output feature is calculated using the following formula: ； in, Features are output for the correction.

3. The multi-modal medical image fusion model with variable modality input according to claim 1, characterized in that: The Mamba-based conditional diffusion fusion model includes a first input layer, a patching block, a coding block, a plurality of sequentially connected DFM blocks, a first regularization layer, a linear layer and a first output layer; the patching block is connected between the first input layer and the first DFM block, the coding block is used to encode the time step and then input it into each DFM block; the last DFM block, the first regularization layer and the linear layer are sequentially connected; the output of the first input layer and the output of the linear layer are subtracted and then input into the first output layer; The patching block is used to patch the input image to generate multiple tag sequences; The DFM block is used to encode the input tag sequence; The linear layer is used to decrypt the label sequence to obtain and output the predicted noise.

4. The multi-modal medical image fusion model with variable modality input according to claim 3, characterized in that: The DFM block includes a second input layer, a second regularization layer, a scaling and translation layer, a DF-Mamba block, a scaling layer, a first fully connected layer and a second output layer; the second input layer, the second regularization layer, the scaling and translation layer, the DF-Mamba block and the scaling layer are connected in sequence; the first fully connected layer is used to perform dimension alignment and feature transformation on the encoded time step output by the encoding block and then input it into the scaling layer; The output of the scaling layer and the output of the second input layer are added and then input into the second output layer.

5. The multi-modal medical image fusion model with variable modality input according to claim 4, characterized in that: The DF-Mamba block includes a third input layer, a flattening and linear projection block, a diffusion fusion Mamba encoder, a second fully connected layer and a third output layer connected in sequence.

6. The multi-modal medical image fusion model with variable modality input according to claim 5, characterized in that: The diffusion fusion Mamba encoder includes a fourth input layer, a third regularization layer, a forward propagation conv1d block, a forward propagation SSM block, an activation layer and a fourth output layer; the input end of the third regularization layer is connected to the output end of the fourth input layer, the output end of the third regularization layer is respectively connected to the input end of the forward propagation conv1d block and the input end of the activation layer, the input end of the forward propagation SSM block is connected to the output end of the forward propagation conv1d block, the output of the forward propagation SSM block is multiplied by the output of the activation layer and then added to the output of the fourth input layer, and the addition result is input into the fourth output layer.

7. A multi-modal medical image fusion method with variable modality input, characterized in that: Includes steps: A1. Constructing an initial multimodal medical image fusion model with variable modality input; the initial multimodal medical image fusion model with variable modality input is the multimodal medical image fusion model with variable modality input according to any one of claims 1 to 6; A2. training the Mamba-based conditional diffusion fusion model in the initial multimodal medical image fusion model with variable modal input to obtain a trained multimodal medical image fusion model with variable modal input; A3. Using the trained multimodal medical image fusion model with variable modality input to perform image fusion on source images to be fused; the source images to be fused include source images of two modalities or source images of three modalities.

8. The multi-modal medical image fusion method with variable modality input according to claim 7, characterized in that: Step A2 includes: A201. Acquire a medical image set, wherein the medical image set includes medical images of multiple different modalities; A202. Dividing the medical image set into a training set and a validation set; A203. Using the training set to train the Mamba-based conditional diffusion fusion model; A204. Use the validation set to validate the trained Mamba-based conditional diffusion fusion model.

9. The multi-modal medical image fusion method with variable modality input according to claim 8, characterized in that: Step A203 includes: B1. randomly selecting a medical image from the training set as a training image; B2. adding random noise to the training image step by step according to a preset number of steps; B3. Execute at each time step: Inputting the training image after adding random noise at the current time step into the Mamba-based conditional diffusion fusion model to obtain a predicted image of the training image before adding random noise at the current time step output by the Mamba-based conditional diffusion fusion model; Calculating a loss function based on the predicted image and the training image before adding random noise at the current time step; According to the loss function, updating the model parameters of the Mamba-based conditional diffusion fusion model through back propagation; B4. Repeat steps B1 to B3 until the preset end condition is met, then stop training.

10. The multi-modal medical image fusion method with variable modality input according to claim 7, characterized in that: Step A3 includes: The source image to be fused is input into the expectation maximization module of the trained multimodal medical image fusion model with variable modal input, and the preset pure noise image is input into the Mamba-based conditional diffusion fusion model of the trained multimodal medical image fusion model with variable modal input to obtain a fused image output by the trained multimodal medical image fusion model with variable modal input.

Images