Multi-modal medical image fusion and disease prediction method, computer program and terminal
Through the explicit classifier-guided diffusion model and cross-focus fusion module, the problem of unsatisfactory image quality and insufficient consistency in multimodal medical image fusion is solved, and high-quality multimodal medical image fusion and lesion area are achieved Visualization improves the effect of medical image fusion.
Patent Information
- Application Number
- CN202411092534.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing multimodal medical image fusion technology has problems such as high computational complexity, poor real-time performance, poor image quality, and insufficient correlation between generated samples and conditional factors. Especially in implicitly guided diffusion models, the generation of images cannot be guaranteed. Consistency with expected conditions.
Adopt an explicit classifier-guided diffusion model, combining the cross attention fusion module and the cross self-concern module, gradually add and remove noise through the reverse network and noise prediction network, use the classifier to guide the data recovery process of the noise prediction network, and use the cross attention to guide the cross attention process. The fusion module and the cross-concern module extract and fuse feature information of different modal medical images to generate multimodal fusion images.
Improve the quality and consistency of image fusion, enhance the visualization of lesion areas, ensure that the generated image strictly follows preset conditions, improve the effect of medical image fusion, and optimize pixel-level reconstruction accuracy and structural integrity through joint loss functions.
Smart Images

Figure CN118967480A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal medical image fusion and disease prediction method, a computer program and a terminal. Background Art
[0002] Brain imaging plays a vital role in the early detection of abnormalities and in monitoring and evaluating treatment. Currently, the main technologies for brain imaging include magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET). The lesion information provided by different imaging methods has different biological focuses, which makes it difficult for a single imaging technology to fully and deeply characterize brain phenotypes. For example, MRI provides high-resolution three-dimensional images, but has relatively low specificity and performs poorly in showing tiny lesions. CT performs well in depicting bones and other hard tissues, but has relatively low clarity in soft tissue imaging. Therefore, it is crucial to make full use of the advantages of various imaging technologies to provide more accurate and comprehensive diagnostic information for brain diseases.
[0003] Medical image fusion is an effective way to solve the above problems. Traditional image fusion techniques, such as pixel-, feature-, and decision-level fusion methods, have many technical defects. For example, the pixel-level fusion method is simple and direct, but it is easily affected by noise and contrast distortion, and it cannot fully utilize the complementary information when processing images of different modalities. Although the feature-level fusion method can alleviate the above problems to a certain extent, the extracted features may not represent the key information of the image, and it is difficult to effectively fuse the features of different modalities. The decision-level fusion method relies on the image analysis results of a single modality and cannot give full play to the advantages of multimodal information. These traditional methods generally have problems such as high computational complexity and poor real-time performance. With the widespread application of deep learning in medical image analysis, deep neural networks such as Convolutional Neural Network (CNN) and Generative Adversarial Network (GAN) automatically extract and combine image features from a large number of data sets by learning fusion strategies. Multimodal medical image fusion technology based on deep learning uses the efficient fitting ability of neural networks to effectively extract features and integrate information. In recent years, many scholars have also conducted in-depth explorations, which have promoted the development of medical image fusion technology. Although compared with traditional methods, deep learning has shown higher efficiency and stronger generalization ability in medical image fusion tasks, these methods have problems such as poor stability and lack of explanation, which limits the diversity of generated images and leads to unsatisfactory quality of fusion results.
[0004] As an emerging generative model, the diffusion model has shown great potential in the field of image fusion. The diffusion model achieves the generation and fusion of complex images by gradually adding noise and reverse denoising. Diffusion-based fusion methods generate high-quality images by simulating the diffusion process of restoring noise-damaged images to clean images. The image generation process based on denoising diffusion has high interpretability. In addition, the latent representations learned by (implicitly guided) diffusion models have been shown to be effective in discriminative tasks such as medical image segmentation, classification, and anomaly detection. Due to these advantages, recent studies have applied diffusion deterministic process models (DDPM) to medical image fusion, as medical images of different modalities provide different types of information. However, implicitly guided diffusion models introduce conditional information directly into neural networks, allowing neural networks to autonomously use this data to reconstruct the original dataset. However, this approach may sometimes result in the generated samples being weakly correlated with the conditional factors. During the training process, the model may not fully take the conditional factors into account. Although the training objectives are mainly defined in ∈-space (the feature space associated with noise), this conclusion also applies to x-space (the feature space where the original data is located). Therefore, the implicitly guided diffusion model cannot guarantee the generation of images consistent with the expected conditions. Summary of the invention
[0005] The purpose of the present invention is to overcome the problems of the prior art and to provide a multimodal medical image fusion and disease prediction method, a computer program and a terminal.
[0006] The objective of the present invention is to achieve the following technical solution: a multimodal medical image fusion method, the method comprising the following steps:
[0007] Constructing a feature fusion model and a diffusion model guided by an explicit classifier. The feature fusion model includes a cross-attention fusion module and a cross-self-attention module. The diffusion model guided by an explicit classifier includes a reverse network, a classifier, and a noise prediction network. The reverse network is connected to the noise prediction network, and the reverse network is connected to the noise prediction network via the classifier.
[0008] The diffusion model guided by the explicit classifier is trained, including: using an inverse network to gradually add noise to the feature vector of the original medical image at each time step to generate noisy data, a noise prediction network predicts the noise at each time step to remove the noise and then recover the data, the noise prediction network is trained by reducing the difference between the predicted noise and the actually added noise, and the classification result of the classifier on the noisy data is used to guide the data recovery process of the noise prediction network;
[0009] The cross-attention fusion module and the cross-self-attention module are used to extract the fusion feature information of medical images of different modalities, and then the fusion feature information is obtained by splicing.
[0010] The preliminary fusion features are input into the diffusion model guided by the trained explicit classifier, and the noise is removed using the noise prediction network to generate a multimodal fusion image.
[0011] In one example, the cross-attention fusion module is used to extract feature information of medical images of different modalities, including:
[0012] Feature extraction is performed on medical images of different modalities and medical images after splicing to obtain first feature information and second feature information; channel connection is performed on the first feature information and the second feature information, and then feature fusion is performed through a cross-attention module to obtain first feature fusion information; the first feature fusion information is dot-multiplied with the first feature information to generate a weighted feature map, and feature extraction is performed on the weighted feature map and then spliced with the second feature information to obtain a first preliminary fusion feature.
[0013] In one example, a cross self-attention module is used to extract feature information of medical images of different modalities, including:
[0014] Use different weight matrices to perform linear transformation on the first modality medical image and the second modality medical image respectively to obtain a first query q, a first key k and a first value v, as well as a second query q, a second key k and a second value v. Perform dot product operation and softmax processing on the first query q and the first key k to obtain a first attention weight. Perform dot product operation and softmax processing on the second query q and the second key k to obtain a second attention weight. Perform dot product operation on the first attention weight and the second value v, and then perform element addition processing on the second modality medical image to obtain a first preliminary attention feature. Perform dot product operation on the second attention weight and the first value v, and then perform element addition processing on the first modality medical image to obtain a second preliminary attention feature. Perform channel connection on the first preliminary attention feature and the second preliminary attention feature to obtain a second preliminary fusion feature.
[0015] In one example, the difference between the predicted noise and the actual added noise is measured by a loss function. The expression is:
[0016]
[0017] in, represents the expected operation, which means averaging all possible combinations of the original data x0 without noise added and the actual noise added ∈; ∈0 represents the predicted noise; Represents the cumulative noise figure.
[0018] In one example, the noisy data x at the previous time step is defined as t-1 By parameterizing the Gaussian distribution, the noise prediction network uses the modified distribution representation x under the guidance of the classifier t-1 :
[0019]
[0020] Where N represents Gaussian distribution; μ θ represents the mean; s represents the proportional factor that adjusts the influence of the classifier gradient on the diffusion trajectory; represents the gradient operator; P φ (y|x t ) means that given the noisy data x t Under the condition, the conditional probability distribution of the target fusion image data y output by the classifier; P φ Represents the probabilistic model of the classifier parameters φ.
[0021] In one example, the data update method of the data recovery process of the diffusion model guided by the explicit classifier is:
[0022] x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y;
[0023] Among them, h(t) represents the weight function at time step t.
[0024] In one example, the method further includes jointly training the feature fusion model and the diffusion model guided by the explicit classifier:
[0025] The feature fusion model and the diffusion model guided by the explicit classifier are trained according to the joint loss function formed by the mean square error loss and the structural similarity index loss.
[0026] It should be further explained that the technical features corresponding to the above examples can be combined or replaced with each other to form a new technical solution.
[0027] The present invention also includes a disease prediction method, which uses the image fusion method formed by any one of the above examples or a combination of multiple examples to obtain a multimodal fusion image, and the prediction method includes the following steps:
[0028] The disease prediction model extracts the feature vector of the multimodal fusion image, performs disease prediction based on the feature vector, and outputs the disease prediction result.
[0029] It should be further explained that the technical features corresponding to the above examples can be combined or replaced with each other to form a new technical solution.
[0030] The present invention also includes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal medical image fusion method formed by any one of the above examples or a combination of multiple examples, and implements the steps of the disease prediction method when the computer program is executed by a processor.
[0031] The present invention also includes a storage medium on which computer instructions are stored. When the computer instructions are executed, the steps of the multimodal medical image fusion method formed by any one or more of the above examples are executed, or the steps of the disease prediction method are executed when the computer instructions are executed.
[0032] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the steps of the multimodal medical image fusion method formed by any one or more of the above examples are executed, or when the processor executes the computer instructions, the steps of the disease prediction method described in the above examples are executed.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] 1. In one example, the diffusion model guided by an explicit classifier is applied to the field of medical image fusion. The guidance information of the classifier is explicitly used in the data recovery process to ensure that the generated image strictly follows the preset conditions at each stage, so as to make up for the problem that the implicit diffusion model cannot guarantee the generation of images consistent with the expected conditions, thereby ensuring the consistency and coherence of the generated images and improving the quality of image fusion, showing significant advantages in the field of medical image fusion.
[0035] At the same time, the cross-attention fusion module and the cross-self-attention module are used for feature extraction, and the multi-head self-attention mechanism is used to adaptively fuse the different modal features of the source image, thereby effectively utilizing the semantic features of each mode of the medical image to distinguish and adaptively fuse the effective low-frequency and high-frequency information of each mode, thereby enhancing the visualization of the lesion area and changing the tissue contrast, increasing the feature representation and improving the image quality.
[0036] 2. In one example, a joint loss function is used as the training objective, integrating the mean squared error (MSE) loss and the structural similarity index (SSIM) loss. The MSE loss is used to achieve pixel-level reconstruction accuracy, while the SSIM loss is combined to maintain structural integrity and improve image visual quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.
[0038] Figure 1 A method flow chart provided for an example of the present invention;
[0039] Figure 2 A schematic diagram of the Diff-MF model structure provided for an example of the present invention;
[0040] Figure 3 A schematic diagram of the structure of a cross-attention fusion module provided for an example of the present invention;
[0041] Figure 4 A schematic diagram of the cross self-attention module structure provided for an example of the present invention. DETAILED DESCRIPTION
[0042] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.
[0044] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0045] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0046] In one example, if Figure 1 As shown, a multimodal medical image fusion method comprises the following steps:
[0047] S1: Construct a diffusion model-based multi-modal fusion (Diff-MF) framework, including a sequentially connected feature fusion model and a diffusion model guided by an explicit classifier.
[0048] like Figure 2 As shown in FIG. 1 , the feature fusion model includes two concatenation (Concat) layers, a cross-attention fusion module (CrossAttention Fusion Block) and a cross-self-attention module (i.e., auxiliary network, AuxiliaryNetwork). The first concatenation layer (Concat1), the cross-attention fusion module, and the second concatenation layer (Concat2) are sequentially connected, and the cross-self-attention module is connected to the second concatenation layer. Medical images of different modalities are respectively input into the first concatenation layer, the cross-attention fusion module, and the cross-self-attention module. Medical images of different modalities can be any two or more of CT images, MRI images, PET images, etc., or a combination of two images (such as CT-MRI images, PET-MRI images). In this example, I i ,I j They are CT images and MRI images respectively.
[0049] Further, the diffusion model guided by the explicit classifier includes a reverse network, a classifier and a noise prediction network U-Net, the reverse network can be an autoencoder, the reverse network is connected to the noise prediction network, and the reverse network is connected to the noise prediction network via the classifier. Further, the second splicing layer, the first convolutional layer (Conv1), the reverse network, the third splicing layer (Concat3), and the second convolutional layer (Conv2) are connected in sequence, and the first splicing layer and the third splicing layer are globally residually connected. The reverse network and the noise prediction network form a complete diffusion model, and the diffusion model is divided into a forward process and a reverse process. The forward process gradually adds noise to the original data (the feature vector of the original medical image) to form noisy data; the reverse process is to gradually remove the noise and restore the real data. The present invention trains a noise prediction network so that the error of the actual added noise and the real noise in the forward process can be reduced in the restoration process, and the classifier guides the data generation in the reverse generation data generation process to avoid the generated data from deviating from the target fusion image data effect.
[0050] S2: Training the diffusion model guided by the explicit classifier, including: using the inverse network to gradually add noise to the feature vector of the original medical image at each time step to generate noisy data, the noise prediction network predicts the noise at each time step to remove the noise and then recover the data, training the noise prediction network by reducing the difference between the predicted noise and the actual added noise, and using the classification results of the classifier on the noisy data (the conditional probability distribution of the target fused image data output by the classifier under the condition of the noisy data) to guide the data recovery process of the noise prediction network.
[0051] Specifically, the dataset is first grouped, that is, CT-MRI images and PET-MRI images are grouped, and a specific number of pairs of images are randomly selected from each group of the dataset as a test set, and the remaining images are used for training.
[0052] Furthermore, the diffusion model is trained with the original image data. Given the original image data x0, noise is gradually added to the feature vector at each time step t to generate noisy data x T , x T Can be based on Directly derived from x0, where α t represents the noise coefficient at time step t; represents the cumulative noise factor, which can be expressed as α s represents the noise coefficient of time step s; N represents Gaussian distribution; α t =1-β t , β t It is used to control the noise intensity of each time step t. As the time step changes, β t It gradually increases, and the proportion of noise data gradually increases during the forward diffusion process. During the forward diffusion process, more and more noise data are added.
[0053] In the reverse process of the diffusion model, at each time step t, the noise is removed by the noise prediction network and the feature data is gradually restored. The diffusion model can be x T Convert to original data x0, q(x1,…,x T ) represents the joint probability distribution of data in the entire diffusion process; q(x T ) represents the probability distribution of the data at the final time step T; q(x t-1 |x t ) represents the data x given the current time step t t , predict the data x at the previous time step t-1 t-1 The conditional probability distribution of the noise prediction network parameterized by p θ (x t-1 |xt ) is: p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),∑ θ (x t ,t)); where p θ (x t-1 |x t ) is a neural network that learns the diffusion model to approximate the true distribution q(x t-1 |x t );∑ θ (x t ,t) represents the covariance matrix of the denoising network prediction, indicating the uncertainty of the prediction. ∑ θ (x t ,t) can be directly parameterized as β t or μ θ Represents the mean.
[0054] S3: Use the cross-attention fusion module and the cross-self-attention module to extract the fusion feature information of medical images of different modalities, and then perform splicing processing to obtain preliminary fusion features. Specifically, the first preliminary fusion feature output by the cross-attention fusion module and the second preliminary fusion feature output by the cross-self-attention module are spliced (Concat) through the second splicing layer to form a richer and more comprehensive feature expression (preliminary fusion feature), thereby improving the quality of the fused image.
[0055] S4: The preliminary fusion features are input into the diffusion model guided by the trained explicit classifier, and the noise is removed by using the noise prediction network to generate a multimodal fusion image. The effective low-frequency and high-frequency information of each mode is distinguished and adaptively fused to increase the feature representation and improve the image quality.
[0056] In one example, in order to distinguish and adaptively fuse the effective low-frequency and high-frequency information of each mode, the present invention designs a cross-attention fusion module, the input of which is a set of image data, and then a series of convolution, channel connection, dot multiplication and other operations are performed on the data to achieve adaptive fusion of multimodal features, and then the output is spliced to form a richer and more comprehensive feature expression, thereby improving the quality of the fused image. In view of the fact that PET images are in RGB color format and MRI images are in grayscale format, the present invention first converts the PET images to the YCbCr color space for processing. Since the Y channel carries the structure and brightness information of the image, and the Cb and Cr channels contain color information, only the Y channel is used for training during the fusion process. After the image fusion is generated, it is recombined with the Cb and Cr channels to output the final RGB format image. Specifically, the structure of the cross-attention fusion module is as follows: Figure 3 As shown in the figure, the cross-attention fusion module is used to extract the feature information of medical images of different modalities, including:
[0057] For different modal medical images I i ,I i , Medical image after stitching I i,j Perform feature extraction processing to obtain first feature information and second feature information; perform channel concatenation processing on the first feature information and the second feature information respectively, and then perform feature fusion processing through the cross attention module respectively to obtain the first feature fusion information A i , A j ; Fusion information A of the first feature i , A j Respectively with the first feature information (I i The characteristic information of i The feature information of the first convolution layer is processed by point-wise multiplication to generate a weighted feature map, and the weighted feature map is concatenated with the second feature information after feature extraction to obtain the first preliminary fusion feature. Among them, the cross attention module includes two convolution layers and a Sigmoid activation function layer connected in sequence. Each convolution layer uses a 3×3 kernel. The output of the first convolution layer is connected to an activation function layer (ReLU layer) to introduce nonlinearity. The subsequent Sigmoid activation function layer ensures that the output attention map is normalized and its value is limited to the range of [0,1].
[0058] In one example, in order to facilitate feature aggregation and capture long-term dependencies, a cross-self-attention module (auxiliary network) is introduced to evaluate the relationship between source images. Its model structure is as follows: Figure 4As shown in Figure 1, the cross self-attention module is used to extract feature information of medical images of different modalities, including:
[0059] Three different weight matrices are used to classify the medical image I i 、Medical Image I i Linear transformation is performed to obtain the first query q, the first key k and the first value v and the second query q, the second key k and the second value v. The first query q and the first key k are subjected to point-wise multiplication and softmax processing to obtain the first attention weight. The second query q and the second key k are subjected to point-wise multiplication and softmax processing to obtain the second attention weight. The first attention weight and the second value v are subjected to element-wise addition processing to obtain the first preliminary attention feature after the dot product operation is performed, and the second attention weight and the first value v are subjected to element-wise addition processing to obtain the second preliminary attention feature. The first preliminary attention feature and the second preliminary attention feature are subjected to channel concatenation to obtain the second preliminary fusion feature, thereby realizing the multi-modal feature adaptive fusion of a set of image data information through the multi-head self-attention mechanism. For example, in MRI images, low-frequency information includes the overall shape of the brain and the boundary data information of the main area. In CT images, high-frequency information includes the edge of the bone and the small lesion area. The multi-head self-attention mechanism can be used to adaptively fuse different feature information.
[0060] In the cross self-attention module, the present invention initially uses a 1×1 convolution kernel to concatenate the input image I i and I i Generate a 32-channel feature map. The cross-self-attention mechanism is defined as: where y i,j represents the output value of the cross-self-attention mechanism; f i 、f j represents the infrared image part and the visible light image part in the CT image and MRI image respectively; the function f is used to calculate the adaptive pixel-level weight vector of the source image; the function g derives the feature representation from each image; the normalization factor C(x) is calculated as: C(x) = ∑f(f i ,f j ), thereby ensuring that the cross-self-attention module effectively integrates the features of the two modes and enhances the ability to interpret complex visual information.
[0061] In one example, the diffusion model guided by the explicit classifier is to train a noise prediction network to predict the noise ∈ θ (x t,t), the loss function is the mean square error, which measures the difference between the predicted noise and the actual added noise. The expression is:
[0062]
[0063] in, represents the expected operation, which means averaging all possible combinations of the original data x0 without noise added and the actual noise added ∈ to minimize the prediction error, and then train the noise prediction network; ∈0 represents the predicted noise; Represents the cumulative noise figure.
[0064] In one example, the diffusion model guided by an explicit classifier is to train a classifier on noisy data and use the gradient To guide the diffusion sampling process (the reverse process of the diffusion model), assume that x t-1 is a parameterized Gaussian distribution. Under the guidance of the classifier, the noise prediction network uses the modified distribution to represent x t-1 :
[0065]
[0066] Among them, s represents the proportional factor that adjusts the influence of the classifier gradient on the diffusion trajectory; represents the gradient operator; P φ (y|x t ) means that given the noisy data x t Under the condition, the conditional probability distribution of the target fusion image data y output by the classifier; P φ represents the probability model of the classifier parameter φ. In practice, the present invention changes the optimization objective from maximizing P φ (y|x t ) is transferred to minimizing the difference between the two variables, specifically using the weighted mean square error (MSE) criterion as a method to minimize the difference:
[0067]
[0068] Among them, h(t) represents the weight function of time step t; x t represents the data generated by the diffusion model at each time step t. As the timestamp t decreases and the reverse sampling continues, the generated fused image should get closer and closer to the real image. In order to ensure the consistency of the generated samples with the ground truth source and target, the gradient guidance should be strengthened accordingly. This application assumes that the defined weighted mean square error criterion exhibits low curvature as t changes, especially when the number of diffusion steps is large. Under these conditions, the weighted mean square error criterion is used to calculate the weighted mean square error of the image. t-1 Replace x t, in order to simplify the diffusion operation while maintaining its integrity and effectiveness. Therefore, it can be approximately expressed as:
[0069]
[0070] h(t)=N0e -λt
[0071] In the formula, h(·) is determined by the exponential decay function; λ is a hyperparameter representing the exponential decay constant; N0 represents an additional hyperparameter that controls the adjustment strength. Differentiating the above formula, at timestamp t, x t The gradient becomes 2h(t-1)(x t-1 -y), substitute the gradient into the formula Setting the product of s∑ to a constant (1 / 2), we get: t-1 ~N(μ-h(t-1)(x t-1 -y),∑). For image fusion, this application first starts from N(μ θ (·),∑) generates candidate x t Then, this application calculates x t The difference between y and y, and further calculate the gradient 2h(t)(x t -y), finally, this application will x t Update to x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y. Therefore, the diffusion model guided by the explicit classifier of the present application is based on x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y is updated and trained with Target.
[0072] In one example, the method further includes jointly training the feature fusion model and the diffusion model guided by the explicit classifier:
[0073] The feature fusion model and the diffusion model guided by the explicit classifier are trained according to the joint loss function formed by the mean square error loss and the structural similarity index loss.
[0074] Specifically, in order to stabilize the convergence of Diff-MF training, the present invention uses the joint loss function L FIJAs a training target, the joint loss function integrates the mean square error loss and the structural similarity index loss, taking advantage of the MSE loss to achieve pixel-level reconstruction accuracy, while combining the SSIM loss to maintain structural integrity and improve image visual quality. MSE , SSIM loss L SSIM , joint loss function L FIJ The calculation expressions are:
[0075]
[0076] L FIJ =λ1L MSE +λ2L SSIM
[0077] Among them, μ x , μ y Respectively represent the mean of x and y; σ yx represents the covariance of x and y; σ y 2 , σ x 2 Represent the variance of x and y respectively; c1 and c2 are two constants; λ1,λ2∈[0,1] represent the weights of the two losses respectively. Using these two loss components, L FIJ At the same time, it effectively optimizes pixel accuracy and visual quality, thereby balancing different visual and quality indicators to achieve better image enhancement effects.
[0078] In the training phase, this application sets the diffusion time step T to 1000 and uses linear β scheduling with initial and final values of 10 -6 and 10 -2 The total number of training iterations is 14,000, with validation and checkpoint saving every 2,000 iterations. The learning rate used is 10 -4 The model is trained with the Adam optimizer until convergence.
[0079] The above examples are combined to obtain a preferred example of the present invention, in which the image fusion method includes:
[0080] S10: Construct feature fusion model and diffusion model guided by explicit classifier;
[0081] S20: training a diffusion model guided by an explicit classifier, and jointly training a feature fusion model and a diffusion model guided by an explicit classifier; wherein the training of the diffusion model guided by an explicit classifier includes: using an inverse network to gradually add noise to a feature vector of an original medical image at each time step to generate noisy data, a noise prediction network predicts the noise at each time step to remove the noise and then recover the data, training the noise prediction network by reducing the difference between the predicted noise and the actually added noise, and using the classification result of the classifier on the noisy data to guide the data recovery process of the noise prediction network;
[0082] S30: using a cross-attention fusion module and a cross-self-attention module to respectively extract fusion feature information of medical images of different modalities, and then performing splicing processing to obtain preliminary fusion features;
[0083] S40: Input the preliminary fusion features into the diffusion model guided by the trained explicit classifier, use the noise prediction network to remove noise, and generate a clear multimodal fusion image.
[0084] This paper proposes a multimodal medical image fusion method based on diffusion model and cross-attention fusion, which uses the cross-attention module to adaptively fuse the image data features, and uses the diffusion model guided by an explicit classifier to generate a clear fused image for the fused noisy image.
[0085] Based on the inventive concept of the image fusion method proposed in one or more of the above examples, the present invention proposes a disease prediction method, which includes the following steps:
[0086] The disease prediction model extracts the feature vector of the multimodal fusion image, performs disease prediction based on the feature vector, and outputs the disease prediction result. The disease prediction model preferably includes the feature fusion model, the diffusion model guided by the explicit classifier, and the classifier connected to the output end of the diffusion model guided by the explicit classifier. After the feature fusion model, the diffusion model guided by the explicit classifier, and the classifier are jointly trained, the above prediction method can be executed, and then the patient's disease development can be predicted based on the multimodal fusion image, providing a reference for disease diagnosis.
[0087] An example of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal medical image fusion method or disease prediction method formed by any one of the above examples or a combination of multiple examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.
[0088] An example of the present invention also provides a storage medium, which has the same inventive concept as a multimodal medical image fusion method or a disease prediction method formed by any one of the above examples or a combination of multiple examples, and stores computer instructions thereon, which, when executed, execute the steps of the multimodal medical image fusion method or disease prediction method formed by any one of the above examples or a combination of multiple examples.
[0089] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0090] An example of the present invention further provides a terminal, which has the same inventive concept as any example or a combination of multiple examples corresponding to the above-mentioned multimodal medical image fusion method or disease prediction method, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned multimodal medical image fusion method or disease prediction method when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.
[0091] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).
[0092] The storage unit stores a program code, and the program code can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned multimodal medical image fusion method.
[0093] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).
[0094] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0095] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0096] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0097] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.
[0098] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A multimodal medical image fusion method, characterized in that: The following steps are involved: Constructing a feature fusion model and a diffusion model guided by an explicit classifier. The feature fusion model includes a cross-attention fusion module and a cross-self-attention module. The diffusion model guided by an explicit classifier includes a reverse network, a classifier, and a noise prediction network. The reverse network is connected to the noise prediction network, and the reverse network is connected to the noise prediction network via the classifier. The diffusion model guided by the explicit classifier is trained, including: using an inverse network to gradually add noise to the feature vector of the original medical image at each time step to generate noisy data, a noise prediction network predicts the noise at each time step to remove the noise and then recover the data, the noise prediction network is trained by reducing the difference between the predicted noise and the actually added noise, and the classification result of the classifier on the noisy data is used to guide the data recovery process of the noise prediction network; The cross-attention fusion module and the cross-self-attention module are used to extract the fusion feature information of medical images of different modalities, and then the fusion feature information is obtained by splicing. The preliminary fusion features are input into the diffusion model guided by the trained explicit classifier, and the noise is removed using the noise prediction network to generate a multimodal fusion image.
2. The multimodal medical image fusion method according to claim 1, characterized in that: The cross-attention fusion module is used to extract feature information of medical images of different modalities, including: Feature extraction is performed on medical images of different modalities and medical images after splicing to obtain first feature information and second feature information; channel connection is performed on the first feature information and the second feature information, and then feature fusion is performed through a cross-attention module to obtain first feature fusion information; the first feature fusion information is dot-multiplied with the first feature information to generate a weighted feature map, and feature extraction is performed on the weighted feature map and then spliced with the second feature information to obtain a first preliminary fusion feature.
3. The multimodal medical image fusion method according to claim 1, characterized in that: The cross self-attention module is used to extract feature information of medical images of different modalities, including: Use different weight matrices to perform linear transformation on the first modality medical image and the second modality medical image respectively to obtain a first query q, a first key k and a first value v, as well as a second query q, a second key k and a second value v. Perform dot product operation and softmax processing on the first query q and the first key k to obtain a first attention weight. Perform dot product operation and softmax processing on the second query q and the second key k to obtain a second attention weight. Perform dot product operation on the first attention weight and the second value v, and then perform element addition processing on the second modality medical image to obtain a first preliminary attention feature. Perform dot product operation on the second attention weight and the first value v, and then perform element addition processing on the first modality medical image to obtain a second preliminary attention feature. Perform channel connection on the first preliminary attention feature and the second preliminary attention feature to obtain a second preliminary fusion feature.
4. The multimodal medical image fusion method according to claim 1, characterized in that: The difference between the predicted noise and the actual added noise is measured by the loss function. The expression is: in, represents the expected operation, which means averaging all possible combinations of the original data x0 without noise added and the actual noise added ∈; ∈0 represents the predicted noise; Represents the cumulative noise figure.
5. The multimodal medical image fusion method according to claim 1, characterized in that: The classification results of the classifier on the noisy data are used to guide the data recovery process of the noise prediction network, including: Define the noisy data x at the previous time step t-1 By parameterizing the Gaussian distribution, the noise prediction network uses the modified distribution representation x under the guidance of the classifier t-1 : Where N represents Gaussian distribution; μ θ represents the mean; s represents the proportional factor that adjusts the influence of the classifier gradient on the diffusion trajectory; represents the gradient operator; P φ (y|x t ) means that given the noisy data x t Under the condition, the conditional probability distribution of the target fusion image data y output by the classifier; P φ Represents the probabilistic model of the classifier parameters φ.
6. The multimodal medical image fusion method according to claim 5, characterized in that: The data update method of the data recovery process of the diffusion model guided by the explicit classifier is: x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y; Among them, h(t) represents the weight function at time step t.
7. The multimodal medical image fusion method according to claim 1, characterized in that: The method further includes jointly training the feature fusion model and the diffusion model guided by the explicit classifier: The feature fusion model and the diffusion model guided by the explicit classifier are trained according to the joint loss function formed by the mean square error loss and the structural similarity index loss.
8. A disease prediction method, characterized in that: The multimodal fusion image is obtained by using the image fusion method according to any one of claims 1 to 7, and the prediction method comprises the following steps: The disease prediction model extracts the feature vector of the multimodal fusion image, performs disease prediction based on the feature vector, and outputs the disease prediction result.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal medical image fusion method described in any one of claims 1 to 7 are implemented, or when the computer program is executed by a processor, the steps of the disease prediction method described in claim 8 are implemented.
10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, it executes the steps of the multimodal medical image fusion method described in any one of claims 1 to 7, or when the processor runs the computer instructions, it executes the steps of the disease prediction method described in claim 8.
Citation Information
Patent Citations
Multimodal fusion audio generation method and device based on diffusion model
CN116884391A
Characteristic function-based conditional generative adversarial network training method and device
CN117408313A
Super-resolution tri-modal medical fusion image generation method
CN118314021A
Multi-modal image classification system and method using attention-based multi-interaction network
US20230377318A1
Cited By
Cross-modal medical image registration method based on submerged space diffusion model
CN119494863A
A cross-modal medical image registration method based on latent space diffusion model
CN119494863B
3D Gaussian model generation and rendering method for low-dose PET image processing
CN119625190A
Steel-wood combined beam column joint damage identification method and system applying machine vision
CN120014284A
Method and system for identifying damage at steel-wood composite beam-column joints using machine vision
CN120014284B
