A multi-modal medical image fusion and disease prediction method, computer program and terminal

Abstract

The application discloses a kind of multimodal medical image fusion and prediction method, computer program and terminal, belong to image processing technical field, construct feature fusion model, explicit classifier guided diffusion model;The explicit classifier guided diffusion model is trained;The fusion feature information of different modal medical images is extracted, and input is completed the explicit classifier guided diffusion model of training, remove noise using noise prediction network, generate multimodal fusion image.The application applies explicit classifier guided diffusion model to medical image fusion field, explicitly uses the guidance information of classifier in data recovery process, to ensure that the generated image strictly follows preset condition at each stage, to make up the problem that implicit diffusion model cannot guarantee that the generated image is consistent with the expected condition, to ensure the consistency and coherence of the generated image, generate clear multimodal fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal medical image fusion and disease prediction method, computer program and terminal. Background Technology

[0002] Brain imaging plays a crucial role in the early detection of abnormalities and in monitoring and evaluating treatment. Currently, the main brain imaging techniques include magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET). Different imaging modalities provide lesion information with different biological focuses, making it difficult for a single imaging technique to comprehensively and deeply characterize the brain phenotype. For example, MRI provides high-resolution three-dimensional images but has relatively low specificity and performs poorly in displaying small lesions. CT excels in depicting bone and other hard tissues but has relatively low clarity in soft tissue imaging. Therefore, it is essential to fully utilize the advantages of various imaging technologies to provide more accurate and comprehensive diagnostic information for brain diseases.

[0003] Medical image fusion is an effective way to solve the above problems. Traditional image fusion techniques, such as pixel-based, feature-based, and decision-based fusion methods, have many technical shortcomings. For example, pixel-level fusion methods are simple and direct, but are easily affected by noise and contrast distortion, and cannot fully utilize their complementary information when processing images of different modalities. Feature-level fusion methods can alleviate the above problems to some extent, but the features extracted may not represent the key information of the image, and it is difficult to effectively fuse features from different modalities. Decision-level fusion methods rely on the image analysis results of a single modality and cannot fully utilize the advantages of multimodal information. These traditional methods generally suffer from high computational complexity and poor real-time performance. With the widespread application of deep learning in medical image analysis, deep neural networks such as Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) automatically extract and combine image features from large datasets by learning fusion strategies. Deep learning-based multimodal medical image fusion technology utilizes the efficient fitting ability of neural networks to effectively extract features and integrate information. In recent years, many scholars have conducted in-depth research, which has promoted the development of medical image fusion technology. Although deep learning has shown higher efficiency and stronger generalization ability in medical image fusion tasks compared with traditional methods, these methods suffer from problems such as poor stability and lack of interpretation, which limit the diversity of generated images and result in less than ideal quality of fusion results.

[0004] Diffusion models, as an emerging generative model, have shown great potential in the field of image fusion. Diffusion models achieve the generation and fusion of complex images through a process of progressively adding noise and then reversing the noise reduction. Diffusion-based fusion methods generate high-quality images by simulating the diffusion process of restoring a noisy, damaged image to a clean one, and the image generation process based on denoising diffusion has high interpretability. Furthermore, the latent representations learned by (implicitly guided) diffusion models have proven effective in discriminative tasks such as medical image segmentation, classification, and anomaly detection. Due to these advantages, recent research has applied Diffusion Deterministic Process Models (DDPM) to medical image fusion, as different modalities of medical images provide different types of information. However, implicitly guided diffusion models directly introduce conditional information into neural networks, allowing the neural network to autonomously reconstruct the original dataset using this data. However, this method may sometimes result in the generated samples having a weak correlation with the conditional factors. During the training process, the model may not have fully considered the conditional factors. Although the training objective is mainly defined in the ∈-space (the feature space related to noise), this conclusion also applies to the x space (the feature space where the original data is located). Therefore, the implicitly guided diffusion model cannot guarantee that the generated image is consistent with the expected conditions. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems of the prior art and provide a multimodal medical image fusion and disease prediction method, computer program and terminal.

[0006] The objective of this invention is achieved through the following technical solution: a multimodal medical image fusion method, which includes the following steps:

[0007] Construct a feature fusion model and an explicit classifier-guided diffusion model. The feature fusion model includes a cross-attention fusion module and a cross-self-attention module. The explicit classifier-guided diffusion model includes an inverse network, a classifier, and a noise prediction network. The inverse network is connected to the noise prediction network, and the inverse network is connected to the noise prediction network via the classifier.

[0008] Training the diffusion model guided by the explicit classifier includes: using an inverse network to progressively add noise to the feature vector of the original medical image at each time step to generate noisy data; a noise prediction network to remove noise and recover the data by predicting the noise at each time step; training the noise prediction network by reducing the difference between the predicted noise and the actual added noise; and using the classification results of the classifier on the noisy data to guide the data recovery process of the noise prediction network.

[0009] The cross-attention fusion module and the cross-self-attention module are used to extract the fusion feature information of medical images of different modalities, and then the images are stitched together to obtain the preliminary fusion features.

[0010] The initial fused features are input into a diffusion model guided by an explicit classifier that has been trained. Noise is removed using a noise prediction network to generate a multimodal fused image.

[0011] In one example, a cross-attention fusion module is used to extract feature information from different modalities of medical images, including:

[0012] Feature extraction is performed on medical images of different modalities and medical images after stitching to obtain first feature information and second feature information. After channel connection processing of the first feature information and second feature information, feature fusion processing is performed through a cross-attention module to obtain first feature fusion information. The first feature fusion information and the first feature information are multiplied by a dot product to generate a weighted feature map. After feature extraction of the weighted feature map, it is stitched with the second feature information to obtain the first preliminary fusion feature.

[0013] In one example, a cross-focusing module is used to extract feature information from different modalities of medical images, including:

[0014] Linear transformations are performed on the first and second modal medical images using different weight matrices to obtain a first query q, a first key k, and a first value v, as well as a second query q, a second key k, and a second value v. The first query q and the first key k are then subjected to a dot product and softmax operation to obtain the first attention weight. The second query q and the second key k are then subjected to a dot product and softmax operation to obtain the second attention weight. The first attention weight and the second value v are then subjected to a dot product operation and element-adding operation with the second modal medical image to obtain the first preliminary attention feature. The second attention weight and the first value v are then subjected to a dot product operation and element-adding operation with the first modal medical image to obtain the second preliminary attention feature. The first preliminary attention feature and the second preliminary attention feature are then concatenated to obtain the second preliminary fusion feature.

[0015] In one example, a loss function is used to measure the difference between the predicted noise and the actual added noise. The expression is:

[0016]

[0017] in, The expected operation represents the averaging of all possible combinations of the original data x0 without noise and the actual noise ∈; ∈0 represents the predicted noise; This represents the cumulative noise figure.

[0018] In one example, the noisy data x from the previous time step is defined. t-1 Based on a parameterized Gaussian distribution, a noise prediction network, guided by a classifier, uses a modified distribution representation of x. t-1 :

[0019]

[0020] Where N represents a Gaussian distribution; μ θ The mean is represented by s; s represents the scaling factor that adjusts the influence of the classifier gradient on the diffusion trajectory. P represents the gradient operator; φ (y|x t ) represents the situation where, given noisy data x t Under the given conditions, the conditional probability distribution of the target fused image data y output by the classifier; P φ This represents the probabilistic model for the classifier parameter φ.

[0021] In one example, the data update method for the data recovery process of the explicit classifier-guided diffusion model is as follows:

[0022] x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y;

[0023] Where h(t) represents the weight function for time step t.

[0024] In one example, the method further includes jointly training the feature fusion model and the diffusion model guided by the explicit classifier:

[0025] The feature fusion model and the explicit classifier-guided diffusion model are trained using a joint loss function formed by mean squared error loss and structural similarity index loss.

[0026] It should be further noted that the technical features corresponding to the above examples can be combined or substituted to form new technical solutions.

[0027] The present invention also includes a disease prediction method, which uses the image fusion method formed by any or more of the above examples to obtain a multimodal fused image, the prediction method comprising the following steps:

[0028] The disease prediction model extracts feature vectors from the multimodal fusion image, performs disease prediction based on the feature vectors, and outputs the disease prediction results.

[0029] It should be further noted that the technical features corresponding to the above examples can be combined or substituted to form new technical solutions.

[0030] The present invention also includes a computer program product comprising a computer program that, when executed by a processor, implements the steps of the multimodal medical image fusion method formed by any or a combination of the above examples, and the steps of the disease prediction method when executed by a processor.

[0031] The present invention also includes a storage medium storing computer instructions that, when executed, perform the steps of the multimodal medical image fusion method formed by any or more of the above examples, or perform the steps of the disease prediction method when executed.

[0032] The present invention also includes a terminal comprising a memory and a processor, the memory storing computer instructions executable on the processor, wherein the processor executes the steps of the multimodal medical image fusion method formed by any or more of the above examples when executing the computer instructions, or executes the steps of the disease prediction method described in the above examples when executing the computer instructions.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] 1. In one example, an explicit classifier-guided diffusion model is applied to the field of medical image fusion. During the data recovery process, the guidance information of the classifier is explicitly used to ensure that the generated images strictly follow the preset conditions at each stage. This compensates for the problem that the implicit diffusion model cannot guarantee that the generated images are consistent with the expected conditions, thereby ensuring the consistency and coherence of the generated images and improving the quality of image fusion. This demonstrates significant advantages in the field of medical image fusion.

[0035] Simultaneously, feature extraction is performed using a cross-attention fusion module and a cross-self-attention module. The multi-head self-attention mechanism is used to adaptively fuse different modal features of the source image, thereby effectively utilizing the semantic features of each mode of medical image to distinguish and adaptively fuse effective low-frequency and high-frequency information of each mode. This enhances the visualization of lesion areas, changes tissue contrast, increases feature representation, and improves image quality.

[0036] 2. In one example, a joint loss function is used as the training objective, integrating Mean Squared Error (MSE) loss and Structural Similarity Index (SSIM) loss. MSE loss is used to achieve pixel-level reconstruction accuracy, while SSIM loss is combined to maintain structural integrity and improve image visual quality. Attached Figure Description

[0037] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0038] Figure 1 A flowchart illustrating a method provided as an example of the present invention;

[0039] Figure 2 This is a schematic diagram of the Diff-MF model structure provided as an example of the present invention;

[0040] Figure 3 This is a schematic diagram of the cross-attention fusion module structure provided in an example of the present invention;

[0041] Figure 4 This is a schematic diagram of the cross-self-focus module structure provided as an example of the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the use of ordinal numbers (e.g., "first and second," "first to fourth," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0045] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] In one example, such as Figure 1 As shown, a multimodal medical image fusion method includes the following steps:

[0047] S1: Construct a multi-modal fusion (Diff-MF) framework based on diffusion models, including a sequentially connected feature fusion model and an explicit classifier-guided diffusion model.

[0048] like Figure 2 As shown, the feature fusion model includes two concat layers, a cross-attention fusion block, and a cross-self-attention module (i.e., an auxiliary network). The first concat layer (Concat1), the cross-attention fusion block, and the second concat layer (Concat2) are connected sequentially. The cross-self-attention module is connected to the second concat layer. Different modalities of medical images are input to the first concat layer, the cross-attention fusion block, and the cross-self-attention module, respectively. The different modalities of medical images can be any two or more of CT images, MRI images, PET images, etc., or a combination of two images (such as CT-MRI images and PET-MRI images). In this example, I... i I j These are CT images and MRI images, respectively.

[0049] Furthermore, the diffusion model guided by the explicit classifier includes an inverse network, a classifier, and a noise prediction network U-Net. The inverse network can be an autoencoder, and it is connected to the noise prediction network, which in turn is connected to the noise prediction network via the classifier. Further, the second stitching layer, the first convolutional layer (Conv1), the inverse network, the third stitching layer (Concat3), and the second convolutional layer (Conv2) are sequentially connected, with the first and third stitching layers connected via global residual connections. The inverse network and the noise prediction network form a complete diffusion model, which consists of a forward process and a reverse process. The forward process gradually adds noise to the original data (feature vectors of the original medical image), forming noisy data; the reverse process gradually removes noise to restore the true data. This invention trains a noise prediction network to reduce the error between the actual added noise and the true noise in the forward process during the restoration process. The classifier guides the data generation process during the reverse data generation, preventing the generated data from deviating from the target fused image data effect.

[0050] S2: Training the diffusion model guided by the explicit classifier includes: using an inverse network to progressively add noise to the feature vector of the original medical image at each time step to generate noisy data; a noise prediction network to remove noise and recover the data by predicting the noise at each time step; training the noise prediction network by reducing the difference between the predicted noise and the actual added noise; and using the classification results of the classifier on the noisy data (the conditional probability distribution of the target fused image data output by the classifier under the given noisy data) to guide the data recovery process of the noise prediction network.

[0051] Specifically, the dataset is first divided into groups, namely CT-MRI images and PET-MRI images, and a certain number of images are randomly selected from each group as the test set, with the remaining images used for training.

[0052] Furthermore, a diffusion model is trained using the original image data. Given the original image data x0, noise is progressively added to the feature vector at each time step t to generate noisy data x. T x T According to Derived directly from x0, where α t The noise figure represents the time step t; The cumulative noise figure can be expressed as: α s The noise figure represents time step s; N represents a Gaussian distribution; α t =1-β t ,β t β represents the noise intensity used to control each time step t, and it varies with time step t. t As noise gradually increases, its proportion increases during the forward diffusion process, with more and more noise data being added.

[0053] In the reverse process of the diffusion model, at each time step t, noise is removed through a noise prediction network to gradually recover the feature data. The diffusion model can then... x T Convert to original data x0, q(x1,…,x T ) represents the joint probability distribution of the data throughout the diffusion process; q(x) T q(x) represents the probability distribution of the data at the final time step T; t-1 |x t () represents the data x given the current time step t. t Predict the data x at the previous time step t-1. t-1 The conditional probability distribution of the noise prediction network parameterization p. θ (x t-1 |xt ) is: p θ (x t-1 |x t )=N(x t-1 μ θ (x t ,t),∑ θ (x t ,t)); where p θ (x t-1 |x t ) is a neural network learned by a diffusion model to approximate the true distribution q(x). t-1 |x t );∑ θ (x t Let ,t) represent the covariance matrix of the predictions made by the denoising network, and let represent the uncertainty of the prediction. ∑ θ (x t ,t) can be directly parameterized as β t or μ θ This represents the mean.

[0054] S3: The cross-attention fusion module and the cross-self-attention module are used to extract fusion feature information from different modalities of medical images, and then the images are stitched together to obtain preliminary fusion features. Specifically, the first preliminary fusion feature output by the cross-attention fusion module and the second preliminary fusion feature output by the cross-self-attention module are stitched together (Concat) through the second stitching layer to form a richer and more comprehensive feature expression (preliminary fusion features), thereby improving the quality of the fused image.

[0055] S4: Input the initial fused features into the explicit classifier-guided diffusion model that has been trained, use the noise prediction network to remove noise, generate a multimodal fused image, distinguish and adaptively fuse the effective low-frequency and high-frequency information of each mode, thereby increasing feature representation and improving image quality.

[0056] In one example, to distinguish and adaptively fuse effective low-frequency and high-frequency information from different modes, this invention designs a cross-attention fusion module. The input is a set of image data, which is then subjected to a series of operations such as convolution, channel concatenation, and dot product to achieve adaptive fusion of multimodal features. The outputs are then concatenated to form a richer and more comprehensive feature representation, thereby improving the quality of the fused image. Given that PET images are in RGB color format while MRI images are in grayscale format, this invention first converts the PET image to the YCbCr color space for processing. Since the Y channel carries structural and brightness information, while the Cb and Cr channels contain color information, only the Y channel is used for training during the fusion process. After the image fusion is generated, it is recombinated with the Cb and Cr channels to output the final RGB format image. Specifically, the structure of the cross-attention fusion module is as follows: Figure 3 As shown, the cross-attention fusion module is used to extract feature information from different modalities of medical images, including:

[0057] Medical images of different modalities I i I i Medical images after stitching i,j Feature extraction is performed to obtain first and second feature information. Channel concatenation is then applied to both the first and second feature information, followed by feature fusion processing via a cross-attention module to obtain the first feature fusion information A. i A j ; fuse the first feature information A i A j respectively with the first feature information (I) i Feature information, I i The first feature map is generated by point-wise multiplication of the first feature map (the second feature map). After feature extraction, the weighted feature map is concatenated with the second feature map to obtain the first preliminary fused feature. The cross-attention module consists of two sequentially connected convolutional layers and a sigmoid activation function layer. Each convolutional layer uses a 3×3 kernel. The output of the first convolutional layer is connected to an activation function layer (ReLU layer) to introduce non-linearity. The subsequent sigmoid activation function layer ensures that the output attention map is normalized, with its values ​​limited to the range [0,1].

[0058] In one example, to facilitate feature aggregation and capture long-term dependencies, a cross-self-attention module (auxiliary network) is introduced to evaluate the relationships between source images. Its model structure is as follows: Figure 4As shown, a cross-focused self-attention module is used to extract feature information from medical images of different modalities, including:

[0059] Three different weight matrices were used to evaluate the medical image I. i Medical Images I i A linear transformation is performed to obtain a first query q, a first key k, and a first value v, as well as a second query q, a second key k, and a second value v. The first query q and the first key k are then subjected to point-wise multiplication and softmax processing to obtain the first attention weight. The second query q and the second key k are then subjected to point-wise multiplication and softmax processing to obtain the second attention weight. The first attention weight and the second value v are then subjected to point-wise multiplication and element-wise addition with the second modality medical image to obtain the first preliminary attention feature. The second attention weight and the first value v are then subjected to point-wise multiplication and element-wise addition with the first modality medical image to obtain the second preliminary attention feature. The first and second preliminary attention features are then channel-concatenated to obtain the second preliminary fusion feature. This multi-head self-attention mechanism enables the adaptive fusion of multi-modal features from a set of image data. For example, in MRI images, low-frequency information includes the overall shape of the brain and boundary data of major regions; in CT images, high-frequency information includes the edges of bones and small lesion areas. The multi-head self-attention mechanism can adaptively fuse different feature information.

[0060] In the cross-self-attention module, the present invention initially uses a 1×1 convolution kernel from the connected input image I i and I i Generate 32-channel feature maps. The cross-self-attention mechanism is defined as follows: Where y i,j f represents the output value of the cross-self-attention mechanism; i f j Let f represent the infrared and visible light portions of CT and MRI images, respectively; function f is used to calculate the adaptive pixel-level weight vector of the source image; function g derives the feature representation from each image; the normalization factor C(x) is calculated as: C(x) = ∑f(f i ,f j This ensures that the cross-focus module effectively integrates the features of the two modes, enhancing the ability to interpret complex visual information.

[0061] In one example, an explicit classifier-guided diffusion model trains a noise prediction network to predict the noise ∈ at each time step t. θ (x tThe loss function is the mean squared error (MSE). The loss function measures the difference between the predicted noise and the actual added noise. The expression is:

[0062]

[0063] in, The expected operation represents averaging all possible combinations of the original data x0 without noise and the actual noise ∈, minimizing the prediction error, and thus training the noise prediction network; ∈0 represents the predicted noise. This represents the cumulative noise figure.

[0064] In one example, the explicit classifier-guided diffusion model trains the classifier on noisy data and uses gradients. Guide the diffusion sampling process (the reverse process of the diffusion model), assuming x t-1 It is a parameterized Gaussian distribution, and the noise prediction network, guided by the classifier, uses the corrected distribution representation x. t-1 :

[0065]

[0066] Where s represents the scaling factor that adjusts the influence of the classifier gradient on the diffusion trajectory; P represents the gradient operator; φ (y|x t ) represents the situation where, given noisy data x t Under the given conditions, the conditional probability distribution of the target fused image data y output by the classifier; P φ This represents the probabilistic model of the classifier parameters φ. In practice, this invention shifts the optimization objective from maximizing P... φ (y|x t The approach shifts to minimizing the difference between the two variables, specifically using the weighted mean squared error (MSE) metric as the method for minimizing the difference:

[0067]

[0068] Where h(t) represents the weight function at time step t; x t This represents the data generated by the diffusion model at each time step t. As the timestamp t decreases and backsampling continues, the resulting fused image should become increasingly closer to the real image. To ensure consistency between the generated samples and the ground truth source and target, gradient guidance should be strengthened accordingly. This application assumes that the weighted mean square error criterion defined exhibits low curvature with changing t, especially when the number of diffusion steps is large. Under these conditions, x... t-1 Replace x tThis simplifies diffusion-type operations while maintaining their integrity and effectiveness. Therefore, it can be approximated as:

[0069]

[0070] h(t) = N0e -λt

[0071] In the formula, h(·) is determined by the exponential decay function; λ is a hyperparameter representing the exponential decay constant; N0 represents an additional hyperparameter that controls the adjustment intensity. Differentiating the above formula, at timet t, x t The gradient becomes 2h(t-1)(x t-1 -y), substitute the gradient into the equation Setting the product of s∑ as a constant (1 / 2), we get: x t-1 ~N(μ-h(t-1)(x) t-1 -y),∑). For image fusion, this application first starts from N(μ) θ (·),∑) generate candidate x t Then, this application calculates x. t The difference between y and y, and further calculate the gradient 2h(t)(x) t -y), finally, this application will x t Updated to x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y. Therefore, the explicit classifier-guided diffusion model of this application is based on x t :=x t -h(t)(x t -y)=(1-h(t))x t +h(t)y is updated, and the training data is used as the basis. Target.

[0072] In one example, the method also includes jointly training the feature fusion model and the diffusion model guided by the explicit classifier:

[0073] The feature fusion model and the explicit classifier-guided diffusion model are trained using a joint loss function formed by mean squared error loss and structural similarity index loss.

[0074] Specifically, to stabilize the convergence of Diff-MF training, this invention uses a joint loss function L FIJAs a training objective, this joint loss function integrates mean squared error loss and structural similarity exponential loss. It leverages the advantages of MSE loss to achieve pixel-level reconstruction accuracy, while combining SSIM loss to maintain structural integrity and improve image visual quality. (MSE loss L) MSE SSIM loss L SSIM Joint loss function L FIJ The calculation expressions are as follows:

[0075]

[0076] L FIJ =λ1L MSE +λ2L SSIM

[0077] Where, μ x μ y Let x and y represent the mean values, respectively; σ yx σ represents the covariance of x and y; y 2 σ x 2 Let x and y represent the variances respectively; c1 and c2 are two constants; λ1, λ2 ∈ [0,1] represent the weights of the two types of losses respectively. Using these two loss components, L FIJ It effectively optimizes pixel precision and visual quality, thereby balancing different visual and quality indicators to achieve better image enhancement results.

[0078] During the training phase, this application sets the diffusion time step T to 1000 and uses linear β scheduling with initial and final values ​​of 10. -6 and 10 -2 The total number of training iterations is 14,000, with validation and checkpoint saving performed every 2,000 iterations. A learning rate of 10 is used. -4 The Adam optimizer trains the model until convergence is achieved.

[0079] Combining the above examples yields a preferred embodiment of the present invention, in which the image fusion method includes:

[0080] S10: Construct a feature fusion model and an explicit classifier-guided diffusion model;

[0081] S20: Train the diffusion model guided by the explicit classifier, and jointly train the feature fusion model and the diffusion model guided by the explicit classifier. The training of the diffusion model guided by the explicit classifier includes: using an inverse network to progressively add noise to the feature vector of the original medical image at each time step to generate noisy data; the noise prediction network removes noise and recovers the data by predicting the noise at each time step; the noise prediction network is trained by reducing the difference between the predicted noise and the actual added noise; and the classification results of the classifier on the noisy data guide the data recovery process of the noise prediction network.

[0082] S30: The cross-attention fusion module and the cross-self-attention module are used to extract the fusion feature information of medical images of different modalities, and then the images are stitched together to obtain the preliminary fusion features;

[0083] S40: Input the initial fused features into the trained explicit classifier-guided diffusion model, use a noise prediction network to remove noise, and generate a clear multimodal fused image.

[0084] This invention proposes a multimodal medical image fusion method based on a diffusion model and cross-attention fusion. It utilizes a cross-attention module to adaptively fuse image data based on its features, and employs a diffusion model guided by an explicit classifier to generate a clear fused image from the noisy image.

[0085] Based on the inventive concept of the image fusion method proposed in one or more examples above, the present invention proposes a disease prediction method, which includes the following steps:

[0086] The disease prediction model extracts feature vectors from the multimodal fusion image and performs disease prediction based on these feature vectors, outputting the prediction result. Preferably, the disease prediction model includes the aforementioned feature fusion model, an explicit classifier-guided diffusion model, and a classifier connected to the output of the explicit classifier-guided diffusion model. After jointly training the feature fusion model, the explicit classifier-guided diffusion model, and the classifier, the prediction method described above can be executed. This allows for the prediction of disease progression in patients based on the multimodal fusion image, providing a reference for disease diagnosis.

[0087] An example of the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal medical image fusion method or disease prediction method formed by any or a combination of the above examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0088] An example of the present invention also provides a storage medium having the same inventive concept as a multimodal medical image fusion method or disease prediction method formed by any or more of the above examples, wherein computer instructions are stored thereon, which, when executed, perform the steps of the multimodal medical image fusion method or disease prediction method formed by any or more of the above examples.

[0089] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] An example of this invention also provides a terminal that shares the same inventive concept as any or a combination of examples corresponding to the aforementioned multimodal medical image fusion method or disease prediction method. The terminal includes a memory and a processor. The memory stores computer instructions executable on the processor. When the processor executes the computer instructions, it performs the steps of the aforementioned multimodal medical image fusion method or disease prediction method. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement this invention.

[0091] In one example, the terminal, i.e., the electronic device, is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0092] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit can perform the aforementioned multimodal medical image fusion method.

[0093] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).

[0094] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0095] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0096] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method of the exemplary embodiment of this application.

[0098] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A multimodal medical image fusion method, characterized in that, Includes the following steps: Construct a feature fusion model and an explicit classifier-guided diffusion model. The feature fusion model includes a cross-attention fusion module and a cross-self-attention module. The explicit classifier-guided diffusion model includes an inverse network, a classifier, and a noise prediction network. The inverse network is connected to the noise prediction network, and the inverse network is connected to the noise prediction network via the classifier. Training the diffusion model guided by the explicit classifier includes: using an inverse network to progressively add noise to the feature vector of the original medical image at each time step to generate noisy data; a noise prediction network to remove noise and recover the data by predicting the noise at each time step; training the noise prediction network by reducing the difference between the predicted noise and the actual added noise; and using the classification results of the classifier on the noisy data to guide the data recovery process of the noise prediction network. The cross-attention fusion module and the cross-self-attention module are used to extract the fusion feature information of medical images of different modalities, and then the images are stitched together to obtain the preliminary fusion features. The initial fused features are input into the diffusion model guided by the trained explicit classifier, and noise is removed using a noise prediction network to generate a multimodal fused image. The cross-attention fusion module is used to extract feature information from medical images of different modalities, including: Feature extraction is performed on medical images of different modalities and medical images after stitching to obtain first feature information and second feature information. After channel connection processing of the first feature information and second feature information, feature fusion processing is performed through a cross-attention module to obtain first feature fusion information. The first feature fusion information and the first feature information are multiplied by the first feature information to generate a weighted feature map. After feature extraction of the weighted feature map, it is stitched with the second feature information to obtain the first preliminary fusion feature. The cross-self-focus module is used to extract feature information from different modalities of medical images, including: Linear transformations are performed on the first and second modal medical images using different weight matrices to obtain a first query q, a first key k, and a first value v, as well as a second query q, a second key k, and a second value v. The first query q and the first key k are then subjected to a dot product and softmax operation to obtain the first attention weight. The second query q and the second key k are then subjected to a dot product and softmax operation to obtain the second attention weight. The first attention weight and the second value v are then subjected to a dot product operation and element-adding operation with the second modal medical image to obtain the first preliminary attention feature. The second attention weight and the first value v are then subjected to a dot product operation and element-adding operation with the first modal medical image to obtain the second preliminary attention feature. The first preliminary attention feature and the second preliminary attention feature are then concatenated to obtain the second preliminary fusion feature.

2. The multimodal medical image fusion method according to claim 1, characterized in that, The loss function measures the difference between the predicted noise and the actual added noise. The expression is: ； in, Represents the desired operation, indicating the operation on all possible raw data without added noise. And actual added noise The combination is averaged; Indicates predicted noise; This represents the cumulative noise figure.

3. The multimodal medical image fusion method according to claim 1, characterized in that, The classification results of noisy data by the classifier guide the data recovery process of the noisy prediction network, including: Define the noisy data of the previous time step The noise prediction network, guided by a classifier, uses a modified distribution based on a parameterized Gaussian distribution. : ； in, Indicates a Gaussian distribution; This represents the mean; This represents the scaling factor that adjusts the influence of the classifier gradient on the diffusion trajectory. Represents the gradient operator; Indicates that given noisy data Under the given conditions, the target fused image data output by the classifier The conditional probability distribution; Represents classifier parameters The probability model.

4. The multimodal medical image fusion method according to claim 3, characterized in that, The data update method for the data recovery process of an explicit classifier-guided diffusion model is as follows: ； in, Indicates time step The weighting function.

5. The multimodal medical image fusion method according to claim 1, characterized in that, The method also includes jointly training the feature fusion model and the explicit classifier-guided diffusion model: The feature fusion model and the explicit classifier-guided diffusion model are trained using a joint loss function formed by mean squared error loss and structural similarity index loss.

6. A disease prediction method, characterized in that, A multimodal fused image is obtained by employing the image fusion method according to any one of claims 1-5, wherein the prediction method includes the following steps: The disease prediction model extracts feature vectors from the multimodal fusion image, performs disease prediction based on the feature vectors, and outputs the disease prediction results.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal medical image fusion method according to any one of claims 1-5, or when the computer program is executed by a processor, it implements the steps of the disease prediction method according to claim 6.

8. A terminal comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, characterized in that, When the processor executes the computer instructions, it performs the steps of the multimodal medical image fusion method according to any one of claims 1-5, or when the processor executes the computer instructions, it performs the steps of the disease prediction method according to claim 6.

Images