Multi-modal medical image processing method and device, storage medium and computer equipment

Abstract

The invention discloses a multi-modal medical image processing method and device, a storage medium and computer equipment. Comprising the following steps: performing three-dimensional discrete wavelet transform on a brain MRI image of a target patient to generate an MRI wavelet coefficient; inputting the MRI wavelet coefficient into a diffusion model to obtain a reference PET wavelet coefficient of the brain of the target patient in a healthy state; performing inverse wavelet transform on the reference PET wavelet coefficient to generate a reference PET image; and comparing the brain PET image of the target patient with the reference PET image, and determining the metabolic deviation index of the brain of the target patient. Therefore, each patient can take the condition without the neurodegenerative change as a contrast, space standardization does not need to be carried out on a group template, anatomical structure distortion caused by the space standardization is greatly reduced, voxel-level accurate analysis of the neurodegenerative disease is realized, tiny pathological change aiming at the patient can be identified, and the accuracy of voxel-level accurate analysis of the neurodegenerative disease is improved. And clinical doctors are assisted in early diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and medical image processing technology, in particular to a multi-modal medical image processing method and device, a storage medium and a computer device. BACKGROUND

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease characterized by insidious onset and a long preclinical phase during which subtle brain changes accumulate. Neuroimaging biomarkers play a central role in the early detection and monitoring of AD, complementing clinical assessment and neuropsychological testing. In particular, 2-deoxy-2-[18F]fluoro-D-glucose positron emission tomography (FDG-PET) provides a map of brain glucose metabolism, a sensitive indicator of neuronal impairment and synaptic dysfunction in AD.

[0003] However, the routine clinical application of FDG-PET is limited by cost, radiation exposure, and accessibility. Moreover, traditional neuroPET analysis relies on comparing a patient's scan to a normative database, which requires standardizing the image space to a common template space before conducting differential analysis based on population data. This approach distorts the anatomy, such that normal anatomical differences between individuals can be masked or misjudged after registration, especially for cases with significant brain atrophy or small brain regions. SUMMARY

[0004] Therefore, the present application provides a multi-modal medical image processing method and device, a storage medium and a computer device, which can consider individual variability and avoid over-smoothing or distortion to achieve accurate anatomical analysis.

[0005] According to one aspect of the present application, a multi-modal medical image processing method is provided, comprising: performing three-dimensional discrete wavelet transform on a brain MRI image of a target patient to generate MRI wavelet coefficients; inputting the MRI wavelet coefficients into a diffusion model to obtain baseline PET wavelet coefficients of the brain of the target patient in a healthy state, wherein the diffusion model is trained based on MRI image samples and PET image samples of the brain in a healthy state; performing inverse wavelet transform on the baseline PET wavelet coefficients to generate a baseline PET image; comparing a brain PET image of the target patient with the baseline PET image to determine a metabolic deviation index of the brain of the target patient.

[0006] Optionally, the multi-modal medical image processing method further comprises: performing three-dimensional discrete wavelet transform on the MRI image sample and the PET image sample respectively to generate an MRI wavelet coefficient sample and a PET wavelet coefficient sample; adding random noise to the PET wavelet coefficient sample based on a forward diffusion algorithm to generate a noisy PET wavelet coefficient; inputting the MRI wavelet coefficient sample and the noisy PET wavelet coefficient into a denoising diffusion probability model to obtain a predicted noise of the PET wavelet coefficient sample; minimizing mean square error of the random noise and the predicted noise as an objective, optimizing model parameters of the denoising diffusion probability model based on a loss function until a model iteration termination condition is met, and outputting the diffusion model.

[0007] Optionally, the denoising diffusion probability model includes an encoder of multiple frequency bands; and the inputting the MRI wavelet coefficient sample into the denoising diffusion probability model includes: determining MRI wavelet subbands of multiple frequency bands based on the MRI wavelet coefficient sample; splicing the MRI wavelet subbands and feature maps of the encoder under the same frequency band; inputting the spliced joint feature maps into the encoder corresponding to a preset observation scale.

[0008] Optionally, the comparing the brain PET image of the target patient and the reference PET image to determine a metabolic deviation index of the brain of the target patient includes: obtaining at least one brain region in the brain PET image; calculating a standardized metabolic value ratio of the brain PET image and the reference PET image in the brain region respectively; calculating a metabolic difference value of the brain region based on the standardized metabolic value ratio of the brain PET image and the standardized metabolic value ratio of the reference PET image; wherein, the metabolic deviation index includes the standardized metabolic value ratio and / or the metabolic difference value.

[0009] Optionally, the obtaining at least one brain region in the brain PET image includes: segmenting the brain PET image or the brain MRI image based on brain anatomical structure information to determine at least one brain region; or inputting the brain PET image or the brain MRI image into a segmentation model of a target observation scale to determine at least one brain region, wherein the segmentation model is trained based on an individualized segmentation label standardized to a standard brain template.

[0010] Optionally, the method for processing multi-modal medical images further comprises: obtaining a standard brain template and a medical image sample under different preset observation scales, and forming an image space of the standard brain template and the medical image sample, wherein the medical image sample comprises the MRI image sample or the PET image sample, and the standard brain template is marked with a partition label of a brain region; mapping the image space of the medical image sample to the image space of the standard brain atlas by using an affine transformation algorithm and a nonlinear registration algorithm, to form a reference space; performing inverse transformation on the brain region in the reference space to generate an individualized partition label; training a partition model corresponding to the preset observation scale by using the individualized label as a supervision signal.

[0011] Optionally, the method for processing multi-modal medical images further comprises: classifying the metabolic difference index based on a preset metabolic classification interval, to determine brain regions of different metabolic categories; rendering the brain regions in the brain PET image based on a first rendering parameter of the preset metabolic classification interval and / or a second rendering parameter of different brain regions, to generate a metabolic difference image; displaying the metabolic difference image and the metabolic deviation index.

[0012] According to another aspect of the present application, a processing device for multi-modal medical images is provided, comprising: a wavelet transform module configured to perform three-dimensional discrete wavelet transform on a brain MRI image of a target patient to generate MRI wavelet coefficients; a prediction module configured to input the MRI wavelet coefficients into a diffusion model to obtain reference PET wavelet coefficients of the brain of the target patient in a healthy state, wherein the diffusion model is trained based on MRI image samples and PET image samples of the brain in a healthy state; the wavelet transform module is further configured to perform inverse wavelet transform on the reference PET wavelet coefficients to generate a reference PET image; an analysis module configured to compare the brain PET image of the target patient with the reference PET image to determine a metabolic deviation index of the brain of the target patient.

[0013] Optionally, the wavelet transform module is further configured to perform three-dimensional discrete wavelet transform on the MRI image samples and the PET image samples respectively to generate MRI wavelet coefficient samples and PET wavelet coefficient samples; the processing device for multi-modal medical images further comprises: an interference module, configured to add random noise to the PET wavelet coefficient sample based on a forward diffusion algorithm to generate a noisy PET wavelet coefficient; a first training module, configured to input the MRI wavelet coefficient sample and the noisy PET wavelet coefficient into a denoising diffusion probability model to obtain a predicted noise of the PET wavelet coefficient sample; and based on a loss function, optimize model parameters of the denoising diffusion probability model with a minimum mean square error of the random noise and the predicted noise as an objective until a model iteration termination condition is met, and output the diffusion model.

[0014] Optionally, the denoising diffusion probability model comprises an encoder of a plurality of frequency bands; and the multi-modal medical image processing apparatus further comprises: a data preprocessing module, configured to determine MRI wavelet subbands of a plurality of frequency bands based on the MRI wavelet coefficient sample; and splice the MRI wavelet subbands and feature maps of the encoder in the same frequency band; The first training module is specifically configured to input the spliced joint feature map into the encoder corresponding to a preset observation scale.

[0015] Optionally, the multi-modal medical image processing apparatus further comprises: a segmentation module, configured to obtain at least one brain region in the brain PET image; an analysis module, specifically configured to calculate a standardized metabolic value ratio of the brain PET image and the reference PET image in the brain region respectively; and based on the standardized metabolic value ratio of the brain PET image and the standardized metabolic value ratio of the reference PET image, calculate a metabolic difference value of the brain region; wherein the metabolic deviation index comprises the standardized metabolic value ratio and / or the metabolic difference value.

[0016] Optionally, the segmentation module is specifically configured to segment the brain PET image or the brain MRI image based on brain anatomical structure information to determine at least one brain region; or input the brain PET image or the brain MRI image into a segmentation model of a target observation scale to determine at least one brain region, wherein the segmentation model is trained based on an individualized segmentation label standardized to a standard brain template.

[0017] Optionally, the multi-modal medical image processing apparatus further comprises: The mapping module is configured to obtain a standard brain template and a medical image sample under different preset observation scales, and form an image space of the standard brain template and the medical image sample, wherein the medical image sample comprises the MRI image sample or the PET image sample, and the standard brain template is marked with a partition label of a brain region; and the mapping module is further configured to map the image space of the medical image sample to an image space of the standard brain atlas by using an affine transformation algorithm and a nonlinear registration algorithm, to form a reference space; and the mapping module is further configured to perform inverse transformation on the brain region in the reference space, to generate an individualized partition label. The second training module is configured to train a partition model corresponding to the preset observation scale by using the individualized label as a supervision signal.

[0018] Optionally, the analysis module is further configured to classify the metabolic difference index based on a preset metabolic classification interval, to determine brain regions of different metabolic categories. The processing device of the multi-modal medical image further comprises: The rendering module is configured to render the brain region in the brain PET image based on a first rendering parameter of the preset metabolic classification interval and / or a second rendering parameter of different brain regions, to generate a metabolic difference image. The visualization module is configured to display the metabolic difference image and the metabolic deviation index.

[0019] According to another aspect of the present application, a readable storage medium is provided, which stores a program or instructions, and the program or instructions are executed by a processor to implement the steps of the processing method of the multi-modal medical image.

[0020] According to another aspect of the present application, a computer device is provided, which comprises a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, and the processor implements the steps of the processing method of the multi-modal medical image when executing the program.

[0021] According to the above technical solution, the MRI structure of a patient is mapped to the metabolic activity of a multi-resolution by using a 3D wave diffusion model (WaveDM), to simulate the real distribution of the PET intensity in a healthy state, and to generate a benchmark PET image that can represent the specificity of the patient. The benchmark PET image is used as a healthy reference PET of the patient, and by comparing with the actual brain PET image of the patient, metabolic abnormalities specific to the patient can be highlighted, instead of deviations from the population average. Thus, each patient can use his own condition without neurodegenerative changes as a control, without spatial standardization to the population template, greatly reducing the distortion of anatomical structure caused by spatial standardization, and achieving voxel-level precise analysis of neurodegenerative diseases, which is helpful to identify small and patient-specific pathological changes, and assists clinicians in early diagnosis.

[0022] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application and can be implemented according to the content of the specification, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and the description thereof are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a processing method of a multi-modal medical image provided by an embodiment of the present application is shown; Figure 2 A structural block diagram of a processing device of a multi-modal medical image provided by an embodiment of the present application is shown; Figure 3 A comparison result diagram of a brain PET image and the reference PET image provided by an embodiment of the present application is shown; Figure 4 A brain region segmentation diagram provided by an embodiment of the present application is shown; Figure 5 A structural diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] Hereinafter, the present application will be described in detail with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0026] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprising," "including," "containing," and "having" and the like, when used in the specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. Furthermore, "connected" or "coupled" as used herein can include wirelessly connected or wirelessly coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] Exemplary embodiments according to this application will now be described in greater detail below with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in various different forms and should not be construed as being limited to the embodiments set forth herein. It is understood that the embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0028] A method for processing multi-modal medical images is provided in the present embodiment, as shown in Figure 1 The method includes: At step 101, a three-dimensional discrete wavelet transform is performed on a brain MRI image of a target patient to generate MRI wavelet coefficients.

[0029] It is to be noted that the multi-modal medical images in the embodiments of the present application include MRI (magnetic resonance imaging) images and PET (positron emission tomography) images. The MRI images are generated by exciting hydrogen nuclei in the human body using a strong magnetic field and radio frequency pulses, and receiving signals released during the relaxation process of the protons, and are used to distinguish high soft tissues. The PET images are generated by injecting a radiopharmaceutical, and detecting the positron-electron annihilation radiation (gamma photons) released by the decay of the radiopharmaceutical, and are used to reflect biochemical metabolic activities. The PET image radiopharmaceuticals can be FDG (deoxyglucose), FET (ethyl tyrosine), DOPA (fluorodopa), MET (methyl- methionine), etc., which are not limited in the embodiments of the present application.

[0030] In this embodiment, the voxel data in the brain MRI image is decomposed into low-frequency approximation components (LLL) and high-frequency detail components (such as LHL, HLL, HHL, etc.) by three-dimensional discrete wavelet transform, so as to capture the spatial and frequency domain features of the brain MRI image under multi-observation scales, thereby realizing multi-resolution analysis, providing high-resolution anatomical structure information for subsequent synthesis of PET images, and improving the spatial positioning accuracy of metabolic activities.

[0031] It can be understood that before the three-dimensional discrete wavelet transform is performed, brain feature extraction can be performed on the brain MRI image to remove non-brain tissues in the image.

[0032] In step 102, the MRI wavelet coefficients are input into the diffusion model to obtain the baseline PET wavelet coefficients of the brain of the target patient in a healthy state.

[0033] The diffusion model is trained based on MRI image samples and PET image samples of the brain in a healthy state.

[0034] In this embodiment, through the diffusion model, in the wavelet domain, based on the MRI structural information (MRI wavelet coefficients) of the patient itself, the normal metabolism mode (baseline PET wavelet coefficients) when the patient has no disease is predicted. In order to quantify the degree of metabolic deviation by comparing the generated healthy baseline PET with the actual PET, accurately locate the position where the reality deviates from the expectation, and objectively evaluate the metabolic changes.

[0035] In an embodiment, the training process of the diffusion model specifically includes: performing three-dimensional discrete wavelet transform on the MRI image samples and the PET image samples respectively to generate MRI wavelet coefficient samples and PET wavelet coefficient samples; adding random noise to the PET wavelet coefficient samples based on a forward diffusion algorithm to generate noisy PET wavelet coefficients; inputting the MRI wavelet coefficient samples and the noisy PET wavelet coefficients into a denoising diffusion probability model to obtain predicted noise of the PET wavelet coefficient samples; optimizing model parameters of the denoising diffusion probability model based on a loss function with the objective of minimizing the mean square error of the random noise and the predicted noise, until a model iteration termination condition is met, and outputting the diffusion model.

[0036] The MRI image samples and the PET image samples correspond to each other, and the corresponding MRI image samples and PET image samples are derived from the same healthy individual.

[0037] In this embodiment, the MRI image sample has high spatial resolution and clear anatomical structure, and the PET image sample can reflect metabolic function. The frequency domain features of both are extracted by wavelet transform. The forward diffusion algorithm is used to gradually add noise with different noise intensities to the PET wavelet coefficient sample to generate multiple groups of PET wavelet coefficient samples with different noise levels, so as to accurately simulate the influence of noise on the high / low frequency components of the image. With the MRI wavelet coefficient sample as the condition, the denoising diffusion probabilistic model (DDPM) maps the noisy PET wavelet coefficient to the clean PET coefficient sample through backward diffusion, thereby gradually denoising the random noise vector into the PET wavelet coefficient matching the MRI anatomical structure, so that the model accurately separates the predicted PET wavelet coefficient and the predicted noise. The denoising diffusion probabilistic model is iteratively trained by minimizing the noise residual of the predicted noise and the true noise as the target, forcing the model to learn the potential association between anatomy and function, and finally forming a diffusion model that can predict the PET wavelet coefficient. In this way, the trained diffusion model can significantly improve the signal-to-noise ratio when predicting the PET wavelet coefficient from the MRI wavelet coefficient, avoid over-smoothing, and at the same time retain the lesion metabolic information in the PET.

[0038] Further, considering that the original image resolution of MRI (high-resolution anatomy) and PET (low-resolution function) does not match, direct fusion will cause information misplacement. Therefore, a multi-band encoder architecture is introduced in the denoising diffusion probabilistic model. Then the MRI wavelet coefficient sample is input into the denoising diffusion probabilistic model, which specifically includes: determining a plurality of frequency bands of MRI wavelet subbands based on the MRI wavelet coefficient sample; splicing the MRI wavelet subbands and the feature maps of the encoder in the same frequency band to form a joint feature map; performing dynamic affine transformation on the activation values of the intermediate layers of the encoder based on scaling and shifting parameters; and inputting the joint feature map into the encoder after affine transformation in the corresponding frequency band.

[0039] The scaling and shifting parameters are determined based on the MRI wavelet coefficient sample.

[0040] In this embodiment, the MRI wavelet coefficients correspond to different levels of anatomical information (e.g., local edges or global structures) at different observation scales. The wavelet subbands at the same scale are concatenated with the encoder features. The scaling and bias parameters of the corresponding encoder intermediate layers are determined by the wavelet coefficient samples at the same frequency band, and the FiLM (Feature-wise Linear Modulation) modulation operation is performed. Then the joint feature map is input into the affine transformed encoder at the corresponding frequency band. This allows the model to process image structures at multiple observation scales, ensuring that the MRI and PET features are aligned at the same spatial frequency. The model is forced to learn the association between MRI and PET at different frequencies, so as to establish the local correspondence between anatomy and metabolic function under healthy state, and to constrain the diffusion model to generate PET functional distribution matching the anatomical structure. In this way, the precise alignment of anatomical-metabolic function is achieved, which is conducive to generating a clearer and more anatomically accurate reference PET image.

[0041] For example, global intensity gradients and regional uptake levels (e.g., whole brain region uptake levels) are captured by low-frequency subbands (LL). Fine anatomical profiles (e.g., cortical gyrus patterns) are captured by high-frequency subbands (LH / HL / HH). The wavelet subbands at the same scale (e.g., low-frequency LL or high-frequency LH) are directly concatenated with the feature maps at the corresponding levels of the UNet architecture in the channel dimension. Without destroying the original features of the UNet, the feature distribution is fine-tuned by scaling parameters γ and bias parameters β to adapt to the structural constraints of MRI, thereby enhancing the model's ability to adapt to cross-modal conditions and facilitating cross-modal feature fusion.

[0042] Step 103: performing inverse wavelet transform on the reference PET wavelet coefficients to generate a reference PET image.

[0043] In this embodiment, in the wavelet domain, the diffusion model replaces the low-frequency information of PET with low-frequency information about anatomical structure in MRI, while retaining the high-frequency metabolic details of PET about individual metabolic uptake. Then, through inverse transform, a healthy state PET image matching the patient's brain anatomical structure (MRI) is reconstructed as a reference for difference mapping, which helps doctors identify abnormal metabolic areas.

[0044] Step 104: comparing the target patient's brain PET image with the reference PET image to determine the metabolic deviation index of the target patient's brain.

[0045] The metabolic deviation index includes a standardized metabolic value ratio and / or a metabolic difference value.

[0046] The method for processing multi-modal medical images provided by the embodiments of the present application maps the MRI structure of a patient to the multi-resolution of metabolic activity through a 3D wave diffusion model (WaveDM), to simulate the real distribution of PET intensity in a healthy state, and to generate a benchmark PET image that can represent the specificity of the patient. Taking the benchmark PET image as the healthy reference PET of the patient, by comparing with the actual brain PET image of the patient, metabolic abnormalities specific to the patient can be highlighted, instead of deviation from the population average. Thus, each patient can take his / her condition without neurodegenerative changes as a control, without spatial standardization to the population template, greatly reducing the distortion of anatomical structure caused by spatial standardization, avoiding the masking of individual differences by the population average standard, realizing voxel-level precise analysis of brain metabolism, and helping to identify small, patient-specific pathological changes, assisting clinicians in early diagnosis.

[0047] For example, a set of cognitively normal (healthy) PET / MR pairs are used to train the diffusion model. Each MRI is first decomposed into wavelet coefficients. These coefficients are used as conditional input to the diffusion network. The corresponding FDG-PET images from the healthy cohort are also decomposed into wavelet coefficients. The diffusion model is trained to progressively denoise a random noise vector into PET wavelet coefficients that match the MRI anatomical structure. During training, the forward diffusion process adds noise to the real PET coefficients, and the model learns the reverse diffusion process to iteratively refine the structure from the noise. The model integrates multiple resolution levels to ensure that both coarse-grained metabolic patterns (low-frequency content) and fine details (edges, textures in high-frequency subbands) are fully synthesized. Given a target patient's MRI image and its wavelet coefficients, the MRI wavelet coefficients are input into the trained diffusion model. Starting from pure noise, the model generates synthetic FDG-PET wavelet coefficients that are consistent with the target patient's anatomical structure but represent a healthy metabolic state. Inverse wavelet transform reconstructs the FDG-PET wavelet coefficients into a full-resolution benchmark PET image. Comparing the patient's actual brain PET with this synthetic benchmark PET image generates a voxel-level difference map.

[0048] Figure 3 Benchmark PET images and metabolic difference images are shown for a normal control (Normal), an Alzheimer's disease patient (AD), and a mild cognitive impairment patient (MCI). Figure 3From left to right, the images show: input MRI image (T2 weighted), original brain FDG-PET image, reference PET image (i.e. personalized healthy template) synthesized by diffusion model (WaveDM), and difference map (brain FDG-PET image minus reference PET image). For the normal subject (first row), the original PET and the synthesized PET look very similar; correspondingly, the difference map shows minimal deviations, which indicates that the metabolism of the normal subject is close to the normal state predicted by WaveDM according to its anatomy. In the AD patient (second row), the synthesized PET shows high uptake in the posterior cingulate and parietal regions expected for a healthy individual, while the actual PET shows significantly reduced uptake in the same regions. Therefore, the difference map for AD is dominated by blue regions in the bilateral parietal and posterior cingulate cortex, indicating significant hypometabolism. Slight hypermetabolism (red) is observed in a few regions. The synthesized PET for the MCI patient (third row) again presents a full and "healthy" appearance in the cortex, while the actual PET has slight reductions in the posterior cingulate and temporoparietal regions. The difference map identifies these changes as light blue regions in the posterior cingulate and precuneus, indicating moderate hypometabolism. This corresponds to the early AD metabolic pattern, consistent with the MCI diagnosis.

[0049] In an embodiment, the step 104, i.e. comparing the brain PET image of the target patient and the reference PET image, determines the metabolic deviation index of the brain of the target patient, specifically comprising the following steps: Step 104-1, obtaining at least one brain region in the brain PET image.

[0050] It can be understood that, since the MRI and PET have been registered, the segmentation result can be determined by the MRI first, and then the segmentation result is directly superimposed on the PET, without separately segmenting the PET; vice versa.

[0051] In a specific application scenario, the step 104-1 can be implemented in the following manner: Manner one: segmenting the brain PET image or the brain MRI image based on the brain anatomical structure information to determine at least one brain region.

[0052] The brain anatomical structure information can be obtained by a pre-defined anatomical atlas, tissue characteristics or expert knowledge.

[0053] In this embodiment, the segmentation of the brain anatomical structure is quickly completed based on the known anatomical rules. The labeling cost is saved, and the segmentation efficiency is improved.

[0054] Manner two: inputting the brain PET image or the brain MRI image into a segmentation model of a target observation scale to determine at least one brain region.

[0055] The segmentation model is trained based on individualized segmentation labels that are spatially standardized to a standard brain template. The standard brain template can be the MNI ICBM 152 atlas.

[0056] In this embodiment, individual brain images are standardized to a standard brain template space. The brain regions are automatically predicted using the trained segmentation model. Thus, brain structures of different individuals can be mapped to the same coordinate space, eliminating anatomical individual differences, making the segmentation results comparable across individuals, and reducing the limitations of a single atlas.

[0057] Specifically, training the segmentation model specifically includes: obtaining a standard brain template and medical image samples under different preset observation scales, and forming an image space of the standard brain template and the medical image samples; using an affine transformation algorithm and a nonlinear registration algorithm, mapping the image space of the medical image samples to the image space of the standard brain atlas to form a reference space; performing inverse transformation on the brain region in the reference space to generate individualized segmentation labels; and training the segmentation model corresponding to the preset observation scale using the individualized labels as a supervision signal.

[0058] The medical image samples include MRI image samples or PET image samples, and the standard brain template is marked with partition labels of brain regions. It can be understood that data augmentation can be performed by random flipping, rotation and intensity scaling to improve the generalization ability to the external test set.

[0059] It is worth mentioning that the T1-weighted image (T1WI) and the T2-weighted image (T2WI) of the MRI are obtained respectively. The T1-weighted image and the T2-weighted image highlight the T1 relaxation difference and the T2 relaxation difference respectively to form different observation scales. T1 and T2 are physical parameters in magnetic resonance imaging that describe the transverse and longitudinal relaxation characteristics of hydrogen nuclei (protons) in the human body in a magnetic field. The T1-weighted image has clear anatomical structure and can be used to distinguish gray matter, white matter and cerebrospinal fluid, and is suitable for displaying the outline of brain tissue. The T2-weighted image has high lesion sensitivity and can be used to distinguish tissues with high water content such as edema, inflammation and tumor.

[0060] In this embodiment, affine transformation and nonlinear registration are used to map the patient's medical image samples to a standard brain template space. Standard brain region segmentation labels are combined with the patient's simulated brain structure, making the reference space more closely resemble the patient's own brain structure. Inverse transformation is performed on the brain regions in the reference space to extract individualized segmentation labels that closely resemble the patient's own brain structure. Training the segmentation model using these individualized segmentation labels allows the model to better fit the individual's brain structure, fully considering individual characteristics during segmentation, eliminating differences in size, shape, and location between different individuals' brains, and improving the accuracy of segmenting complex structures and lesion regions. Furthermore, training the model for different preset observation scales can meet the analytical needs from macroscopic brain regions to microscopic structures, allowing the model to perform well at multiple scales, applicable to different research and clinical scenarios, and improving the model's generalization ability.

[0061] For example, such as Figure 4 The image shows the brain region segmentation results for example subjects on an external test dataset. The example subjects included one normal control, one Alzheimer's disease (AD) patient, and one patient with mild cognitive impairment (MCI). Using T2-weighted images as an example, for each case, Figure 4 The image shows T2-weighted MRI, FDG-PET, nnU-Net segmentation results superimposed on axial slices, and 3D renderings of the segmentation. Color coding corresponds to major brain structures (e.g., different lobes of the cortex are represented by different colors, and subcortical nuclei are represented by other colors). Figure 4 As shown, even in AD cases with enlarged ventricles and some cortical atrophy, the model accurately depicts the brain region-ventricular system (purple in axial sections, light blue in 3D) as enlarged, while the cortical bands (various colors) remain continuous despite atrophy. The segmentation results in normal and MCI brains are anatomically very close to expectations, with smooth boundaries and precise locations. Therefore, the segmentation model output in this embodiment provides an accurate mask of the actual MRI boundaries, demonstrating superior performance.

[0062] Furthermore, for complex brain structures, segmentation can be manually corrected to help the network learn to adjust for such biases.

[0063] Step 104-2: Calculate the normalized metabolic ratio of brain PET images and baseline PET images within the brain region.

[0064] Step 104-3: Calculate the metabolic difference value of brain regions based on the normalized metabolic value ratio of brain PET images and the normalized metabolic value ratio of baseline PET images.

[0065] In this embodiment, the brain region segmentation network running on the brain PET image divides at least one brain region. For each brain region, the average standardized uptake value ratio (SUVr) in the actual brain PET image, the average SUVr in the synthesized reference PET image, and the difference between the two can be calculated. The difference is divided by the average SUVr of the reference PET image to give the region-specific, atrophy-corrected metabolic difference value. Thus, voxel-level difference comparison can be achieved, highlighting the individual-specific metabolic abnormalities. By averaging the standardized uptake value ratio and the metabolic difference value, and combining the target patient-specific anatomical segmentation region, the system can output a detailed map of metabolic abnormalities in the patient's own anatomical space, facilitating precise positioning and quantification of metabolic defects. Further, sensitive and interpretable neurodegenerative change detection can be achieved without population-based standardization, assisting clinicians in early diagnosis and accurate tracking of neurodegenerative disease progression.

[0066] The specific calculation formula is as follows: Standardized uptake value ratio SUVr = average SUV of target brain region / average SUV of reference region; SUV = pixel radioactivity concentration / (injection dose / body weight); Metabolic difference value = (SUVr 实际 -SUVr 合成 ) / SUVr 合成 × 100%.

[0067] In an embodiment, the processing method of multi-modal medical images further comprises: classifying the metabolic difference index based on a preset metabolic classification interval to determine brain regions of different metabolic categories; rendering the brain regions in the brain PET image based on the first rendering parameter of the preset metabolic classification interval and / or the second rendering parameter of different brain regions to generate a metabolic difference image; and displaying the metabolic difference image and the metabolic deviation index.

[0068] The preset metabolic classification interval can be reasonably set according to the reference values of different metabolic categories (such as high metabolism, low metabolism, and normal metabolism) of the disease.

[0069] In this embodiment, by classifying the metabolic deviation indicators by preset metabolic classification intervals, abstract metabolic data can be converted into explicit regional division, helping doctors quickly lock the abnormal brain regions of glucose metabolism, blood oxygen level, etc. In combination with the first rendering parameter (such as the color mapping rule) of the classification interval and the second rendering parameter (such as the transparency and brightness) specific to the brain region, image rendering is performed for different brain regions or brain regions of different metabolic categories, forming intuitive metabolic difference images to strengthen the visual presentation of brain region boundaries and metabolic differences, facilitating doctors to monitor the metabolic changes of the brain through metabolic deviation.

[0070] It is worth mentioning that the processing method of the multi-modal medical image provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server can be configured as a standalone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application that implements the processing method of the multi-modal medical image, but is not limited to the above forms.

[0071] It should be noted that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0072] Further, as shown in Figure 2 As a specific implementation of the processing method of the multi-modal medical image, the embodiments of the present application provide a processing device 200 of multi-modal medical image, which comprises a wavelet transform module 201, a prediction module 202 and an analysis module 203.

[0073] The wavelet transform module 201 is configured to perform three-dimensional discrete wavelet transform on the brain MRI image of the target patient to generate MRI wavelet coefficients; The prediction module 202 is configured to input the MRI wavelet coefficients into a diffusion model to obtain reference PET wavelet coefficients of the brain of the target patient in a healthy state, wherein the diffusion model is trained based on MRI image samples and PET image samples in the healthy state of the brain; The wavelet transform module 201 is further configured to perform inverse wavelet transform on the reference PET wavelet coefficients to generate reference PET images; The analysis module 203 is configured to compare the brain PET image of the target patient with the reference PET image, and determine a metabolic deviation index of the brain of the target patient.

[0074] In this embodiment, the MRI structure of the patient is mapped to the multi-resolution of metabolic activity by a 3D wavelet diffusion model (WaveDM) to simulate the real distribution of PET intensity in a healthy state and generate a reference PET image capable of representing the patient specificity. By taking the reference PET image as the healthy reference PET of the patient, by comparing with the actual brain PET image of the patient, metabolic abnormalities specific to the patient can be highlighted instead of deviation from the population average. Thus, each patient can take his own condition without neurodegenerative changes as a reference, without spatial standardization to the population template, greatly reducing the distortion of anatomical structure caused by spatial standardization, realizing voxel-level precise analysis of neurodegenerative diseases, and helping to identify small, patient-specific pathological changes to assist clinicians in early diagnosis.

[0075] Further, the wavelet transform module 201 is further configured to perform three-dimensional discrete wavelet transform on the MRI image sample and the PET image sample respectively to generate an MRI wavelet coefficient sample and a PET wavelet coefficient sample; The multi-modal medical image processing apparatus 200 further comprises: An interference module (not shown in the figure) is configured to add random noise to the PET wavelet coefficient sample based on a forward diffusion algorithm to generate a noisy PET wavelet coefficient; A first training module (not shown in the figure) is configured to input the MRI wavelet coefficient sample and the noisy PET wavelet coefficient into a denoising diffusion probability model to obtain a predicted noise of the PET wavelet coefficient sample; and based on a loss function, optimize model parameters of the denoising diffusion probability model with the objective of minimizing the mean square error of the random noise and the predicted noise until a model iteration termination condition is met, and output the diffusion model.

[0076] Further, the denoising diffusion probability model comprises an encoder of a plurality of frequency bands; and the multi-modal medical image processing apparatus 200 further comprises: A data preprocessing module (not shown in the figure) is configured to determine MRI wavelet subbands of the plurality of frequency bands based on the MRI wavelet coefficient sample; and splice the MRI wavelet subbands and the feature map of the encoder in the same frequency band; The first training module is specifically configured to input the spliced joint feature map into the encoder corresponding to the preset observation scale.

[0077] Further, the multi-modal medical image processing apparatus 200 further comprises: A segmentation module (not shown in the figure) is configured to obtain at least one brain region in the brain PET image. The analysis module (not shown in the figure) is specifically configured to calculate the standardized metabolic value ratio of the brain PET image and the reference PET image in the brain region respectively, and calculate the metabolic difference value of the brain region based on the standardized metabolic value ratio of the brain PET image and the standardized metabolic value ratio of the reference PET image; wherein the metabolic deviation index includes the standardized metabolic value ratio and / or the metabolic difference value.

[0078] Further, the segmentation module is specifically configured to segment the brain PET image or the brain MRI image based on the brain anatomical structure information to determine at least one brain region; or input the brain PET image or the brain MRI image into a segmentation model of a target observation scale to determine at least one brain region, wherein the segmentation model is trained based on individualized segmentation labels standardized to a standard brain template in space.

[0079] Further, the multi-modal medical image processing device 200 further comprises: The mapping module (not shown in the figure) is configured to obtain the standard brain template and the medical image sample under different preset observation scales, and form an image space of the standard brain template and the medical image sample, wherein the medical image sample includes an MRI image sample or a PET image sample, and the standard brain template is marked with a partition label of the brain region; and the image space of the medical image sample is mapped to the image space of the standard brain atlas by using an affine transformation algorithm and a nonlinear registration algorithm to form a reference space; and the brain region in the reference space is inversely transformed to generate an individualized segmentation label; The second training module (not shown in the figure) is configured to train the segmentation model corresponding to the preset observation scale by taking the individualized label as a supervision signal.

[0080] Further, the analysis module 203 is further configured to classify the metabolic difference index based on a preset metabolic classification interval to determine the brain region of different metabolic categories; The multi-modal medical image processing device 200 further comprises: The rendering module (not shown in the figure) is configured to render the brain region in the brain PET image based on the first rendering parameter of the preset metabolic classification interval and / or the second rendering parameter of different brain regions to generate a metabolic difference image; The visualization module (not shown in the figure) is configured to display the metabolic difference image and the metabolic deviation index.

[0081] The specific definition of the processing device for the multi-modal medical image can refer to the definition of the processing method for the multi-modal medical image in the above, which will not be repeated here. Each module in the processing device for the multi-modal medical image described above can be realized by software, hardware and a combination thereof in whole or in part. The above-mentioned each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operation corresponding to each module.

[0082] Based on the method as described above, Figure 1 the corresponding embodiments of the present application also provide a readable storage medium, which stores a computer program, and the program is executed by a processor to realize the processing method for the multi-modal medical image as described above. Figure 1

[0083] Based on such understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each implementation scenario of the present application.

[0084] Based on the method as described above, Figure 1 and the virtual device embodiment as described above, Figure 2 in order to achieve the above-mentioned purpose, as described above, Figure 5 the embodiments of the present application also provide a computer device, which includes a processor 301 and a memory 302, and the memory 302 stores a program or instructions that can be run on the processor 301, and the program or instructions are executed by the processor 301 to realize the processing method for the multi-modal medical image as described above. Figure 1

[0085] ​​The memory 302 can be used to store software programs and various data. The memory 302 can mainly include a first storage area storing programs or instructions, and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 302 can include a volatile memory or a non-volatile memory, or the memory 302 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 302 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0086] The processor 301 can include one or more processing units; optionally, the processor 301 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 301.

[0087] The computer device can specifically be a personal computer, a server, a network device, and the like.

[0088] Optionally, the computer device can further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface can include a display, an input unit such as a keyboard, and the like. Optionally, the user interface can further include a USB interface, a card reader interface, and the like. The network interface can optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), and the like.

[0089] Those skilled in the art can understand that the computer device structure provided by the embodiment does not constitute a limitation on the computer device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware platforms, or by hardware to perform three-dimensional discrete wavelet transform on the brain MRI image of the target patient to generate MRI wavelet coefficients; input the MRI wavelet coefficients into a diffusion model to obtain the baseline PET wavelet coefficients of the brain of the target patient in a healthy state, wherein the diffusion model is trained based on MRI image samples and PET image samples in the healthy state; perform inverse wavelet transform on the baseline PET wavelet coefficients to generate a baseline PET image; compare the brain PET image of the target patient with the baseline PET image to determine the metabolic deviation index of the brain of the target patient. The embodiment of the present application maps the MRI structure of the individual patient to the multi-resolution of metabolic activity through the 3D wavelet diffusion model to simulate the real distribution of PET intensity in the healthy state and generate a baseline PET image that can represent the specificity of the patient. Taking the baseline PET image as the healthy reference PET of the patient, by comparing with the actual brain PET image of the patient, the metabolic abnormalities specific to the patient can be highlighted, rather than the deviation from the population average. Thus, each patient can take his or her own condition without neurodegenerative changes as a control, without spatial standardization to the population template, greatly reducing the distortion of anatomical structure caused by spatial standardization, realizing voxel-level precise analysis of neurodegenerative diseases, and helping to identify tiny, patient-specific pathological changes to assist clinicians in early diagnosis.

[0091] Those skilled in the art can understand that the modules or flows in the drawings are not necessarily required for implementing the present application. Those skilled in the art can understand that the modules in the devices in the implementation scenarios can be distributed in the devices in the implementation scenarios according to the description of the implementation scenarios, or can be changed to be located in one or more devices different from the implementation scenarios. The modules in the above implementation scenarios can be combined into one module, or can be further split into multiple sub-modules.

[0092] The above application numbers are only for description, and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only some specific implementation scenarios of the present application, but the present application is not limited thereto, and any variations that can be thought of by those skilled in the art should fall within the protection scope of the present application.

Claims

1. A method of processing multi-modal medical images, characterized in that, The method comprises: performing three-dimensional discrete wavelet transform on a brain MRI image of a target patient to generate MRI wavelet coefficients; inputting the MRI wavelet coefficients into a diffusion model to obtain baseline PET wavelet coefficients of the brain of the target patient in a healthy state, wherein the diffusion model is trained based on MRI image samples and PET image samples of the brain in the healthy state; performing inverse wavelet transform on the baseline PET wavelet coefficients to generate a baseline PET image; comparing the brain PET image of the target patient with the baseline PET image to determine a metabolic deviation index of the brain of the target patient.

2. The method of processing multi-modality medical images according to claim 1, characterized in that, The method further comprises: performing three-dimensional discrete wavelet transform on the MRI image samples and the PET image samples respectively to generate MRI wavelet coefficient samples and PET wavelet coefficient samples; adding random noise to the PET wavelet coefficient samples based on a forward diffusion algorithm to generate noisy PET wavelet coefficients; inputting the MRI wavelet coefficient samples and the noisy PET wavelet coefficients into a denoising diffusion probability model to obtain predicted noise of the PET wavelet coefficient samples; minimizing the mean square error of the random noise and the predicted noise as an objective, optimizing model parameters of the denoising diffusion probability model based on a loss function until a model iteration termination condition is met, and outputting the diffusion model.

3. The method of processing multi-modality medical images according to claim 2, characterized in that, The denoising diffusion probability model comprises an encoder of multiple frequency bands; inputting the MRI wavelet coefficient samples into the denoising diffusion probability model comprises: determining MRI wavelet subbands of multiple frequency bands based on the MRI wavelet coefficient samples; splicing the MRI wavelet subbands and feature maps of the encoder in the same frequency band to form joint feature maps; performing dynamic affine transformation on activation values of intermediate layers of the encoder based on scaling and shifting parameters, wherein the scaling and shifting parameters are determined based on the MRI wavelet coefficient samples; inputting the joint feature maps into the encoder after affine transformation in the corresponding frequency band.

4. The processing method of multi-modality medical images according to any one of claims 1 to 3, characterized in that, The comparison of the brain PET image of the target patient with the baseline PET image to determine the metabolic deviation index of the brain of the target patient comprises: obtaining at least one brain region in the brain PET image; calculating the standardized metabolic value ratio of the brain PET image and the baseline PET image in the brain region respectively; calculating the metabolic difference value of the brain region based on the standardized metabolic value ratio of the brain PET image and the standardized metabolic value ratio of the baseline PET image; wherein the metabolic deviation index comprises the standardized metabolic value ratio and / or the metabolic difference value.

5. The method of processing multi-modality medical images according to claim 4, characterized in that, The obtaining of at least one brain region in the brain PET image comprises: segmenting the brain PET image or the brain MRI image based on brain anatomical structure information to determine at least one brain region; or inputting the brain PET image or the brain MRI image into a segmentation model of a target observation scale to determine at least one brain region, wherein the segmentation model is trained based on individualized segmentation labels standardized to a standard brain template.

6. The method of processing multi-modality medical images according to claim 5, characterized in that, The method further comprises: acquire a standard brain template and a medical image sample under different preset observation scales, and form an image space of the standard brain template and the medical image sample, wherein the medical image sample includes the MRI image sample or the PET image sample, and the standard brain template is labeled with a partition label of a brain region; map the image space of the medical image sample to the image space of the standard brain atlas by using an affine transformation algorithm and a nonlinear registration algorithm, to form a reference space; perform inverse transformation on the brain region in the reference space to generate an individualized partition label; train a partition model corresponding to the preset observation scale by taking the individualized label as a supervision signal.

7. The method of processing multi-modality medical images according to claim 4, wherein, The method further includes: classify the metabolic difference indicators based on a preset metabolic classification interval, to determine brain regions of different metabolic categories; render the brain regions in the brain PET image based on a first rendering parameter of the preset metabolic classification interval and / or a second rendering parameter of different brain regions, to generate a metabolic difference image; display the metabolic difference image and the metabolic deviation indicator.

8. A processing apparatus of multi-modal medical images, characterized by, The device includes: a wavelet transformation module configured to perform three-dimensional discrete wavelet transformation on a brain MRI image of a target patient to generate MRI wavelet coefficients; a prediction module configured to input the MRI wavelet coefficients into a diffusion model to obtain reference PET wavelet coefficients of the brain of the target patient in a healthy state, wherein the diffusion model is trained based on MRI image samples and PET image samples of the brain in the healthy state; the wavelet transformation module is further configured to perform inverse wavelet transformation on the reference PET wavelet coefficients to generate a reference PET image; an analysis module configured to compare the brain PET image of the target patient with the reference PET image to determine a metabolic deviation indicator of the brain of the target patient.

9. A readable storage medium, on which a program or instructions are stored, characterized in that, The program or instructions, when executed by a processor, implement the steps of the processing method of the multi-modal medical image according to any one of claims 1 to 7.

10. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor, when executing the program, implements the processing method of the multi-modal medical image according to any one of claims 1 to 7.

Images