Multimodal medical image processing method and device, computer device and storage medium

Abstract

This application relates to a multimodal medical image processing method, apparatus, computer device, and storage medium. The method includes: acquiring a first MRI image sequence and a second MRI image sequence; selecting candidate PET images whose similarity meets a similarity condition as target PET images based on the similarity determined by registration between the second MRI image sequence and each of a plurality of candidate PET images; registering the first MRI image sequence and the second MRI image sequence to determine a first registration parameter between the first MRI image sequence and the target PET image; and fusing the first MRI image sequence and the target PET image according to the first registration parameter to obtain a first PET-MRI image. It can be understood that this method first establishes an accurate registration relationship between the second MRI image sequence and the target PET image, and then transforms the registration of the first MRI image sequence and the target PET image into the registration of the first MRI image sequence and the second MRI image sequence, eliminating the influence of respiratory and cardiac motion, and improving the imaging quality of the PET-MRI image.

Description

Technical Field

[0001] This application relates to the field of image fusion technology, and in particular to a multimodal medical image processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] PET-MRI (Positron Emission Magnetic Resonance Imaging) system is a molecular-level functional imaging and structural imaging system that integrates PET (Positron Emission Computed Tomography) and MRI (Magnetic Resonance Imaging) technologies. It leverages the high resolution of high-field superconducting MRI and the high sensitivity of PET to achieve complementary anatomical and functional imaging, realizing a powerful combination and cross-validation of MRI and PET functional imaging, enabling the diagnosis and monitoring of some complex diseases.

[0003] In cardiac imaging, integrated PET-MRI systems have significantly reduced mismatches compared to separate-machine scanning because PET and MRI images are acquired simultaneously. However, due to the influence of respiratory motion and involuntary cardiac movements, partial mismatches still exist between PET and MRI images during cardiac imaging with integrated PET-MRI systems, resulting in poor image quality of the fused PET-MRI images. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimodal medical image processing method, device, computer equipment, and storage medium that can improve the registration accuracy of PET-MRI images in cardiac imaging, addressing the aforementioned technical problems.

[0005] A multimodal medical image processing method, the method comprising:

[0006] Acquire the first MRI image sequence and the second MRI image sequence;

[0007] Based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, the candidate PET image whose similarity meets the similarity condition is selected as the target PET image.

[0008] The first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image;

[0009] Based on the first registration parameters, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image.

[0010] In one embodiment, the method further includes:

[0011] Based on the second registration parameters obtained by registering the second MRI image sequence with the target PET image, the second MRI image sequence and the target PET image are fused to obtain a second PET-MRI image.

[0012] In one embodiment, the first MRI image sequence is a sequence composed of two-dimensional MRI images;

[0013] The step of registering the first MRI image sequence with the second MRI image sequence to determine the first registration parameters between the first MRI image sequence and the target PET image includes:

[0014] Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image corresponding to the first MRI image sequence in three-dimensional space;

[0015] Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0016] In one embodiment, the first MRI image sequence is a sequence composed of three-dimensional MRI images;

[0017] The step of registering the first MRI image sequence with the second MRI image sequence to determine the first registration parameters between the first MRI image sequence and the target PET image includes:

[0018] Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image corresponding to the first MRI image sequence;

[0019] Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0020] In one embodiment, the method further includes:

[0021] Breathing-gated reconstruction was performed on the PET data of the subjects to obtain multi-phase reconstructed PET images;

[0022] The multi-stage reconstructed PET images are identified as the multiple sets of candidate PET images.

[0023] In one embodiment, the method further includes:

[0024] Based on the second MRI image sequence, an attenuation map is generated;

[0025] During the respiratory gating reconstruction process, the PET data is corrected according to the attenuation map.

[0026] In one embodiment, the second MRI image sequence is determined as follows:

[0027] Multiple sets of two-dimensional MRI image sequences of the object to be detected are acquired, and each set of two-dimensional MRI image sequences corresponds to at least one slice of the object to be detected.

[0028] The multiple two-dimensional MRI image sequences are combined into a three-dimensional MRI image sequence to obtain a second MRI image sequence.

[0029] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0030] Acquire the first MRI image sequence and the second MRI image sequence;

[0031] Based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, the candidate PET image whose similarity meets the similarity condition is selected as the target PET image.

[0032] The first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image;

[0033] Based on the first registration parameters, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image.

[0034] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0035] Acquire the first MRI image sequence and the second MRI image sequence;

[0036] Based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, the candidate PET image whose similarity meets the similarity condition is selected as the target PET image.

[0037] The first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image;

[0038] Based on the first registration parameters, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image.

[0039] The aforementioned multimodal medical image processing method, apparatus, computer equipment, and storage medium first determine the similarity between the second MRI image sequence and each of the multiple candidate PET images based on registration. A target PET image with high similarity is selected. Then, the first MRI image sequence is registered with the second MRI image sequence to determine a first registration parameter between the first MRI image sequence and the target PET image. Finally, based on the first registration parameter, the first MRI image sequence and the target PET image are fused to obtain a first PET-MRI image. It can be understood that this application first determines a target PET image with good registration with the second MRI image sequence, ensuring an accurate registration relationship between the second MRI image sequence and the target PET image. This eliminates the influence of respiratory and cardiac motion. Furthermore, the registration of other first MRI image sequences with the target PET image can be transformed into the registration of other first MRI image sequences with the second MRI image sequence, similarly eliminating the influence of respiratory and cardiac motion on the first MRI image sequence, thus improving the accuracy of image registration. Finally, based on the accurately registered first MRI image sequence and the target PET image, the fusion is performed, improving the imaging quality of the fused PET-MRI image. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating a multimodal medical image processing method in one embodiment;

[0041] Figure 2 This is a schematic diagram of the PET data gating reconstruction process in one embodiment;

[0042] Figure 3 This is a schematic diagram of a scanned object being the heart in one embodiment;

[0043] Figure 4 This is a schematic diagram illustrating image acquisition under multiple scanning phases in one embodiment;

[0044] Figure 5This is a schematic diagram of image registration using an MRI water-lipid separation sequence in one embodiment;

[0045] Figure 6 This is a schematic diagram illustrating the registration of images acquired using an LGE sequence with images acquired using a water-lipid separation sequence in one embodiment.

[0046] Figure 7 This is a structural block diagram of a multimodal medical image processing device in one embodiment;

[0047] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] In one embodiment, such as Figure 1 As shown, a multimodal medical image processing method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. Optionally, the terminal can be a PET-MRI imaging system. In this embodiment, the method includes the following steps:

[0050] Step S102: Obtain the first MRI image sequence and the second MRI image sequence.

[0051] The first MRI image sequence is generated based on a first pulse sequence / first scan sequence. The second MRI image sequence is generated based on a second pulse sequence / second scan sequence. The first pulse sequence and the second pulse sequence perform different functions. Optionally, as... Figure 5 As shown, the first pulse sequence is a cardiac magnetic resonance imaging (MRI) sequence, triggered using ECG-gated control. Breath-holding is required during the execution of the first pulse sequence; that is, execution of the first pulse sequence requires both ECG triggering and breath-holding. This cardiac MRI sequence can be one or more of the following: FSE dark blood sequence, Cine sequence, perfusion sequence, LGE sequence, mapping sequence, or flow sequence. The second pulse sequence is a water-lipid separation sequence. Breath-gated control is used for reconstructing the MRI signal excited by the second pulse sequence; that is, the second pulse sequence is executed under the subject's free breathing. This water-lipid separation sequence can be a Quick3d_dixon sequence or a non-Quick3d_dixon sequence.

[0052] Specifically, the terminal acquires a first MRI image sequence based on a first pulse sequence. The terminal acquires a second MRI image sequence based on a second pulse sequence. Optionally, the first and second MRI image sequences are a series of images of the object being detected, and the images can be two-dimensional or three-dimensional images. In this embodiment, the second MRI image sequence may correspond to an image of one phase of the respiratory gating of the object being detected.

[0053] Step S104: Based on the similarity determined by registration between the second MRI image sequence and each of the multiple candidate PET images, select the candidate PET image whose similarity meets the similarity condition as the target PET image.

[0054] The multiple candidate PET images are determined as follows: respiratory-gated reconstruction is performed on the PET data of the subject to obtain multi-phase reconstructed PET images (PET images corresponding to multiple phases); these multi-phase reconstructed PET images are then identified as multiple candidate PET images. Optionally, each phase of the reconstructed PET image may undergo attenuation correction: an attenuation map is generated based on the second MRI image sequence; during the respiratory-gated reconstruction process, the PET data is corrected according to the attenuation map. In this embodiment, the PET data is also acquired under the subject's free breathing, and the reconstruction of the multiple candidate PET images is divided into multiple phases using respiratory gating.

[0055] The similarity score is used to evaluate the registration effect between the second MRI image sequence and the candidate PET image. The higher the similarity score, the better the registration effect between the second MRI image sequence and the candidate PET image; conversely, the lower the similarity score, the worse the registration effect between the second MRI image sequence and the candidate PET image.

[0056] Specifically, the terminal registers the second MRI image sequence with each of the multiple candidate PET images to obtain second registration parameters. Furthermore, based on the second MRI image sequence, the candidate PET images, and the corresponding second registration parameters, the terminal calculates a similarity score to evaluate the registration effect, and then selects the candidate PET image whose similarity meets the similarity criteria as the target PET image. Optionally, the terminal calculates the similarity score corresponding to the registration between the second MRI image sequence and each of the multiple candidate PET images using a mutual information algorithm.

[0057] Optionally, the similarity condition can be the highest similarity, the second highest similarity, or a similarity greater than a similarity threshold, etc. The purpose of setting similarity conditions is to select the target PET image with better registration results.

[0058] Step S106: Register the first MRI image sequence with the second MRI image sequence to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0059] Specifically, the terminal registers the first MRI image sequence with the second MRI image sequence to determine the first registration parameters between the first MRI image sequence and the target PET image. Optionally, the first registration parameters can be represented in matrix form.

[0060] Step S108: Based on the first registration parameters, fuse the first MRI image sequence and the target PET image to obtain the first PET-MRI image.

[0061] The registration parameters can be offset fields, displacement fields, or translation amounts.

[0062] PET-MRI images are obtained by fusing PET and MRI images.

[0063] Specifically, the terminal fuses the first MRI image sequence and the target PET image according to the first registration parameters to obtain the first PET-MRI image.

[0064] In the aforementioned multimodal medical image processing method, firstly, based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, a target PET image with high similarity is selected. Then, the first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image. Finally, based on the first registration parameter, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image. It can be understood that this application first determines the target PET image with good registration with the second MRI image sequence, ensuring an accurate registration relationship between the second MRI image sequence and the target PET image. This eliminates the influence of respiratory and cardiac motion, and the registration of other first MRI image sequences with the target PET image can be transformed into the registration of other first MRI image sequences with the second MRI image sequence, similarly eliminating the influence of respiratory and cardiac motion on the first MRI image sequence, thus improving the accuracy of image registration. Furthermore, the fusion of the accurately registered first MRI image sequence and the target PET image improves the imaging quality of the fused PET-MRI image.

[0065] In one embodiment, the method further includes the following steps:

[0066] Step S109: Based on the second registration parameters obtained by registering the second MRI image sequence with the target PET image, fuse the second MRI image sequence and the target PET image to obtain the second PET-MRI image.

[0067] Specifically, the terminal fuses the second MRI image sequence and the target PET image using the second registration parameters obtained by registering the second MRI image sequence with the target PET image to obtain a second PET-MRI image. Optionally, the second registration parameters can be represented in matrix form.

[0068] In this embodiment, a second MRI image sequence with accurate registration is fused with a target PET image. Since the effects of respiratory and cardiac movements are eliminated, the imaging quality of the fused second PET-MRI image is improved.

[0069] In one embodiment, the first MRI image sequence is a sequence composed of two-dimensional MRI images. Based on this, in one embodiment, a possible implementation of step S106, "registering the first MRI image sequence with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image," is provided. Based on the above embodiment, step S106 can be specifically implemented through the following steps:

[0070] Step S1062: Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image corresponding to the first MRI image sequence in three-dimensional space.

[0071] Step S1064: Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using the generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0072] The second pulse sequence is a three-dimensional MRI imaging sequence, and the second MRI imaging sequence is a sequence composed of three-dimensional MRI images.

[0073] Specifically, the terminal acquires eight degrees of freedom corresponding to the first MRI image sequence, namely, translational degrees of freedom in three-dimensional space, rotational degrees of freedom in three-dimensional space, and scaling degrees of freedom in two-dimensional image. Then, based on the translational degrees of freedom in three-dimensional space, rotational degrees of freedom in three-dimensional space, and scaling degrees of freedom in two-dimensional image, the terminal registers the first MRI image sequence with the second MRI image sequence using the Generalized Pattern Search (GPS) method, obtaining the registration result between the first MRI image sequence and the second MRI image sequence. This registration result is determined as the first registration parameter between the first MRI image sequence and the target PET image. At the same time, the terminal evaluates the registration effect using the Normalized Mutual Information method and calculates the similarity between the registration of the first MRI image sequence and the target PET image.

[0074] In this embodiment, registration of two-dimensional MRI images with three-dimensional MRI images is performed based on eight degrees of freedom and through a generalized pattern search method, which helps to improve the accuracy of registration.

[0075] In one embodiment, the first MRI image sequence is a sequence composed of three-dimensional MRI images. Based on this, in one embodiment, a possible implementation of step S106, "registering the first MRI image sequence with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image," is provided. Based on the above embodiment, step S106 can be specifically implemented through the following steps:

[0076] Step S106a: Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image corresponding to the first MRI image sequence.

[0077] Step S106b: Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using the generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0078] The second pulse sequence is a three-dimensional MRI imaging sequence, and the second MRI imaging sequence is a sequence composed of three-dimensional MRI images.

[0079] Specifically, the terminal acquires nine degrees of freedom corresponding to the first MRI image sequence, namely, translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom in three-dimensional space. Then, based on the translational, rotational, and scaling degrees of freedom in three-dimensional space, the terminal registers the first MRI image sequence with the second MRI image sequence using the Generalized Pattern Search (GPS) method, obtaining the registration result between the first and second MRI image sequences. This registration result is determined as the first registration parameter between the first MRI image sequence and the target PET image. Simultaneously, the terminal evaluates the registration effect using the Normalized Mutual Information (MMI) method, calculating the similarity between the registration of the first MRI image sequence and the target PET image.

[0080] In this embodiment, registration of three-dimensional MRI images is performed based on nine degrees of freedom and using a generalized pattern search method, which helps to improve the accuracy of registration.

[0081] In one embodiment, the method further includes the following steps:

[0082] Step S112: Perform respiratory-gated reconstruction on the PET image to be reconstructed to obtain multi-phase reconstructed PET images;

[0083] Step S114: The PET images reconstructed from multiple phases are identified as multiple sets of candidate PET images.

[0084] Among them, respiratory gating technology refers to selecting PET biodata within a specific respiratory frequency range according to the respiratory physiological curve acquired synchronously during image acquisition, dividing the respiratory cycle into several equal parts by amplitude or phase angle, and integrating and reconstructing the PET biodata corresponding to specific respiratory periods to obtain multiple boxes of PET data, which represent PET images acquired under different respiratory states.

[0085] Specifically, such as Figure 5 As shown, the terminal performs respiratory-gated reconstruction on the PET images to be reconstructed, obtaining multi-stage reconstructed PET images, and identifies the multi-stage reconstructed PET images as multiple sets of candidate PET images.

[0086] In this embodiment, respiratory gating technology is used to reconstruct PET images, which can reduce artifacts caused by respiratory movements and improve the imaging quality of PET images.

[0087] In one embodiment, such as Figure 2 The method shown also includes the following steps:

[0088] Step S122: Generate an attenuation map based on the second MRI image sequence;

[0089] Step S124: Respiratory gating reconstruction is performed on the PET data of the test subject to obtain multi-phase reconstructed PET images. During the respiratory gating reconstruction process, the PET data is corrected according to the attenuation map.

[0090] The attenuation coefficient is used for attenuation correction of PET images. Optionally, air corresponds to the smallest attenuation coefficient, while bone corresponds to the largest. Attenuation correction of PET images refers to the phenomenon of photons being scattered or absorbed by tissues when traveling through a medium; compensation for attenuation is necessary to obtain images for quantitative analysis.

[0091] The initial PET image can be a PET image scanned by a positron emission tomography (PET) scanner.

[0092] Specifically, the terminal acquires a second pulse sequence used to generate a second MRI image sequence, then generates an attenuation coefficient based on the second pulse sequence, and finally corrects the initial PET image based on the attenuation coefficient to obtain the PET image to be reconstructed. Optionally, the terminal can generate a μ-MAP based on a Quick3d_dixon sequence; the terminal can also generate a μ-MAP based on a non-Quick3d_dixon sequence.

[0093] In this embodiment, attenuation correction is performed on the PET image using an attenuation coefficient, which further improves the imaging quality of the PET image.

[0094] In one embodiment, a target neural network model is provided that provides a mapping relationship between MRI image sequences and corresponding attenuation correction data.

[0095] In one embodiment, the target neural network model may include a convolutional neural network (CNN) model, a backpropagation neural network (BP) model, a radial basis function neural network (RBF) model, a deep belief network (DBN) model, an Elman neural network model, or a combination thereof.

[0096] In one embodiment, the target neural network model is trained using multiple sets of sample data. Each set of sample data includes attenuation maps corresponding to prior MRI and prior CT images of the same location. The attenuation map corresponding to the prior CT image can be determined as follows: the classification information of multiple tissues, such as lungs, fat, ribs, spine, and heart, is determined in the prior CT image; based on the classification information of multiple tissues, a corresponding attenuation value is assigned to each voxel of the location, generating the attenuation map corresponding to the prior CT image.

[0097] In one embodiment, the target neural network model is trained using multiple sets of sample data. Each set of sample data includes attenuation maps corresponding to prior MRI and prior CT images of the same location. The attenuation map corresponding to the prior CT image can be determined as follows: the classification information of multiple tissues, such as lungs, fat, ribs, spine, and heart, is determined from the prior CT image; the prior CT image is registered with a prior PET image of the same location; and a corresponding attenuation value is assigned to each voxel of the registered prior PET image based on the classification information of multiple tissues, thereby generating the attenuation map corresponding to the prior CT image.

[0098] Optionally, such as Figure 3 As shown, the object being scanned is the heart. Respiratory movements cause the heart to shift in the direction of breathing, resulting in a discrepancy between the actual cardiac scan slice and the pre-defined slice.

[0099] In one embodiment, the slice being scanned is the short axial plane (SA plane) of the heart. Due to the imaging of respiratory motion, there is a certain displacement between the expected and acquired short axial plane. Consequently, there is also a degree of misregistration between the images of the two different slices.

[0100] In the prior art, gating technology can be used to suppress the influence of respiratory or cardiac movements on medical images.

[0101] In this embodiment, such as Figure 4 As shown, images acquired during various scanning phases are illustrated: Acquisition Mode 1 - ECG gating (ECG gating trigger) + end-expiratory breath-hold; Acquisition Mode 2 - ECG gating + end-inspiratory breath-hold; Acquisition Mode 3 - Free heartbeat + respiratory gating (respiratory gating reconstruction); Acquisition Mode 4 - Free heartbeat + free breathing; Acquisition Mode 5 - Free heartbeat + respiratory gating reconstruction. In the figure, the dark-colored ring represents the MRI myocardial image acquired using Acquisition Mode 1, and this image serves as the reference image. The light-colored ring represents the myocardial images acquired using Acquisition Modes 2-5 respectively. In this embodiment, there is a significant mismatch between the MR image acquired using Acquisition Mode 2 and the reference image; the MR image acquired using Acquisition Mode 3 shows the same myocardial location as the reference image, but the myocardial thickness differs significantly. In actual cardiac imaging, the scanning technician sets the cardiac scan to Acquisition Mode 1, but when the patient does not strictly adhere to the instructions for breath-holding, the actual acquisition mode is Acquisition Mode 2, resulting in a significant mismatch between the two acquisition modes.

[0102] Acquisition method 4 specifically performs a PET imaging scan, and the acquired PET images show both mismatch and positional differences compared to the reference images. Acquisition method 5 also performs a PET imaging scan, and the acquired PET images show essentially the same myocardial thickness and position as the MRI images from acquisition method 3. Based on this, during PET-MRI imaging, the MRI images from acquisition method 3 and the PET images from acquisition method 5 can be selected for registration / matching.

[0103] In one embodiment, such as Figure 5 As shown, the MRI water-lipid separation sequence (water_Dixon in the figure) was acquired using a respiratory-gated method, and it was accurately matched with the end-expiratory PET image obtained by PET respiratory-gated reconstruction. The MRI water-lipid separation sequence can be used to replace PET images for direct registration with cardiac magnetic resonance imaging sequences (cine, fse dark blood, LGE, etc. in the figure).

[0104] In one embodiment, such as Figure 6 As shown, the cardiac magnetic resonance imaging sequence selected was the lategadolinium enhancement (LGE) sequence, whose acquired images were influenced by respiratory motion; the MRI water-lipid separation sequence used was the GRE_quick3d sequence, which employed respiratory gating.

[0105] Registering images acquired using the LGE sequence with images acquired using the water-lipid separation sequence yields a registration matrix, which represents the motion vector field between the two sequences. Applying this registration matrix to PET images results in transformed PET images, which show a more accurate match with the images acquired using the LGE sequence. In contrast, untransformed PET images exhibit mismatches with the images acquired using the LGE sequence.

[0106] In multimodal cardiac imaging, image registration between different modalities is often required. When certain modalities, such as PET or SPECT, display cardiac structures in a blurry or indistinct manner, users typically cannot determine the accuracy of the registration between the two modalities, let alone perform manual or automatic registration. This invention, through research on the processing characteristics of respiratory and cardiac movements in different modalities, suggests that if similar strategies are used to process cardiac and respiratory movements across different modalities, the size and position of the heart will remain consistent in both modalities. Based on this principle, replacing unclear images with images showing clear anatomical structures for registration can significantly improve registration efficiency. For example, in PET / MRI imaging of myocardial inflammation, only the lesion site of the myocardium shows high uptake, while the rest of the myocardium is not clearly displayed; therefore, it is impossible to determine whether there is a mismatch between the PET image and the MRI, and how to correct it if a mismatch occurs. This invention solves a problem that was previously unsolvable in clinical practice by replacing PET images with Water-Dixon sequences showing clear anatomical structures for registration with other cardiac MRI sequences.

[0107] In one embodiment, one possible implementation of "acquiring the second MRI image sequence" in step S102 described above is as follows: Based on the above embodiment, this step can be specifically implemented through the following steps:

[0108] Step S1022: Obtain the second pulse sequence;

[0109] Step S1024: If the second pulse sequence is a two-dimensional MRI image imaging sequence, then imaging is performed according to the second pulse sequence to obtain a two-dimensional MRI image sequence.

[0110] Step S1026: The two-dimensional MRI image sequence is converted into a three-dimensional MRI image sequence to obtain the second MRI image sequence.

[0111] Specifically, the terminal acquires a second pulse sequence. If the second pulse sequence is a two-dimensional MRI image sequence, imaging is performed based on the second pulse sequence to obtain a two-dimensional MRI image sequence. This two-dimensional MRI image sequence is then converted into a three-dimensional MRI image sequence, and this three-dimensional MRI image sequence is identified as the second MRI image sequence. It can be understood that this second MRI image sequence is a sequence composed of three-dimensional MRI images. Optionally, the second pulse sequence can be acquired using a respiratory gating method.

[0112] In this embodiment, converting the two-dimensional MRI image sequence into a three-dimensional MRI image sequence is beneficial for the accurate implementation of subsequent image registration.

[0113] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0114] In one embodiment, such as Figure 7 As shown, a multimodal medical image processing device is provided, including: a sequence acquisition module 202, an image selection module 204, an image registration module 206, and an image fusion module 208, wherein:

[0115] Sequence acquisition module 202 is used to acquire a first MRI image sequence and a second MRI image sequence;

[0116] The image selection module 204 is used to select candidate PET images whose similarity meets the similarity conditions as target PET images based on the similarity determined by the registration between the second MRI image sequence and each of the multiple candidate PET images.

[0117] The image registration module 206 is used to register the first MRI image sequence with the second MRI image sequence and determine the first registration parameters between the first MRI image sequence and the target PET image;

[0118] The image fusion module 208 is used to fuse the first MRI image sequence and the target PET image according to the first registration parameters to obtain the first PET-MRI image.

[0119] In the aforementioned multimodal medical image processing device, firstly, based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, a target PET image with high similarity is selected. Then, the first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image. Finally, based on the first registration parameter, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image. It can be understood that this device first determines the target PET image with good registration with the second MRI image sequence, ensuring an accurate registration relationship between the second MRI image sequence and the target PET image. This eliminates the influence of respiratory and cardiac motion. Furthermore, the registration of other first MRI image sequences with the target PET image can be transformed into the registration of other first MRI image sequences with the second MRI image sequence, similarly eliminating the influence of respiratory and cardiac motion on the first MRI image sequence, thus improving the accuracy of image registration. Finally, based on the accurately registered first MRI image sequence and the target PET image, the fusion is performed, improving the imaging quality of the fused PET-MRI image.

[0120] In one embodiment, the image fusion module 208 is further configured to fuse the second MRI image sequence and the target PET image according to the second registration parameters obtained by registering the second MRI image sequence with the target PET image to obtain a second PET-MRI image.

[0121] In one embodiment, the first MRI image sequence is a sequence composed of two-dimensional MRI images. The image registration module 206 is specifically used to obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional images corresponding to the first MRI image sequence in three-dimensional space; based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional images, the first MRI image sequence is registered with the second MRI image sequence using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0122] In one embodiment, the first MRI image sequence is a sequence composed of three-dimensional MRI images. The image registration module 206 is specifically used to obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image corresponding to the first MRI image sequence; based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image, the first MRI image sequence is registered with the second MRI image sequence using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0123] In one embodiment, the apparatus further includes: an image reconstruction module and an image determination module, wherein:

[0124] The image reconstruction module is used to perform respiratory-gated reconstruction on the PET images to be reconstructed, resulting in multi-phase reconstructed PET images.

[0125] The image determination module is used to identify multiple sets of candidate PET images after multi-stage reconstruction.

[0126] In one embodiment, the device further includes: a parameter generation module and an image correction module, wherein:

[0127] The parameter generation module is used to generate an attenuation map based on the second MRI image sequence;

[0128] The image correction module is used to correct the PET data according to the attenuation map during the respiratory gating reconstruction process.

[0129] In one embodiment, the sequence acquisition module 202 is specifically used to acquire a second pulse sequence; if the second pulse sequence is a two-dimensional MRI image imaging sequence, imaging is performed based on the second pulse sequence to obtain a two-dimensional MRI image sequence; the two-dimensional MRI image sequence is converted / combined into a three-dimensional MRI image sequence to obtain a second MRI image sequence. In this embodiment, multiple sets of two-dimensional MRI image sequences of the detection object are acquired, each set of two-dimensional MRI image sequences corresponding to at least one slice of the detection object; multiple two-dimensional MRI image sequences are combined into a three-dimensional MRI image sequence to obtain a second MRI image sequence.

[0130] Specific limitations regarding the multimodal medical image processing device can be found in the limitations of the multimodal medical image processing method described above, and will not be repeated here. Each module in the aforementioned multimodal medical image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0131] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multimodal medical image processing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0132] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0134] Acquire the first MRI image sequence and the second MRI image sequence;

[0135] Based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, the candidate PET image that meets the similarity condition is selected as the target PET image.

[0136] The first MRI image sequence and the second MRI image sequence are registered to determine the first registration parameters between the first MRI image sequence and the target PET image;

[0137] Based on the first registration parameters, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image.

[0138] In the aforementioned computer device, firstly, based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, a target PET image with high similarity is selected. Then, the first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image. Finally, based on the first registration parameter, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image. It can be understood that this computer device, by first determining the target PET image with good registration with the second MRI image sequence, ensures an accurate registration relationship between the second MRI image sequence and the target PET image. This eliminates the influence of respiratory and cardiac motion. Furthermore, it transforms the registration of other first MRI image sequences with the target PET image into the registration of other first MRI image sequences with the second MRI image sequence, similarly eliminating the influence of respiratory and cardiac motion on the first MRI image sequence, thus improving the accuracy of image registration. Finally, based on the accurately registered first MRI image sequence and the target PET image, fusion is performed, improving the imaging quality of the fused PET-MRI image.

[0139] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0140] The second PET-MRI image is obtained by fusing the second MRI image sequence and the target PET image based on the second registration parameters obtained by registering the second MRI image sequence with the target PET image.

[0141] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0142] The first MRI image sequence and the second MRI image sequence are registered to obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image corresponding to the first MRI image sequence. Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0143] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0144] Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image corresponding to the first MRI image sequence; based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image, register the first MRI image sequence with the second MRI image sequence using a generalized pattern search method, and determine the first registration parameters between the first MRI image sequence and the target PET image.

[0145] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0146] Breathing-gated reconstruction was performed on the PET images to be reconstructed to obtain multi-stage reconstructed PET images; the multi-stage reconstructed PET images were then identified as multiple candidate PET images.

[0147] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0148] Based on the second MRI image sequence, generate an attenuation map;

[0149] During respiratory gating reconstruction, PET data are attenuated based on the attenuation map.

[0150] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0151] Acquire the first MRI image sequence and the second MRI image sequence;

[0152] Based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, the candidate PET image that meets the similarity condition is selected as the target PET image.

[0153] The first MRI image sequence and the second MRI image sequence are registered to determine the first registration parameters between the first MRI image sequence and the target PET image;

[0154] Based on the first registration parameters, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image.

[0155] In the aforementioned computer-readable storage medium, firstly, based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, a target PET image with high similarity is selected. Then, the first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image. Finally, based on the first registration parameter, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image. It can be understood that this storage medium, by first determining the target PET image with good registration with the second MRI image sequence, ensures an accurate registration relationship between the second MRI image sequence and the target PET image. This eliminates the influence of respiratory and cardiac motion, and transforms the registration of other first MRI image sequences with the target PET image into the registration of other first MRI image sequences with the second MRI image sequence, similarly eliminating the influence of respiratory and cardiac motion on the first MRI image sequence, thus improving the accuracy of image registration. Furthermore, the fusion of the accurately registered first MRI image sequence and the target PET image improves the imaging quality of the fused PET-MRI image.

[0156] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0157] The second PET-MRI image is obtained by fusing the second MRI image sequence and the target PET image based on the second registration parameters obtained by registering the second MRI image sequence with the target PET image.

[0158] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0159] The first MRI image sequence and the second MRI image sequence are registered to obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image corresponding to the first MRI image sequence. Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

[0160] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0161] Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image corresponding to the first MRI image sequence; based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image, register the first MRI image sequence with the second MRI image sequence using a generalized pattern search method, and determine the first registration parameters between the first MRI image sequence and the target PET image.

[0162] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0163] Breathing-gated reconstruction was performed on the PET images to be reconstructed to obtain multi-stage reconstructed PET images; the multi-stage reconstructed PET images were then identified as multiple candidate PET images.

[0164] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0165] Based on the second MRI image sequence, generate an attenuation map;

[0166] During respiratory gating reconstruction, PET data are corrected based on the attenuation map.

[0167] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0168] Obtain the second pulse sequence; if the second pulse sequence is a two-dimensional MRI imaging sequence, convert the second pulse sequence into a three-dimensional MRI imaging sequence; perform imaging based on the three-dimensional MRI imaging sequence to obtain the second MRI imaging sequence.

[0169] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0170] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-modality medical image processing method, characterized by, The method includes: Acquire a first MRI image sequence and a second MRI image sequence; the first MRI image sequence is generated based on a first pulse sequence, which is triggered by ECG gating and involves breath-holding during the execution of the first pulse sequence; the second MRI image sequence is generated based on a second pulse sequence, the magnetic resonance signal excited by the second pulse sequence is reconstructed using respiratory gating and the second pulse sequence is executed under free breathing. Based on the similarity determined by registering the second MRI image sequence with each of the multiple candidate PET images, the candidate PET images that meet the similarity conditions are selected as the target PET images; the multiple candidate PET images are obtained by respiratory gating reconstruction, and each candidate PET image corresponds to PET images acquired under different respiratory states; The first MRI image sequence is registered with the second MRI image sequence to determine the first registration parameter between the first MRI image sequence and the target PET image; Based on the first registration parameters, the first MRI image sequence and the target PET image are fused to obtain the first PET-MRI image.

2. The method of claim 1, wherein, The method further includes: Based on the second registration parameters obtained by registering the second MRI image sequence with the target PET image, the second MRI image sequence and the target PET image are fused to obtain a second PET-MRI image.

3. The method of claim 1, wherein, The first MRI image sequence is a sequence composed of two-dimensional MRI images; The step of registering the first MRI image sequence with the second MRI image sequence to determine the first registration parameters between the first MRI image sequence and the target PET image includes: Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image corresponding to the first MRI image sequence in three-dimensional space; Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the two-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

4. The method of claim 1, wherein, The first MRI image sequence is a sequence composed of three-dimensional MRI images; The step of registering the first MRI image sequence with the second MRI image sequence to determine the first registration parameters between the first MRI image sequence and the target PET image includes: Obtain the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image corresponding to the first MRI image sequence; Based on the translational degrees of freedom, rotational degrees of freedom, and scaling degrees of freedom of the three-dimensional image, the first MRI image sequence and the second MRI image sequence are registered using a generalized pattern search method to determine the first registration parameters between the first MRI image sequence and the target PET image.

5. The method according to claim 1, characterized in that, The method further includes: Breathing-gated reconstruction was performed on the PET data of the subjects to obtain multi-phase reconstructed PET images; The multi-stage reconstructed PET images are identified as the multiple sets of candidate PET images.

6. The method of claim 5, wherein, The method further includes: Based on the second MRI image sequence, an attenuation map is generated; During the respiratory gating reconstruction process, the PET data is corrected according to the attenuation map.

7. The method of claim 1, wherein, The second MRI image sequence was determined as follows: Acquire multiple sets of two-dimensional MRI image sequences of the object to be detected, each set of two-dimensional MRI image sequences corresponding to at least one slice of the object to be detected; The multiple two-dimensional MRI image sequences are combined into a three-dimensional MRI image sequence to obtain a second MRI image sequence.

8. A multi-modality medical image processing apparatus, characterized by, The device includes: The sequence acquisition module is used to acquire a first MRI image sequence and a second MRI image sequence; the first MRI image sequence is generated based on a first pulse sequence, which is triggered by ECG gating and involves breath-holding during the execution of the first pulse sequence; the second MRI image sequence is generated based on a second pulse sequence, the magnetic resonance signal excited by the second pulse sequence is reconstructed using respiratory gating and the second pulse sequence is executed under free breathing. The image selection module is used to select candidate PET images that meet the similarity conditions as target PET images based on the similarity determined by registration between the second MRI image sequence and each of the multiple candidate PET images; the multiple candidate PET images are obtained by respiratory gating reconstruction, and each candidate PET image corresponds to PET images acquired under different respiratory states; The image registration module is used to register the first MRI image sequence with the second MRI image sequence and determine the first registration parameters between the first MRI image sequence and the target PET image; The image fusion module is used to fuse the first MRI image sequence and the target PET image according to the first registration parameters to obtain a first PET-MRI image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images