A method of registering multi-modal medical images and related apparatus

Abstract

The application discloses a multi-modal medical image registration method and related device, the method comprising inputting a first modal image and a second modal image into a style conversion module, determining a first modal conversion image of the first modal image and a second modal conversion image of the second modal image through the style conversion module; inputting the first modal image and the first modal conversion image into a first registration module, determining a first registration image through the first registration module; inputting the second modal image and the second modal conversion image into a second registration module, and determining a second registration image through the second registration module. The application realizes bidirectional image style conversion of the first modal image and the second modal image through the style conversion module, and improves the quality of the converted image by constraining the shape of the multi-modal image through shape prior knowledge, thereby retaining the structural information of the original image, which can improve the registration accuracy of the multi-modal image.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a multimodal medical image registration method and related devices. Background Art

[0002] In recent years, with the rapid development of deep learning, computer-aided diagnosis (CAD) technology has become increasingly mature. Medical image registration has become a research hotspot for CAD, with numerous applications in medical fields such as imaging diagnosis, surgical navigation, and radiotherapy planning.

[0003] Image registration primarily predicts spatial transformation parameters between images based on similarity metrics, establishing linear or nonlinear point-to-point position correspondences to achieve spatial consistency. With the rapid development of medical imaging equipment, medical imaging technology is trending towards multidimensionality, multimodality, and high resolution. For the same patient, multiple images with different anatomical information can be acquired, such as computed tomography (CT) and magnetic resonance imaging (MRI). Simultaneously, images with functional information, such as single-photon emission computed tomography (SPECT), can also be acquired. However, because each imaging technique has its own advantages and disadvantages, a single imaging technique is no longer sufficient for disease diagnosis. Therefore, image registration technology can be used to fuse multimodal images according to the physician's needs, enabling faster and more accurate visualization of lesions and other anatomical structures from various angles, making clinical diagnoses, surgical procedures, and radiotherapy planning more accurate and reliable. For example, the clinical applications of multimodal image registration include: targeted prostate puncture biopsy guided by MRI-TRUS fusion; expression of the structural morphology of the lesion area based on PET / CT image fusion; and three-dimensional reconstruction technology based on image registration.

[0004] Traditional multimodal image registration methods primarily use similarity metrics as optimization targets, achieving optimal image matching through continuous iterative optimization. However, this method requires a long registration time for high-dimensional medical images and advanced computing resources. Furthermore, similarity metrics are not applicable to images of multiple modalities, resulting in poor generalization performance of existing multimodal image registration methods. While existing deep learning-based multimodal image registration methods effectively address the issue of long registration times, they generally fail to preserve image detail information, thus failing to guarantee the quality of the registered images.

[0005] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0006] The technical problem to be solved by this application is to provide a multimodal medical image registration method and related devices in response to the deficiencies of the existing technology.

[0007] In order to solve the above technical problems, a first aspect of an embodiment of the present application provides a multimodal medical image registration method, the registration method comprising:

[0008] Inputting the first modality image and the second modality image to be registered into a style conversion module, and determining, by the style conversion module, a second modality conversion image corresponding to the first modality image and a first modality conversion image corresponding to the second modality image;

[0009] Inputting the first modality image and the first modality conversion image into a first registration module, determining a first deformation field by the first registration module, and determining a first registered image corresponding to the first modality image based on the first deformation field;

[0010] The second modality image and the second modality conversion image are input into a second registration module, a second deformation field is determined by the second registration module, and a second registered image corresponding to the second modality image is determined based on the second deformation field.

[0011] The multimodal medical image registration method, wherein the module structure of the first registration module is the same as the module structure of the second registration module, the first registration module includes a rigid transformation unit, an affine unit, a non-rigid transformation unit, and a deformation unit, and the inputting of the first modality image and the first modality conversion image into the first registration module, determining the first deformation field by the first registration module, and determining the first registered image corresponding to the first modality image based on the first deformation field specifically includes:

[0012] Inputting the first modality image and the first modality conversion image into a rigid body transformation unit, and determining an affine matrix through the rigid body transformation unit;

[0013] Inputting the affine matrix and the first modal conversion image into an affine unit, and outputting an affine image through the affine unit;

[0014] Inputting the affine image and the first modality image into a non-rigid transformation unit, and determining a first deformation field through the non-rigid transformation unit;

[0015] The first deformation field and the affine image are input into the deformation unit, and the first registration image is determined by the deformation unit.

[0016] The multimodal medical image registration method, wherein the style conversion module includes a first generation unit and a second generation unit, inputting the first modality image and the second modality image to be registered into the style conversion module, and determining the second modality conversion image corresponding to the first modality image and the first modality conversion image corresponding to the second modality image through the style conversion module specifically includes:

[0017] Inputting the first modality image to be registered into a first generating unit, and determining the second modality conversion image corresponding to the first modality image by the first generating unit;

[0018] The second modality image to be registered is input into the second generation unit, and the first modality conversion image corresponding to the second modality image is determined by the second generation unit.

[0019] The multimodal medical image registration method, wherein the training process of the style conversion module specifically includes:

[0020] Input the first modality training image T1 into the first generator G1 and the second generator G2 respectively, generate a first predicted image sT2 through the first generator G1, generate a second predicted image iT1 through the second generator G2, and input the first predicted image sT2 into the second generator G2, and generate a third predicted image rT1 through the second generator G2;

[0021] Input the second modality training image T2 into the first generator G1 and the second generator G2 respectively, generate a fourth predicted image iT2 through the first generator G1, generate a fifth predicted image sT1 through the second generator G2, and input the fifth predicted image sT1 into the first generator G1, and generate a sixth predicted image rT2 through the first generator G1;

[0022] Input the first modality training image T1 and the fifth predicted image sT1 into the first discriminator D1, and determine the first true probability corresponding to the first modality training image T1 and the second true probability corresponding to the fifth predicted image sT1 through the first discriminator D1;

[0023] Input the second modality training image T2 and the first predicted image sT2 into the second discriminator D2, and determine through the second discriminator D2 whether the second modality training image T2 corresponds to the third true probability and the first predicted image sT2 corresponds to the fourth true probability;

[0024] The first generator G1, the second generator G2, the first discriminator D1 and the second discriminator D2 are trained based on the first modality training image T1, the second modality training image T2, the first predicted image sT2, the second predicted image iT1, the third predicted image rT1, the fourth predicted image iT2, the fifth predicted image sT1, the sixth predicted image rT2, the first true probability, the second true probability, the third true probability and the fourth true probability, and the trained first generator G1 is used as the first generation unit and the second generator G2 is used as the second generation unit to obtain a style transfer module.

[0025] The multimodal medical image registration method, wherein the training of the first generator G1, the second generator G2, the first discriminator D1, and the second discriminator D2 based on the first modality training image T1, the second modality training image T2, the first predicted image sT2, the second predicted image iT1, the third predicted image rT1, the fourth predicted image iT2, the fifth predicted image sT1, the sixth predicted image rT2, the first true probability, the second true probability, the third true probability, and the fourth true probability specifically includes:

[0026] Determine a first loss term based on the first true probability and the second true probability and train a first discriminator D1 based on the first loss term;

[0027] Determining a second loss term based on the third true probability and the fourth true probability and training a second discriminator D2 based on the second loss term;

[0028] determining a third loss term based on the third predicted image rT1 and the first modality training image T1, determining a fourth loss term based on the sixth predicted image rT2 and the second modality training image T2, determining a fifth loss term based on the fourth predicted image iT2 and the second modality training image T2, determining a sixth loss term based on the second predicted image iT1 and the first modality training image T1, determining a seventh loss term based on the sixth predicted image rT2 and the first predicted image sT2, and determining an eighth loss term based on the third predicted image rT1 and the fifth predicted image sT1;

[0029] Train the first generator G1 based on the first loss term, the third loss term, the fourth loss term, the fifth loss term, the sixth loss term, the seventh loss term, and the eighth loss term;

[0030] The second generator G2 is trained based on the second loss term, the third loss term, the fourth loss term, the fifth loss term, the sixth loss term, the seventh loss term, and the eighth loss term.

[0031] The multimodal medical image registration method, wherein the training process of the first registration module and the second registration module are the same, and the first registration module and the second registration module are jointly trained, and the training process of the first registration module and the second registration module specifically includes:

[0032] Inputting the first modality training image T1 and the first modality predicted image into the first registration module, and determining the first predicted affine image, the first predicted deformation field, and the first predicted registration image corresponding to the first modality predicted image by the first registration module, wherein the first modality predicted image is determined by the second modality training image T2 through the style transformation module;

[0033] Inputting the second modality training image T2 and the second modality predicted image into the second registration module, and determining the second predicted affine image, the second predicted deformation field, and the second predicted registration image corresponding to the second modality predicted image by the second registration module, wherein the second modality predicted image is determined by the first modality training image T1 through the style transformation module;

[0034] Based on the first modality training image T1, the second modality training image T2, the first affine prediction image, the second affine prediction image, the first registration prediction image, the second registration prediction image, the first predicted deformation field and the second predicted deformation field, the first registration module and the second registration module are trained respectively to obtain the trained first registration module and the second registration module.

[0035] The multimodal medical image registration method, wherein the style conversion module, the first registration module and the second registration module constitute a registration network, and the style conversion module, the first registration module and the second registration module are jointly trained.

[0036] A second aspect of an embodiment of the present application provides a multimodal medical image registration device, the device comprising:

[0037] a style conversion module, configured to input the first modality image and the second modality image to be registered into the style conversion module, and determine, through the style conversion module, a second modality conversion image corresponding to the first modality image and a first modality conversion image corresponding to the second modality image;

[0038] a first registration module, configured to input the first modality image and the first modality conversion image into a first registration module, determine a first deformation field through the first registration module, and determine a first registered image corresponding to the first modality image based on the first deformation field;

[0039] The second registration module is configured to input the second modality image and the second modality conversion image into the second registration module, determine a second deformation field through the second registration module, and determine a second registered image corresponding to the second modality image based on the second deformation field.

[0040] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the multimodal medical image registration methods described above.

[0041] A fourth aspect of an embodiment of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program executable by the processor;

[0042] The communication bus realizes the connection and communication between the processor and the memory;

[0043] When the processor executes the computer-readable program, the processor implements the steps in any of the above-described multimodal medical image registration methods.

[0044] Beneficial effects: Compared with the prior art, the present application provides a multimodal medical image registration method and related apparatus, the method comprising inputting a first modality image and a second modality image to be registered into a style conversion module, determining the second modality conversion image corresponding to the first modality image and the first modality conversion image corresponding to the second modality image through the style conversion module; inputting the first modality image and the first modality conversion image into a first registration module, determining a first deformation field through the first registration module and determining a first registration image corresponding to the first modality image based on the first deformation field; inputting the second modality image and the second modality conversion image into a second registration module, determining a second deformation field through the second registration module, and determining a second registration image corresponding to the second modality image based on the second deformation field. The present application implements bidirectional image style conversion of the first modality image and the second modality image through the style conversion module, and constrains the shape of the multimodal image through shape prior knowledge to improve the quality of the converted image and retain the structural information of the original image, thereby improving the registration accuracy of the multimodal image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0046] Figure 1 This is a grayscale difference comparison image between T1-weighted brain magnetic resonance images and T2-weighted brain magnetic resonance images.

[0047] Figure 2 This is a grayscale difference comparison diagram between TRUS images and MRI magnetic resonance prostate images.

[0048] Figure 3 This is a comparison chart of the structural differences between TRUS images and MRI magnetic resonance prostate images.

[0049] Figure 4 This is a flowchart of the multimodal medical image registration method provided in this application.

[0050] Figure 5 This is a flowchart of the principles of the training process of the style conversion module in the multimodal medical image registration method provided in this application.

[0051] Figure 6 This is a flowchart of the working principle of the first registration module in the multimodal medical image registration method provided in this application.

[0052] Figure 7 This is a principle flow chart of the training process of the joint training of the first registration module and the second registration module in the multimodal medical image registration method provided in this application.

[0053] Figure 8 This is a schematic diagram of the structural principle of the multimodal medical image registration device provided in this application.

[0054] Figure 9 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0055] This application provides a multimodal medical image registration method and related apparatus. To clarify the purpose, technical solutions, and effects of this application, the application is further described below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate this application and are not intended to limit it.

[0056] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0059] The inventors have discovered that, with the rapid development of deep learning in recent years, computer-aided diagnosis (CAD) technology has become increasingly mature. Medical image registration has become a research hotspot within CAD, with numerous applications in medical fields such as imaging diagnosis, surgical navigation, and radiotherapy planning.

[0060] Image registration primarily predicts spatial transformation parameters between images based on similarity metrics, establishing linear or nonlinear point-to-point position correspondences to achieve spatial consistency. With the rapid development of medical imaging equipment, medical imaging technology is trending towards multidimensionality, multimodality, and high resolution. For the same patient, multiple images with different anatomical information can be acquired, such as computed tomography (CT) and magnetic resonance imaging (MRI). Simultaneously, images with functional information, such as single-photon emission computed tomography (SPECT), can also be acquired. However, because each imaging technique has its own advantages and disadvantages, a single imaging technique is no longer sufficient for disease diagnosis. Therefore, image registration technology can be used to fuse multimodal images according to the physician's needs, enabling faster and more accurate visualization of lesions and other anatomical structures from various angles, making clinical diagnoses, surgical procedures, and radiotherapy planning more accurate and reliable. For example, the clinical applications of multimodal image registration include: targeted prostate puncture biopsy guided by MRI-TRUS fusion; expression of the structural morphology of the lesion area based on PET / CT image fusion; and three-dimensional reconstruction technology based on image registration.

[0061] The difficulty of multimodal image registration technology lies in the fact that different imaging methods of multimodal data lead to large differences in the grayscale and structural information of different modal images. Figure 1 As shown in , the same brain regions (such as gray matter, white matter and cerebrospinal fluid) have large grayscale differences in two different modal images (T1-weighted MRI / T2-weighted MRI); Figure 2 and Figure 3As shown, transrectal ultrasound (TRUS) images of the prostate have low resolution and high noise, while magnetic resonance imaging (MRI) images clearly display the prostate's tissue structure. Furthermore, TRUS and MRI also suffer from significant and complex tissue deformation caused by the biopsy needle. Due to these grayscale and structural differences, conventional similarity metrics struggle to effectively extract key information that constrains registration. Currently, multimodal registration methods can be categorized into three types: the first type employs novel similarity metrics to extract more effective information for optimized registration; the second type employs machine learning to transform information from different modalities into features of the same dimensionality; and the third type employs style transfer to achieve homomodality before performing single-modality image registration. Traditional multimodal image registration methods primarily rely on a similarity metric as the optimization objective, achieving optimal image matching through iterative optimization. However, these methods require significant time and computational resources for high-dimensional medical images. Furthermore, similarity metrics are not applicable to images from multiple modalities, resulting in poor generalization performance for existing multimodal image registration methods. Although existing multimodal image registration methods based on deep learning have effectively solved the problem of long registration time, existing multimodal image registration methods based on deep learning generally have the problem of being unable to retain image detail information, and thus cannot guarantee the image effect of the registered images.

[0062] In order to solve the above problems, in an embodiment of the present application, the first modal image and the second modal image to be registered are input into a style conversion module, and the style conversion module determines the second modal conversion image corresponding to the first modal image and the first modal conversion image corresponding to the second modal image; the first modal image and the first modal conversion image are input into a first registration module, and the first registration module determines the first deformation field and determines the first registration image corresponding to the first modal image based on the first deformation field; the second modal image and the second modal conversion image are input into a second registration module, and the second registration module determines the second deformation field and determines the second registration image corresponding to the second modal image based on the second deformation field. The present application realizes bidirectional image style conversion of the first modal image and the second modal image through the style conversion module, and constrains the shape of the multimodal image through shape prior knowledge to improve the quality of the converted image and retain the structural information of the original image, which can improve the registration accuracy of the multimodal image.

[0063] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0064] This embodiment provides a multimodal medical image registration method, such as Figure 4 As shown, the method includes:

[0065] S10: Inputting the first modality image and the second modality image to be registered into a style conversion module, and determining, by the style conversion module, a second modality conversion image corresponding to the first modality image and a first modality conversion image corresponding to the second modality image;

[0066] S20: Input the first modal image and the first modal conversion image into a first registration module, determine a first deformation field through the first registration module, and determine a first registered image corresponding to the first modal image based on the first deformation field;

[0067] S30: Input the second modality image and the second modality conversion image into a second registration module, determine a second deformation field through the second registration module, and determine a second registered image corresponding to the second modality image based on the second deformation field.

[0068] This embodiment realizes bidirectional image style conversion of the first modality image and the second modality image through the style conversion module, and constrains the shape of the multimodal image through shape prior knowledge to improve the quality of the converted image and retain the structural information of the original image, so as to improve the registration accuracy of the multimodal image. At the same time, the first registration module and the second registration module both use a single-modality registration network and a loss function of similarity measurement to align the converted image, and add adversarial loss to improve the registration accuracy of the multimodal image. In addition, the style conversion module, the first registration module and the second registration module can form a registration network, and the registration network is jointly trained by multi-task collaborative optimization, so that end-to-end multimodal registration can be achieved through the registration network.

[0069] The following describes in detail the various implementation steps of the multimodal medical image registration method provided in this embodiment.

[0070] In step S10, the first modality image and the second modality image are medical images acquired from the same patient using different imaging modalities. For example, the first modality image is a TI-weighted brain magnetic resonance image, and the second modality image is a T2-weighted brain magnetic resonance image. Alternatively, the first modality image is a prostate magnetic resonance image, and the second modality image is a prostate ultrasound image. Furthermore, before inputting the first modality image and the second modality image into the style conversion module, the first modality image and the second modality image may be preprocessed. The preprocessing may include image scaling and image normalization, so that the image scales of the preprocessed first modality image and the preprocessed second modality image are the same as the image scale of the input item of the style conversion module, and the grayscale values ​​are between 0 and 1.

[0071] The style transfer module is used to convert the image style of the input image. It has a bidirectional style transfer mechanism. When a first modality image and a second modality image are input, the style of the first modality image is converted to the style of the second modality image, and vice versa. It is understood that the second modality-converted image corresponding to the first modality image adopts the style of the second modality image, and the first modality-converted image corresponding to the second modality image adopts the style of the first modality image.

[0072] In one implementation of this embodiment, the style conversion module includes a first generation unit and a second generation unit. Inputting the first modality image and the second modality image to be registered into the style conversion module, and determining, by the style conversion module, the second modality conversion image corresponding to the first modality image and the first modality conversion image corresponding to the second modality image specifically includes:

[0073] Inputting the first modality image to be registered into a first generating unit, and determining the second modality conversion image corresponding to the first modality image by the first generating unit;

[0074] The second modality image to be registered is input into the second generation unit, and the first modality conversion image corresponding to the second modality image is determined by the second generation unit.

[0075] Specifically, the first generation unit is configured to convert the image style of the first modal image into the image style of the second modal image, and the second generation unit is configured to convert the image style of the second modal image into the image style of the first modal image. That is, when the first modal image and the second modal image are input into the style conversion module, the first modal image is input into the first generation unit, and the second modal image is input into the second generation unit. The first generation unit converts the image style of the first modal image into the image style of the second modal image to obtain the second modal conversion image; and the second generation unit converts the image style of the second modal image into the image style of the first modal image to obtain the first modal conversion image.

[0076] In one implementation of this embodiment, the style conversion module is obtained through training based on CycleGAN (Cycle-Consistent Generative Adversarial Networks, the full name is cycle-consistent generative adversarial networks), wherein CycleGAN includes two generators and two discriminators, which are respectively denoted as the first generator G1, the second generator G2, the first discriminator D1 and the second discriminator D2; wherein the first generator G1 is used to convert the image style into the image style of the second modality image, and the second generator G2 is used to convert the image style into the image style of the first modality image. It can be understood that by training CycleGAN, when the CycleGAN training is completed, the first generator G1 in CycleGAN is used as the first generation unit, and the second generator G2 is used as the second generation unit to obtain the style conversion module. Based on this, as Figure 5 As shown, the training process of the style conversion module specifically includes:

[0077] Input the first modality training image T1 into the first generator G1 and the second generator G2 respectively, generate a first predicted image sT2 through the first generator G1, generate a second predicted image iT1 through the second generator G2, and input the first predicted image sT2 into the second generator G2, and generate a third predicted image rT1 through the second generator G2;

[0078] Input the second modality training image T2 into the first generator G1 and the second generator G2 respectively, generate a fourth predicted image iT2 through the first generator G1, generate a fifth predicted image sT1 through the second generator G2, and input the fifth predicted image sT1 into the first generator G1, and generate a sixth predicted image rT2 through the first generator G1;

[0079] Input the first modality training image T1 and the fifth predicted image sT1 into the first discriminator D1, and determine the first true probability corresponding to the first modality training image T1 and the second true probability corresponding to the fifth predicted image sT1 through the first discriminator D1;

[0080] Input the second modality training image T2 and the first predicted image sT2 into the second discriminator D2, and determine through the second discriminator D2 whether the second modality training image T2 corresponds to the third true probability and the first predicted image sT2 corresponds to the fourth true probability;

[0081] The first generator G1, the second generator G2, the first discriminator D1 and the second discriminator D2 are trained based on the first modality training image T1, the second modality training image T2, the first predicted image sT2, the second predicted image iT1, the third predicted image rT1, the fourth predicted image iT2, the fifth predicted image sT1, the sixth predicted image rT2, the first true probability, the second true probability, the third true probability and the fourth true probability, and the trained first generator G1 is used as the first generation unit and the second generator G2 is used as the second generation unit to obtain a style transfer module.

[0082] Specifically, the training process includes a process of converting a first modality training image into an image having second modality characteristics, and a process of converting a second modality training image into an image having first modality characteristics, wherein the process of converting the first modality training image into an image having second modality characteristics can be specifically:

[0083] First, the first modality training image T1 is synthesized into a first predicted image sT2 with the image style of the second modality training image through the first generator G1, wherein the first predicted image sT2 has the modality features of the second modality training image T2. At this time, the adversarial loss between the first predicted image sT2 and the second modality training image T2 can be used as the loss term of the first generator. adv It is determined by inputting the first predicted image sT2 and the second modal training image T2 into the second discriminator D2 respectively. That is, after obtaining the first predicted image sT2, the first predicted image sT2 and the second modal training image T2 are input into the second discriminator D2 respectively, and the second discriminator D2 determines whether the second modal training image T2 corresponds to the third true probability and the first predicted image sT2 corresponds to the fourth true probability.

[0084] Secondly, after obtaining the first predicted image sT2, the first predicted image sT2 is input into the second generator G2, and the first predicted image sT2 is reconstructed back into the third predicted image rT1 having the modal features of the first modal training image T1 through the second generator G2, and the cycle consistency loss is determined based on the third predicted image rT1 and the first modal training image T1 for supervision to ensure the stability of the image structure of the first modal sequence image converted into the first predicted image sT2 having the modal features of the second modal training image T2.

[0085] Finally, to ensure correlation between conversions between different modal domains, the first modality training image T1 is fed into the second generator G2, which generates a second predicted image iT1 of its own modality. The second predicted image iT1 and the first modality training image T1 are then used to determine an identity loss for supervision. Furthermore, to ensure that the converted modality image retains the image details of the original image, a shape consistency loss between the first modality training image T1 and the first predicted image sT2 is also incorporated into the CycleGAN training.

[0086] The process of converting the second modality training image into the first modality training image is basically the same as the process of converting the first modality training image into the second modality training image. The difference between the two is that the second modality training image T2 is input into the first generator G1 and the second generator G2 respectively, and the fourth prediction image iT2 is generated by the first generator G1, and the fifth prediction image sT1 is generated by the second generator G2, and the fifth prediction image sT1 is input into the first generator G1, and the sixth prediction image rT2 is generated by the first generator G1. Among them, the role of each prediction image and the determination of the loss term are the same as those in the conversion of the first modality training image into the second modality training image, and will not be repeated here.

[0087] In one implementation of this embodiment, the training of the first generator G1, the second generator G2, the first discriminator D1, and the second discriminator D2 based on the first modality training image T1, the second modality training image T2, the first predicted image sT2, the second predicted image iT1, the third predicted image rT1, the fourth predicted image iT2, the fifth predicted image sT1, the sixth predicted image rT2, the first true probability, the second true probability, the third true probability, and the fourth true probability specifically includes:

[0088] Determine a first loss term based on the first true probability and the second true probability and train a first discriminator D1 based on the first loss term;

[0089] Determining a second loss term based on the third true probability and the fourth true probability and training a second discriminator D2 based on the second loss term;

[0090] determining a third loss term based on the third predicted image rT1 and the first modality training image T1, determining a fourth loss term based on the sixth predicted image rT2 and the second modality training image T2, determining a fifth loss term based on the fourth predicted image iT2 and the second modality training image T2, determining a sixth loss term based on the second predicted image iT1 and the first modality training image T1, determining a seventh loss term based on the sixth predicted image rT2 and the first predicted image sT2, and determining an eighth loss term based on the third predicted image rT1 and the fifth predicted image sT1;

[0091] Train the first generator G1 based on the first loss term, the third loss term, the fourth loss term, the fifth loss term, the sixth loss term, the seventh loss term, and the eighth loss term;

[0092] The second generator G2 is trained based on the second loss term, the third loss term, the fourth loss term, the fifth loss term, the sixth loss term, the seventh loss term, and the eighth loss term.

[0093] Specifically, the first loss term is the adversarial loss term between the first modality training image T1 and the fifth predicted image sT1, the second loss term is the adversarial loss term between the second modality training image T2 and the first predicted image sT2, the third loss term and the fourth loss term are used to determine the cycle consistency loss, the fifth loss term and the sixth loss term are used to determine the identity loss, and the seventh loss term and the eighth loss term are used to determine the shape consistency loss, wherein the cycle consistency loss is equal to the sum of the third loss term and the fourth loss term; the identity loss is equal to the sum of the fifth loss term and the sixth loss term, and the shape consistency loss is equal to the sum of the seventh loss term and the eighth loss term.

[0094] In one implementation of this embodiment, the first loss term L adv (G2, D1, T1, T2), the second loss term L adv The calculation formulas for (G1, D2, T1, T2), the third loss item, the fourth loss item, the fifth loss item, the sixth loss item, the seventh loss item, and the eighth loss item are:

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] Correspondingly, the cycle consistency loss L cycle , identity loss L idt and shape consistency loss L cons The calculation formulas can be respectively:

[0104]

[0105]

[0106] L cons (G1,G2,sT1,sT2)=||G1(G2(T2))-G2(T2)||1+||G2(G1(T1))-G1(T1)||1

[0107] Among them, T1 represents the first modality training image, T2 represents the second modality training, G1 represents the first generator, G2 represents the second generator, D1 represents the first discriminator D1, D2 represents the second discriminator D2, p data represents the distribution of data, and E represents the expectation.

[0108] Therefore, the loss functions of the first discriminator D1, the second discriminator D2, the first generator G1 and the second generator G2 can be respectively:

[0109] The loss function of the second discriminator D2 is: L adv (G1,D2,T1,T2)

[0110] The loss function of the first discriminator D1 is: L adv (G2,D1,T1,T2)

[0111] The loss function of the first generator G1 is:

[0112] L adv (G1,D2,T1,T2)+L cycle (G1,G2)+L idt (G1,G2)+L cons (G1,G2,sT1,sT2)

[0113] The loss function of the second generator G2 is:

[0114] L adv (G2,D1,T1,T2)+L cycle (G1,G2)+L idt (G1,G2)+L cons (G1,G2,sT1,sT2).

[0115] The above completes the description of step S10. Now, step S20 and step S30 will be described.

[0116] In the steps S20 and S30, the module structure of the first registration module is the same as the module structure of the second registration module. Here, the first registration module is taken as an example for description. Figure 6As shown, the first registration module includes a rigid transformation unit, an affine unit, a non-rigid transformation unit and a deformation unit, wherein the rigid transformation unit, the affine unit, the non-rigid transformation unit and the deformation unit are cascaded in sequence, the affine unit is jump-connected with the deformation unit, the input item of the affine unit includes the output item of the rigid transformation unit and the motion image, and the input item of the deformation unit includes the input item of the non-rigid transformation unit and the fixed image.

[0117] Based on this, in one implementation of this embodiment, inputting the first modality image and the first modality conversion image into a first registration module, determining a first deformation field by the first registration module, and determining a first registered image corresponding to the first modality image based on the first deformation field specifically includes:

[0118] Inputting the first modality image and the first modality conversion image into a rigid body transformation unit, and determining an affine matrix through the rigid body transformation unit;

[0119] Inputting the affine matrix and the first modal conversion image into an affine unit, and outputting an affine image through the affine unit;

[0120] Inputting the affine image and the first modality image into a non-rigid transformation unit, and determining a first deformation field through the non-rigid transformation unit;

[0121] The first deformation field and the affine image are input into the deformation unit, and the first registration image is determined by the deformation unit.

[0122] Specifically, the first registration module and the second registration module operate in the same manner, wherein the first registration module is used to register the first modality image and the first modality transition image in the modality of the first modality image, and the second registration module is used to register the second modality image and the second modality transition image in the modality of the second modality image. Furthermore, when the first registration module registers the first modality image and the first modality transition image, the first modality image is a moving image and the first modality transition image is a fixed image. When the second registration module registers the second modality image and the second modality transition image, the second modality image is a moving image and the second modality transition image is a fixed image.

[0123] In the implementation of this embodiment, the registration process of the first registration module is described as an example. The first registration module is used to register the first modality image and the first modality conversion image. Specifically, the first modality image and the first modality conversion image are input into the rigid transformation unit in the first registration module. The rigid transformation unit generates an affine matrix, wherein the affine matrix can include 12 deformation parameters, including rotation factors, scaling factors, and translation factors. The affine matrix output by the rigid transformation unit and the first modality image are respectively input into the affine unit. The affine unit applies the affine matrix to the first modality image to deform the first modality image to obtain an affine image. The affine image is then input into the non-rigid transformation unit. The non-rigid transformation unit determines a deformation field based on the input affine image and the input first modality conversion image. The deformation field is then input into the deformation unit. The deformation unit performs spatial edge deformation on the affine image based on the deformation field to obtain a first registered image. This embodiment utilizes a registration module that combines rigid and non-rigid deformations, eliminating the need for affine alignment preprocessing of the input image. This allows the registration module to directly input the first modality image and the first modality conversion image, and then directly input the registered first registered image, thereby achieving an end-to-end registration process. Furthermore, the rigid transformation unit and the non-rigid transformation unit can utilize existing rigid transformation networks and non-rigid transformation networks used for image registration, which will not be described in detail here.

[0124] In one implementation of this embodiment, Figure 7 As shown, the training process of the first registration module and the second registration module are the same, and the first registration module and the second registration module are jointly trained. The training process of the first registration module and the second registration module specifically includes:

[0125] Inputting the first modality training image T1 and the first modality predicted image into the first registration module, and determining the first predicted affine image, the first predicted deformation field, and the first predicted registration image corresponding to the first modality predicted image by the first registration module, wherein the first modality predicted image is determined by the second modality training image T2 through the style transformation module;

[0126] Inputting the second modality training image T2 and the second modality predicted image into the second registration module, and determining the second predicted affine image, the second predicted deformation field, and the second predicted registration image corresponding to the second modality predicted image by the second registration module, wherein the second modality predicted image is determined by the first modality training image T1 through the style transformation module;

[0127] Based on the first modality training image T1, the second modality training image T2, the first affine prediction image, the second affine prediction image, the first registration prediction image, the second registration prediction image, the first predicted deformation field and the second predicted deformation field, the first registration module and the second registration module are trained respectively to obtain the trained first registration module and the second registration module.

[0128] Specifically, the process of determining the first predicted affine image, the first predicted deformation field, and the first predicted registration image, as well as the process of determining the second predicted affine image, the second predicted deformation field, and the second predicted registration image are the same as the process of determining the above-mentioned affine image, deformation field, and first registration image, and will not be repeated here. Here, after obtaining the first modality training image, the second modality training image, the first affine predicted image, the second affine predicted image, the first registration predicted image, the second registration predicted image, the first predicted deformation field, and the second predicted deformation field to train the first registration module, how to train the first registration module is explained.

[0129] After acquiring the first modality training image, the second modality training image, the first affine prediction image, the second affine prediction image, the first registration prediction image, the second registration prediction image, the first predicted deformation field and the second predicted deformation field to train the first registration module, the local correlation function, the deformation field loss function and the deformation consistency loss function are determined respectively, and then the loss function for training the first registration module and the second registration module is determined based on the local correlation function, the deformation field loss function and the deformation consistency loss function, and the module parameters of the first registration module and the second registration module are optimized based on the loss function to obtain the trained first registration module and the second registration module, wherein the local correlation function is used to measure the similarity between the two images, the deformation field loss function is used to constrain the deformation field to make it smooth, and the deformation consistency loss function is used to constrain the reversibility of the deformation field in the bidirectional registration.

[0130] In one implementation of this embodiment, the calculation formula of the loss function may be:

[0131]

[0132] Wherein, T1 represents the first modality training image, sT1 represents the second modality prediction image, T2 represents the second modality training image, sT2 represents the first modality prediction image, and A T1 represents the first affine predicted image, W T1 Represents the first registered predicted image, A T2 represents the second affine predicted image, W T2 represents the second registered predicted image, represents the first predicted deformation field, represents the second predicted deformation field, Lncc (·,·) represents the local cross-correlation function, L smooth (·) represents the deformation field loss function, which can use the L1 norm of the deformation field. represents the deformation consistency loss function, The calculation formula can be ρ(x)=(x 2 + ∈ 2 ) γ It is a robust generalized Charbonnier penalty function, and the values ​​of ∈ and γ can be ∈ = 0.001, γ = 0.45.

[0133] In one implementation of this embodiment, the style conversion module, the first registration module and the second registration module constitute a registration network, and the style conversion module, the first registration module and the second registration module are jointly trained. Of course, in actual applications, the style conversion module can also be trained first, and after obtaining the trained style conversion module, the first registration module and the second registration module are jointly trained to obtain the registration network. In addition, the registration network can determine the number of registration modules it includes based on the registration task. For example, when bidirectional registration is required, the first registration module and the second registration module can be included. When only unidirectional registration is required, only the first registration module or the second registration module can be included.

[0134] In summary, this embodiment provides a multimodal medical image registration method, wherein the method includes inputting the first modality image and the second modality image to be registered into a style conversion module, determining the second modality conversion image corresponding to the first modality image and the first modality conversion image corresponding to the second modality image through the style conversion module; inputting the first modality image and the first modality conversion image into the first registration module, determining the first deformation field through the first registration module and determining the first registration image corresponding to the first modality image based on the first deformation field; inputting the second modality image and the second modality conversion image into the second registration module, determining the second deformation field through the second registration module, and determining the second registration image corresponding to the second modality image based on the second deformation field. The present application realizes bidirectional image style conversion of the first modality image and the second modality image through the style conversion module, and constrains the shape of the multimodal image through shape prior knowledge to improve the quality of the converted image and retain the structural information of the original image, thereby improving the registration accuracy of the multimodal image.

[0135] Based on the above multimodal medical image registration method, this embodiment provides a multimodal medical image registration device, such as Figure 8 As shown, the device includes:

[0136] A style conversion module 100 is configured to input a first modality image and a second modality image to be registered into a style conversion module, and determine, through the style conversion module, a second modality conversion image corresponding to the first modality image and a first modality conversion image corresponding to the second modality image;

[0137] A first registration module 200 is configured to input the first modality image and the first modality conversion image into a first registration module, determine a first deformation field through the first registration module, and determine a first registered image corresponding to the first modality image based on the first deformation field;

[0138] The second registration module 300 is configured to input the second modality image and the second modality conversion image into a second registration module, determine a second deformation field through the second registration module, and determine a second registered image corresponding to the second modality image based on the second deformation field.

[0139] Based on the above-mentioned multimodal medical image registration method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the multimodal medical image registration method as described in the above-mentioned embodiment.

[0140] Based on the above multimodal medical image registration method, the present application also provides a terminal device, such as Figure 9 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0141] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0142] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0143] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0144] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal medical image registration method, characterized in that: The registration method comprises: Inputting the first modality image and the second modality image to be registered into a style conversion module, and determining, by the style conversion module, a second modality conversion image corresponding to the first modality image and a first modality conversion image corresponding to the second modality image; Inputting the first modality image and the first modality conversion image into a first registration module, determining a first deformation field through the first registration module, and determining a first registered image corresponding to the first modality image based on the first deformation field, the first registration module including a rigid transformation unit, an affine unit, a non-rigid transformation unit, and a deformation unit; The second modality image and the second modality conversion image are input into a second registration module, a second deformation field is determined by the second registration module, and a second registered image corresponding to the second modality image is determined based on the second deformation field. The module structure of the second registration module is the same as the module structure of the first registration module.

2. The multimodal medical image registration method according to claim 1, characterized in that: Inputting the first modality image and the first modality conversion image into a first registration module, determining a first deformation field by the first registration module, and determining a first registered image corresponding to the first modality image based on the first deformation field specifically includes: Inputting the first modality image and the first modality conversion image into a rigid body transformation unit, and determining an affine matrix through the rigid body transformation unit; Inputting the affine matrix and the first modal conversion image into an affine unit, and outputting an affine image through the affine unit; Inputting the affine image and the first modality image into a non-rigid transformation unit, and determining a first deformation field through the non-rigid transformation unit; The first deformation field and the affine image are input into the deformation unit, and the first registration image is determined by the deformation unit.

3. The multimodal medical image registration method according to claim 1, characterized in that: The style conversion module includes a first generation unit and a second generation unit. Inputting the first modality image and the second modality image to be registered into the style conversion module, and determining the second modality conversion image corresponding to the first modality image and the first modality conversion image corresponding to the second modality image through the style conversion module specifically includes: Inputting the first modality image to be registered into a first generating unit, and determining the second modality conversion image corresponding to the first modality image by the first generating unit; The second modality image to be registered is input into the second generation unit, and the first modality conversion image corresponding to the second modality image is determined by the second generation unit.

4. The multimodal medical image registration method according to claim 1 or 3, characterized in that: The training process of the style conversion module specifically includes: The first modality training image T1 is input into the first generator G1 and the second generator G2 respectively, the first generator G1 generates a first predicted image sT2, the second generator G2 generates a second predicted image iT1, and the first predicted image sT2 is input into the second generator G2, and the second generator G2 generates a third predicted image rT1; Input the second modality training image T2 into the first generator G1 and the second generator G2 respectively, generate a fourth predicted image iT2 through the first generator G1, generate a fifth predicted image sT1 through the second generator G2, and input the fifth predicted image sT1 into the first generator G1, and generate a sixth predicted image rT2 through the first generator G1; Input the first modality training image T1 and the fifth predicted image sT1 into the first discriminator D1, and determine the first true probability corresponding to the first modality training image T1 and the second true probability corresponding to the fifth predicted image sT1 through the first discriminator D1; Input the second modality training image T2 and the first predicted image sT2 into the second discriminator D2, and determine through the second discriminator D2 whether the second modality training image T2 corresponds to the third true probability and the first predicted image sT2 corresponds to the fourth true probability; Based on the first modality training image T1, the second modality training image T2, the first predicted image sT2, the second predicted image iT1, the third predicted image rT1, the fourth predicted image iT2, the fifth predicted image sT1, the sixth predicted image rT2, the first true probability, the second true probability, the third true probability and the fourth true probability, the first generator G1, the second generator G2, the first discriminator D1 and the second discriminator D2 are trained, and the trained first generator G1 is used as the first generation unit and the second generator G2 is used as the second generation unit to obtain a style transfer module.

5. The multimodal medical image registration method according to claim 4, characterized in that: The training of the first generator G1, the second generator G2, the first discriminator D1, and the second discriminator D2 based on the first modality training image T1, the second modality training image T2, the first predicted image sT2, the second predicted image iT1, the third predicted image rT1, the fourth predicted image iT2, the fifth predicted image sT1, the sixth predicted image rT2, the first true probability, the second true probability, the third true probability, and the fourth true probability specifically includes: Determine a first loss term based on the first true probability and the second true probability and train a first discriminator D1 based on the first loss term; Determining a second loss term based on the third true probability and the fourth true probability and training a second discriminator D2 based on the second loss term; determining a third loss term based on the third predicted image rT1 and the first modality training image T1, determining a fourth loss term based on the sixth predicted image rT2 and the second modality training image T2, determining a fifth loss term based on the fourth predicted image iT2 and the second modality training image T2, determining a sixth loss term based on the second predicted image iT1 and the first modality training image T1, determining a seventh loss term based on the sixth predicted image rT2 and the first predicted image sT2, and determining an eighth loss term based on the third predicted image rT1 and the fifth predicted image sT1; Train the first generator G1 based on the first loss term, the third loss term, the fourth loss term, the fifth loss term, the sixth loss term, the seventh loss term, and the eighth loss term; The second generator G2 is trained based on the second loss term, the third loss term, the fourth loss term, the fifth loss term, the sixth loss term, the seventh loss term, and the eighth loss term.

6. The multimodal medical image registration method according to claim 1, characterized in that: The training process of the first registration module and the second registration module is the same, and the first registration module and the second registration module are jointly trained. The training process of the first registration module and the second registration module specifically includes: Inputting the first modality training image T1 and the first modality predicted image into the first registration module, and determining the first predicted affine image, the first predicted deformation field, and the first predicted registration image corresponding to the first modality predicted image by the first registration module, wherein the first modality predicted image is determined by the second modality training image T2 through the style transformation module; Inputting the second modality training image T2 and the second modality predicted image into the second registration module, and determining the second predicted affine image, the second predicted deformation field, and the second predicted registration image corresponding to the second modality predicted image by the second registration module, wherein the second modality predicted image is determined by the first modality training image T1 through the style transformation module; Based on the first modality training image T1, the second modality training image T2, the first affine prediction image, the second affine prediction image, the first registration prediction image, the second registration prediction image, the first predicted deformation field and the second predicted deformation field, the first registration module and the second registration module are trained respectively to obtain the trained first registration module and the second registration module.

7. The multimodal medical image registration method according to claim 1, characterized in that: The style conversion module, the first registration module and the second registration module constitute a registration network, and the style conversion module, the first registration module and the second registration module are jointly trained.

8. A multimodal medical image registration device, characterized in that: The device comprises: a style conversion module, configured to input the first modality image and the second modality image to be registered into the style conversion module, and determine, through the style conversion module, a second modality conversion image corresponding to the first modality image and a first modality conversion image corresponding to the second modality image; a first registration module, configured to input the first modality image and the first modality conversion image into a first registration module, determine a first deformation field through the first registration module, and determine a first registered image corresponding to the first modality image based on the first deformation field, the first registration module including a rigid transformation unit, an affine unit, a non-rigid transformation unit, and a deformation unit; A second registration module is configured to input the second modality image and the second modality conversion image into a second registration module, determine a second deformation field through the second registration module, and determine a second registered image corresponding to the second modality image based on the second deformation field. The module structure of the second registration module is the same as that of the first registration module.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the multimodal medical image registration method according to any one of claims 1 to 7.

10. A terminal device, characterized in that: include: processor, memory, and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the processor implements the steps in the multimodal medical image registration method according to any one of claims 1 to 7.

Images