A cross-modal medical image registration method based on latent space diffusion model

Abstract

The present invention discloses a cross-modal medical image registration method based on a latent space diffusion model, which relates to the technical field of medical image processing and solves the technical problem that existing registration methods are not suitable for multi-modal medical image registration. The method includes: obtaining a first modality original medical image and a second modality original medical image, performing pre-processing operations to obtain a floating image and a target image; constructing a cross-modal conversion network, using a pre-trained autoencoder to compress the target image into a latent space representation and input it into a diffusion model, and training it through forward denoising and backward denoising; inputting the floating image into the diffusion model of the cross-modal conversion network to generate a second modality conversion image; inputting the floating image and the target image into the registration network, and obtaining respective registered images for the floating image and the second modality conversion image; training the registration network to obtain an image after the floating image is registered. The present invention has a wide range of applications and is suitable for registration between medical images of multiple modalities.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a cross-modal medical image registration method based on a latent space diffusion model. Background Art

[0002] With the advancement of imaging technology, the improvement of imaging equipment performance and the continuous innovation and breakthroughs in computer technology, medical imaging technology has been widely used in clinical and medical informationization and has become one of the essential means of clinical diagnosis.

[0003] When using medical imaging to assist in diagnosis, it is often necessary to fuse medical imaging information from different modalities to improve the accuracy of clinical diagnosis. For example, X-ray imaging technology (Xray) is often used to examine diseases such as pulmonary tuberculosis and fractures; digital subtraction angiography (DSA) is used to obtain clear vascular images to display and reconstruct blood vessels; and magnetic resonance imaging (MR) is often used to diagnose cerebrovascular diseases, intracranial tumors, etc. It has the characteristics of multiple sequences, and different sequences can be used to detect different regions and lesions. These different imaging technologies can provide more detailed views in different tissue structure scenarios and are crucial for understanding complex physiological processes. However, due to the different sources of different modal images, they often have spatial deviations, so the application of image registration technology is particularly important.

[0004] Deformable image registration is an image registration technique that establishes nonlinear correspondences between moving and target image pairs. Fast, accurate, and realistic image registration algorithms are crucial for improving the efficiency and accuracy of clinical practice. By observing dynamic changes such as lesions, doctors can more comprehensively design treatment plans for their patients. However, existing multimodal medical image registration techniques greatly increase the complexity of the registration process due to the inherent heterogeneity between multimodal images and the differences in image characteristics caused by different imaging mechanisms. Furthermore, existing registration methods are often limited by the lack of effective similarity metrics, making them unsuitable for image registration in multimodal environments.

[0005] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0006] In multimodal medical image registration, the inherent heterogeneity between multimodal images and the differences in image characteristics caused by different imaging mechanisms greatly increase the complexity of the registration process. Existing registration methods are limited by the lack of effective similarity measurement and are not suitable for multimodal medical image registration. Summary of the Invention

[0007] The present invention aims to provide a cross-modal medical image registration method based on a latent space diffusion model to address the technical problem that existing registration methods are limited by the lack of an effective similarity metric and are therefore unsuitable for multimodal medical image registration. The various technical effects achieved by the preferred technical solutions provided by the present invention are detailed below.

[0008] To achieve the above objectives, the present invention provides the following technical solutions:

[0009] The present invention provides a cross-modal medical image registration method based on a latent space diffusion model, comprising the following steps: S100: obtaining a first modality original medical image and a second modality original medical image, slicing and resizing them respectively to obtain a first modality slice image and a second modality slice image, performing preprocessing operations on the first modality slice image and the second modality slice image to obtain a floating image and a target image, thereby forming a floating image and a target image registration image pair; S200: constructing a cross-modal conversion network, using a pre-trained autoencoder to compress the target image into a latent space representation and input it into a diffusion model, and training it through forward denoising and backward denoising; S300: the floating image is compressed by the autoencoder and input into the diffusion model of the cross-modal conversion network to generate a first modality slice image and a target image. Two-modal conversion image; S400: constructing an image registration network, inputting the floating image and the target image into the registration network, outputting the registration deformation field, and distorting the floating image and the second-modal conversion image based on the spatial transformation network and the registration deformation field to obtain their respective registered images; S500: calculating the reconstruction loss between the image after the floating image registration and the target image, the similarity loss between the image after the second-modal conversion image registration and the target image, and the smoothness loss of the registration deformation field, training the registration network based on minimizing the reconstruction loss, similarity loss, and smoothness loss, and repeating the steps S200 to S500; S600: the floating image and the target image execute step S400 again to obtain the image after the floating image registration.

[0010] Preferably, in step S100, the first modality slice image and the second modality slice image are obtained by slicing each volume of the first modality original medical image and the second modality original medical image respectively and adjusting the size to 256*256 pixels; the preprocessing operation includes affine alignment and normalization.

[0011] Preferably, in step S200, the cross-modal conversion network includes an autoencoder of a perceptual image compression module and a diffusion model of a latent variable space; the autoencoder includes an encoder and a decoder, and is trained using the target image after preprocessing operations.

[0012] Preferably, in the step S200, the autoencoder is pre-trained based on perceptual loss and adversarial loss, and the pre-training loss function of the autoencoder is The expression is: ,in, is the perceptual loss, To combat losses; ,in, is the input image, is the reconstructed image, Represents a specific layer of the input image The extracted feature vectors, Represents a specific layer of the reconstructed image Extracted feature vectors.

[0013] Preferably, in the step S200, during the forward noisy process, the diffusion model gradually adds noise to the latent space data of the input second modality medical image, uses the result of the gradual noisy process to predict noise, calculate the loss function, and perform back propagation using the loss function until the diffusion model training is completed; the loss function of the diffusion model is: ,in Latent space representation of medical images of the second modality as input , conditional input ,noise and diffusion time The joint expectation of is a uniform sampling on the time series {1,2,...,T}, is the noise estimated by the diffusion model, Enter the condition The embedding representation of The representation of the medical image of the second modality after adding noise in the latent space is obtained.

[0014] Preferably, in the step S200, in the reverse denoising, the medical image of the first modality is used as a condition to form a UNet network, and the latent space data of the medical image of the second modality predicted under the input condition of the medical image of the first modality is decoded into the second modality conversion image in the pixel domain by the decoder. .

[0015] Preferably, in step S300, the target image data of the second modality is compressed into a latent space representation using the autoencoder to generate latent space variables of the floating image; the latent space variables of the floating image are diffused by a diffusion model to generate latent space variables of the second modality; the latent space variables of the second modality are decoded into pixel space by a decoder to generate an image of the second modality, so that the floating image is converted from the first modality to the second modality.

[0016] Preferably, in step S400, the image registration network adopts a UNet network structure including an encoder and a decoder; the floating image and the target image are input into the registration network, and a registration deformation field is output; the floating image and the second modal conversion image are distorted and deformed respectively using the spatial transformation network and the registration deformation field to obtain respective registered images.

[0017] Preferably, in step S500, the loss between the registered image and the target image is calculated, including the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after conversion image registration and the target image, and the smoothness loss of the deformation field. The total loss function of the image registration process is: ,in 、 represents the weight parameter, represents the reconstruction loss between the image after floating image registration and the target image, represents the pixel index in the image, represents the total number of all pixels in the image, 、 Represent the floating image and target image after registration, represents the similarity loss between the image after the second modality conversion image registration and the target image; represents the smoothness loss of the deformation field, where is the spatial domain of the image, is the deformation field, is the gradient of the deformation field, represents a point in the spatial domain Ω of the image.

[0018] Preferably, in step S600, the floating image and the obtained optimal deformation field are input into a spatial transformation network to obtain a registered image.

[0019] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects:

[0020] The present invention uses a diffusion model to achieve conversion between image modalities, which not only retains the excellent image generation quality and medical image detailed information of the diffusion model, but also uses unimodal attitude metrics to reduce the differences between modalities; the latent space diffusion model uses the perceptual compression of the autoencoder to perform a diffusion process in the latent space to achieve cross-modal conversion of images and improve the computational efficiency during the modality conversion process; the cross-modal registration method of the present invention has a wide range of applications and is suitable for registration between multiple modalities, including registration between MRI and CT, multi-sequence registration of MRI, and registration of PET to CT / MRI. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work. In the drawings:

[0022] Figure 1 is a flow chart of a cross-modal medical image registration method based on a latent space diffusion model according to an embodiment of the present invention;

[0023] Figure 2 1 is a schematic diagram of a registration process in a cross-modal medical image registration method based on a latent space diffusion model according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of the overall network structure in the cross-modal medical image registration method based on the latent space diffusion model according to an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of the modality conversion network structure in the cross-modal medical image registration method based on the latent space diffusion model in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, in which various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods and devices that are consistent with some aspects of the present disclosure as detailed in the appended claims, and other embodiments may also be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and essence of the present invention.

[0027] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse" and the like indicate the orientation or positional relationship based on the figures, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second" and the like are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0028] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below, in which only the parts related to the embodiment of the present invention are shown.

[0029] Example:

[0030] like Figure 1 、 Figure 2As shown, the present invention provides a cross-modal medical image registration method based on a latent space diffusion model, comprising the following steps: S100: Acquire a first modality original medical image and a second modality original medical image, wherein the first modality and the second modality are different modalities, such as an MR image and a CT image. Since medical images are three-dimensional images, it is necessary to slice from different axes and view slices of different sections to obtain complete image information. The first modality original medical image and the second modality original medical image are sliced ​​and resized to obtain first modality slice images and second modality slice images, respectively. The resizing is unified to facilitate registration operations. The first modality slice images and the second modality slice images are preprocessed to obtain floating images and target images, forming a floating image and target image registration image pair, i.e., the floating object corresponds to the first modality and the target image corresponds to the second modality. The purpose of this method is to register the floating image of the first modality to the medical image of the second modality. S200: Constructing a cross-modal conversion network, which is used to convert between medical images of different modalities. Using a pre-trained autoencoder, the target image is compressed into a latent space representation and input into a diffusion model. Training is performed through forward denoising and backward denoising to obtain the original data corresponding to the target image. S300: The floating image is compressed via the autoencoder and input into the diffusion model of the cross-modal conversion network to generate a second modal conversion image. S400: Constructing an image registration network, inputting the floating image and the target image into the registration network, outputting a registration deformation field, and warping the floating image and the second modal conversion image based on the spatial transformation network and the registration deformation field to obtain their respective registered images. S500: Calculating the reconstruction loss between the registered floating image and the target image, the similarity loss between the registered second modal conversion image and the target image, and the smoothness loss of the registration deformation field. The registration network is trained by minimizing the reconstruction loss, similarity loss, and smoothness loss, and steps S200-S500 are repeated. S600: The floating image and the target image execute step S400 again to obtain an image after floating image registration. The present invention utilizes a diffusion model to achieve conversion between image modalities, which not only retains the excellent image generation quality and medical image detailed information of the diffusion model, but also uses unimodal attitude metrics to reduce the differences between modalities. The latent space diffusion model utilizes the perceptual compression of the autoencoder to perform a diffusion process in the latent space to achieve cross-modal conversion of images and improve the computational efficiency of the modality conversion process. The cross-modal registration method of the present invention has a wide range of applications and is suitable for registration between multiple modalities, including registration between MRI and CT, multi-sequence registration of MRI, and registration from PET to CT / MRI.

[0031] As an optional implementation, in step S100, the first modality slice image and the second modality slice image are obtained by slicing each volume of the first modality original medical image and the second modality original medical image respectively and adjusting the size to 256*256 pixels. The size adjustment can also be set to other sizes as needed. Adjusting the size to a uniform size facilitates the registration operation; the preprocessing operation includes affine transformation, alignment and normalization, so as to achieve coarse registration of the floating image and the target image. Affine transformation refers to the process in geometry where an image is mapped from one vector space to another vector space through a linear transformation and a translation to achieve the purpose of image calibration, image transformation, and enhanced visual effects; image alignment is to align different images in image registration to achieve accurate fusion of image information; normalization is a technology that maps image data to a specific range to improve the effect of data processing and model training. The main purpose is to standardize the value of the image or feature vector to make it easier to process and compare, enhance the stability of the numerical value, reduce the influence of noise, and improve the accuracy of feature extraction and matching.

[0032] As an optional implementation, in step S200, the cross-modal conversion network includes an autoencoder of the perceptual image compression module and a diffusion model of the perceptual latent variable space; the autoencoder includes an encoder and a decoder, and is trained using the target image after preprocessing operations. The autoencoder is an artificial neural network used in semi-supervised and unsupervised learning, mainly used for data compression and feature learning; the diffusion model is a type of deep learning method based on a probabilistic generative model. By simulating the physical diffusion process, it gradually converts data into noise, and then learns the inverse process to gradually recover the original data from the noise, achieving high-quality generation effects.

[0033] As an optional implementation, in step S200, the autoencoder is pre-trained based on perceptual loss and adversarial loss, and the pre-training loss function of the autoencoder is The expression is: ,in, is the perceptual loss, To combat losses; ,in, is the input image, is the reconstructed image, Represents a specific layer of the input image The extracted feature vectors, Represents a specific layer of the reconstructed image Extracted feature vectors.

[0034] As an optional implementation, in step S200, if Figure 4As shown in the figure, during the forward denoising process, the diffusion model gradually adds noise to the latent space data of the second modality of the input medical image, uses the results of the gradual denoising to predict the noise, calculate the loss function, and use the loss function for backpropagation until the diffusion model training is completed; the loss function of the diffusion model is: ,in Latent space representation of medical images of the second modality as input , conditional input ,noise and diffusion time The joint expectation of is a uniform sampling on the time series {1,2,...,T}, is the noise estimated by the diffusion model, Enter the condition The embedding representation of The representation of the medical image of the second modality after adding noise in the latent space is obtained.

[0035] As an optional implementation, in step S200, in reverse denoising, a UNet network is formed with the medical image of the first modality as a condition. The UNet network is a convolutional neural network for image segmentation, which is characterized by adopting a U-shaped network structure. The latent space data of the second modality medical image predicted under the input condition of the first modality medical image is decoded into the second modality conversion image in the pixel domain by the decoder. .

[0036] As an optional implementation, in step S300, an autoencoder is used to compress the target image data of the second modality into a latent space representation to generate latent space variables of the floating image; the latent space variables of the floating image are diffused by a diffusion model to generate latent space variables of the second modality; the latent space variables of the second modality are decoded into pixel space by a decoder to generate an image of the second modality, so that the floating image is converted from the first modality to the second modality.

[0037] As an optional implementation, in step S400, if Figure 3 As shown in the figure, the image registration network adopts a UNet network structure including an encoder and a decoder; the floating image and the target image are input into the registration network, and the registration deformation field is output; the floating image and the second modal conversion image are distorted and deformed respectively using the spatial transformation network and the registration deformation field to obtain the registered images. , .

[0038] As an optional embodiment, in step S500, the loss between the registered image and the target image is calculated, including the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after conversion image registration and the target image, and the smoothness loss of the deformation field. The total loss function of the image registration process is for: ,in 、 represents the weight parameter, represents the reconstruction loss between the image after floating image registration and the target image, represents the pixel index in the image, represents the total number of all pixels in the image, 、 Represent the floating image and target image after registration, represents the similarity loss between the image after the second modality conversion image registration and the target image; represents the smoothness loss of the deformation field, where is the spatial domain of the image, is the deformation field, is the gradient of the deformation field, Represents a point (voxel or pixel) in the spatial domain Ω of the image.

[0039] As an optional implementation, in step S600, the floating image and the obtained optimal deformation field are input into a spatial transformation network to obtain a registered image.

[0040] The embodiment is only a special example and does not represent only one way of implementing the present invention.

[0041] The foregoing is merely a preferred embodiment of the present invention. Those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the guidance of the present invention, these features and embodiments may be modified to suit specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be within the scope of the present invention.

Claims

1. A cross-modal medical image registration method based on latent space diffusion model, characterized in that: The following steps are involved: S100: Acquire a first modality original medical image and a second modality original medical image, slice and resize them respectively to obtain a first modality slice image and a second modality slice image, perform preprocessing operations on the first modality slice image and the second modality slice image to obtain a floating image and a target image, and form a registered image pair of the floating image and the target image; S200: constructing a cross-modal conversion network, using a pre-trained autoencoder to compress the target image into a latent space representation and input the representation into a diffusion model, and training the network through forward denoising and backward denoising; S300: The floating image is compressed by an autoencoder and input into the diffusion model of the cross-modal conversion network to generate a second modal conversion image; S400: constructing an image registration network, inputting the floating image and the target image into the registration network, outputting a registration deformation field, and performing warping and deformation on the floating image and the second modal conversion image based on the spatial transformation network and the registration deformation field to obtain respective registered images; S500: calculating a reconstruction loss between the image after floating image registration and the target image, a similarity loss between the image after second modality conversion image registration and the target image, and a smoothness loss of the registration deformation field, training the registration network based on minimizing the reconstruction loss, similarity loss, and smoothness loss, and repeatedly performing steps S200 to S500; S600: The floating image and the target image execute step S400 again to obtain an image after floating image registration; In step S100, the first modality slice image and the second modality slice image are obtained by slicing each volume of the first modality original medical image and the second modality original medical image and adjusting the size to 256*256 pixels respectively; the preprocessing operation includes affine transformation, alignment and normalization to perform coarse registration of the floating image and the target image; In step S200, the cross-modal conversion network includes an autoencoder of a perceptual image compression module and a diffusion model of a latent variable space; the autoencoder includes an encoder and a decoder, and is trained using the target image after preprocessing; In step S300, the target image data of the second modality is compressed into a latent space representation using the autoencoder to generate latent space variables of the floating image; the latent space variables of the floating image are diffused by the diffusion model to generate latent space variables of the second modality; the latent space variables of the second modality are decoded into pixel space by the decoder to generate an image of the second modality, so that the floating image is converted from the first modality to the second modality.

2. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In the step S200, the autoencoder is pre-trained based on perceptual loss and adversarial loss, and the pre-training loss function of the autoencoder is The expression is: ,in, is the perceptual loss, To combat losses; ,in, is the input image, is the reconstructed image, Represents a specific layer of the input image The extracted feature vectors, Represents a specific layer of the reconstructed image Extracted feature vectors.

3. The cross-modal medical image registration method based on latent space diffusion model according to claim 2, characterized in that: In step S200, during the forward noising process, the diffusion model gradually adds noise to the latent space data of the input second modality medical image, uses the results of the gradual noising to predict noise, calculate the loss function, and perform backpropagation using the loss function until the diffusion model training is completed; the loss function of the diffusion model is: ,in Latent space representation of medical images of the second modality as input , conditional input ,noise and diffusion time The joint expectation of is a uniform sampling on the time series {1,2,...,T}, is the noise estimated by the diffusion model, Enter the condition The embedding representation of The representation of the medical image of the second modality after adding noise in the latent space is input.

4. The cross-modal medical image registration method based on latent space diffusion model according to claim 3, characterized in that: In the step S200, in the reverse denoising, the UNet network is formed with the medical image of the first modality as a condition, and the latent space data of the medical image of the second modality predicted under the input condition of the medical image of the first modality is decoded into the second modality conversion image in the pixel domain by the decoder. .

5. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In step S400, the image registration network adopts a UNet network structure including an encoder and a decoder; the floating image and the target image are input into the registration network, and a registration deformation field is output; the floating image and the second modal conversion image are distorted and deformed respectively using the spatial transformation network and the registration deformation field to obtain their respective registered images.

6. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In step S500, the loss between the registered image and the target image is calculated, including the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after conversion image registration and the target image, and the smoothness loss of the deformation field. The total loss function of the image registration process is: ,in 、 represents the weight parameter, represents the reconstruction loss between the image after floating image registration and the target image, represents the pixel index in the image, represents the total number of all pixels in the image, 、 Represent the floating image and target image after registration, represents the similarity loss between the image after the second modality conversion image registration and the target image; represents the smoothness loss of the deformation field, where is the spatial domain of the image, is the deformation field, is the gradient of the deformation field, represents a point in the spatial domain Ω of the image.

7. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In step S600, the floating image and the obtained optimal deformation field are input into a spatial transformation network to obtain a registered image.

Images