Cross-modal medical image registration method based on submerged space diffusion model

Abstract

The invention discloses a cross-modal medical image registration method based on a submerged space diffusion model, relates to the technical field of medical image processing, and solves the technical problem that an existing registration method is not suitable for multi-modal medical image registration. The method comprises the steps of obtaining a first modal original medical image and a second modal original medical image, and performing preprocessing operation to obtain a floating image and a target image; constructing a cross-modal conversion network, compressing a target image to a submerged space for representation by using a pre-trained auto-encoder, inputting the target image into a diffusion model, and performing training through forward noise addition and reverse noise reduction; inputting the floating image into a diffusion model of a cross-modal conversion network to generate a second modal conversion image; inputting the floating image and the target image into a registration network, and obtaining respective registered images for the floating image and the second modal conversion image; and training the registration network to obtain an image after floating image registration. The method is wide in application field, and is suitable for registration among medical images in various modes.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a cross-modal medical image registration method based on a latent space diffusion model. Background Art

[0002] With the advancement of imaging technology, the improvement of imaging equipment performance and the continuous innovation and breakthroughs in computer technology, medical imaging technology has been widely used in clinical and medical informationization and has become one of the essential means of clinical diagnosis.

[0003] In the process of using medical imaging to assist diagnosis, it is usually necessary to fuse medical imaging information of different modalities to improve the accuracy of clinical diagnosis. For example, X-ray imaging technology (Xray) is often used to examine diseases such as pulmonary tuberculosis and fractures; digital subtraction angiography (DSA) is used to obtain clear vascular images to display and reconstruct blood vessels; magnetic resonance imaging (MR) is often used to diagnose cerebrovascular diseases, intracranial tumors, etc. It has the characteristics of multiple sequences, and different sequences can be used for the detection of different regions and lesions. These different imaging technologies can provide more detailed views in different tissue structure scenarios, which is crucial for understanding complex physiological processes. However, due to the different sources of images of different modalities, they often have deviations in space, so the application of image registration technology is particularly important.

[0004] Deformable image registration is an image registration technique that establishes a nonlinear correspondence between a moving and a target image pair. Fast, accurate, and realistic image registration algorithms are essential to improving the efficiency and accuracy of clinical practice. By observing dynamic changes such as lesions, doctors can design treatment plans for patients more comprehensively. However, in the existing multimodal medical image registration technology, the inherent heterogeneity between multimodal images and the differences in image characteristics caused by different imaging mechanisms greatly increase the complexity of the registration process. In addition, existing registration methods are often limited by the lack of effective similarity metrics and are not suitable for image registration in multimodal environments.

[0005] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: In multimodal medical image registration, the inherent heterogeneity between multimodal images and the differences in image characteristics caused by different imaging mechanisms greatly increase the complexity of the registration process. Existing registration methods are limited by the lack of effective similarity metrics and are not suitable for multimodal medical image registration. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-modal medical image registration method based on a latent space diffusion model to solve the technical problem that the registration method in the prior art is limited by the lack of effective similarity measurement and is not suitable for multi-modal medical image registration. The many technical effects that can be produced by the preferred technical solution among the many technical solutions provided by the present invention are described in detail below.

[0007] To achieve the above object, the present invention provides the following technical solutions: The present invention provides a cross-modal medical image registration method based on a latent space diffusion model, comprising the following steps: S100: acquiring a first modality original medical image and a second modality original medical image, slicing and resizing them respectively to obtain a first modality slice image and a second modality slice image, performing preprocessing operations on the first modality slice image and the second modality slice image to obtain a floating image and a target image, and forming a floating image and a target image registration image pair; S200: constructing a cross-modal conversion network, using a pre-trained autoencoder to compress the target image into a latent space representation and input it into a diffusion model, and training it through forward denoising and reverse denoising; S300: the floating image is compressed by the autoencoder and input into the diffusion model of the cross-modal conversion network to generate a first modality slice image and a second modality slice image. Second modal conversion image; S400: construct an image registration network, input the floating image and the target image into the registration network, output the registration deformation field, and distort the floating image and the second modal conversion image based on the spatial transformation network and the registration deformation field to obtain their respective registered images; S500: calculate the reconstruction loss between the image after the floating image is registered and the target image, the similarity loss between the image after the second modal conversion image is registered and the target image, and the smoothness loss of the registration deformation field, train the registration network based on minimizing the reconstruction loss, similarity loss, and smoothness loss, and repeat the steps S200~S500; S600: the floating image and the target image execute step S400 again to obtain the image after the floating image is registered.

[0008] Preferably, in the step S100, the first modality slice image and the second modality slice image are obtained by slicing each volume of the first modality original medical image and the second modality original medical image respectively and adjusting the size to 256*256 pixels; the preprocessing operation includes affine alignment and normalization.

[0009] Preferably, in step S200, the cross-modal conversion network includes an autoencoder of a perceptual image compression module and a diffusion model of a latent variable space; the autoencoder includes an encoder and a decoder, and is trained using the target image after a preprocessing operation.

[0010] Preferably, in step S200, the autoencoder is pre-trained based on perceptual loss and adversarial loss, and the pre-training loss function of the autoencoder is The expression is: ,in, is the perceived loss, To combat losses; ,in, is the input image, is the reconstructed image, Represents a specific layer from the input image The extracted feature vectors, Represents a specific layer from the reconstructed image Extracted feature vectors.

[0011] Preferably, in the step S200, in the forward noisy process, the diffusion model gradually adds noise to the latent space data of the input second modality medical image, predicts noise and calculates the loss function using the result of the gradual noisy process, and performs back propagation using the loss function until the diffusion model training is completed; the loss function of the diffusion model is: ,in The latent space representation of the medical image of the second modality is , conditional input ,noise and diffusion time The joint expectation of is a uniform sampling on the time series {1,2,...,T}, is the noise estimated by the diffusion model, Enter the condition The embedding representation of The representation of the medical image of the second modality after adding noise in the latent space is input.

[0012] Preferably, in the step S200, in the reverse denoising, the UNet network is formed with the medical image of the first modality as a condition, and the latent space data of the medical image of the second modality predicted under the input condition of the medical image of the first modality is decoded into the second modality conversion image in the pixel domain by the decoder .

[0013] Preferably, in step S300, the target image data of the second modality is compressed into a latent space representation using the autoencoder to generate latent space variables of the floating image; the latent space variables of the floating image are diffused by a diffusion model to generate latent space variables of the second modality; the latent space variables of the second modality are decoded into a pixel space by a decoder to generate an image of the second modality, so that the floating image is converted from the first modality to the second modality.

[0014] Preferably, in step S400, the image registration network adopts a UNet network structure including an encoder and a decoder; the floating image and the target image are input into the registration network, and a registration deformation field is output; the floating image and the second modal conversion image are distorted and deformed respectively using the spatial transformation network and the registration deformation field to obtain respective registered images. Preferably, in the step S500, the loss between the registered image and the target image is calculated, including the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after conversion image registration and the target image, and the smoothness loss of the deformation field. The total loss function of the image registration process is: ,in , represents the weight parameter, represents the reconstruction loss between the image after floating image registration and the target image, represents the pixel index in the image, represents the total number of all pixels in the image, , Respectively represent the floating image and target image after registration, represents the similarity loss between the image after the second modality conversion image registration and the target image; represents the smoothness loss of the deformation field, where is the spatial domain of the image, is the deformation field, is the gradient of the deformation field, represents a point in the spatial domain Ω of the image.

[0015] Preferably, in step S600, the floating image and the obtained optimal deformation field are input into a spatial transformation network to obtain a registered image.

[0016] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects: The present invention utilizes a diffusion model to realize conversion between image modalities, which not only retains the excellent image generation quality and detailed information of medical images of the diffusion model, but also uses unimodal attitude metrics to reduce the difference between modalities; the latent space diffusion model utilizes the perceptual compression of the autoencoder to perform a diffusion process in the latent space to realize cross-modal conversion of images and improve the computational efficiency in the modality conversion process; the cross-modal registration method of the present invention has a wide range of applications and is suitable for registration between multiple modalities, including registration between MRI and CT, multi-sequence registration of MRI, and registration from PET to CT / MRI. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 is a flow chart of a cross-modal medical image registration method based on a latent space diffusion model according to an embodiment of the present invention; Figure 2 is a schematic diagram of a registration process in a cross-modal medical image registration method based on a latent space diffusion model in an embodiment of the present invention; Figure 3 is a schematic diagram of the overall network structure in the cross-modal medical image registration method based on the latent space diffusion model in an embodiment of the present invention; Figure 4 It is a schematic diagram of the modality conversion network structure in the cross-modality medical image registration method based on the latent space diffusion model in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. that are consistent with some aspects of the present disclosure as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present invention.

[0019] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0020] In order to illustrate the technical solution of the present invention, a specific embodiment is used below for description, and only the parts related to the embodiment of the present invention are shown.

[0021] Example: like Figure 1 , Figure 2As shown, the present invention provides a cross-modal medical image registration method based on a latent space diffusion model, comprising the following steps: S100: obtaining a first modality original medical image and a second modality original medical image, the first modality and the second modality are different modalities, such as the first modality is an MR image and the second modality is a CT image. Since the medical image is a three-dimensional image, it is necessary to view slices of different sections after cutting from different axes to obtain complete image information. The first modality original medical image and the second modality original medical image are sliced ​​and resized to obtain a first modality slice image and a second modality slice image respectively, and the resize adjustment is unified to facilitate the registration operation. The first modality slice image and the second modality slice image are preprocessed to obtain a floating image and a target image, forming a floating image and a target image registration image pair, that is, the floating object corresponds to the first modality, and the target image corresponds to the second modality. The purpose of this method is to register the floating image of the first modality to the medical image of the second modality. S200: Construct a cross-modal conversion network, which is used to realize the conversion between medical images of different modalities. Use a pre-trained autoencoder to compress the target image into a latent space representation and input it into a diffusion model. Train it through forward denoising and reverse denoising, so as to obtain the original data corresponding to the target image. S300: The floating image is compressed by the autoencoder and input into the diffusion model of the cross-modal conversion network to generate a second modal conversion image; S400: Construct an image registration network, input the floating image and the target image into the registration network, output the registration deformation field, and distort the floating image and the second modal conversion image based on the spatial transformation network and the registration deformation field to obtain their respective registered images. S500: Calculate the reconstruction loss between the image after the floating image is registered and the target image, the similarity loss between the image after the second modal conversion image is registered and the target image, and the smoothness loss of the registration deformation field, train the registration network based on minimizing the reconstruction loss, similarity loss, and smoothness loss, and repeat the steps S200~S500. S600: The floating image and the target image execute step S400 again to obtain the image after floating image registration. The present invention uses a diffusion model to realize the conversion between image modalities, which not only retains the excellent image generation quality and medical image detailed information of the diffusion model, but also uses unimodal metrics to reduce the difference between modalities; the latent space diffusion model uses the perceptual compression of the autoencoder to perform a diffusion process in the latent space to realize the cross-modal conversion of the image and improve the computational efficiency in the modality conversion process; the cross-modal registration method of the present invention has a wide range of applications and is suitable for registration between multiple modalities, including registration between MRI and CT, multi-sequence registration of MRI, and registration from PET to CT / MRI.

[0022] As an optional implementation, in step S100, the first modality slice image and the second modality slice image are obtained by slicing each volume of the first modality original medical image and the second modality original medical image and adjusting the size to 256*256 pixels. The size adjustment can also be set to other sizes as needed. The size adjustment is a unified size for easy registration operation; the preprocessing operation includes affine transformation, alignment and normalization, so as to achieve rough registration of the floating image and the target image. Affine transformation refers to the process in geometry where an image is mapped from one vector space to another vector space through a linear transformation and a translation to achieve the purpose of image calibration, image transformation, and enhanced visual effects; image alignment is to align different images in image registration to achieve accurate fusion of image information; normalization is a technology that maps image data to a specific range to improve the effect of data processing and model training. The main purpose is to standardize the value of the image or feature vector to make it easier to process and compare, enhance the stability of the numerical value, reduce the influence of noise, and improve the accuracy of feature extraction and matching.

[0023] As an optional implementation, in step S200, the cross-modal conversion network includes an autoencoder of a perceptual image compression module and a diffusion model of a perceptual latent variable space; the autoencoder includes an encoder and a decoder, and is trained using a target image after a preprocessing operation. The autoencoder is an artificial neural network used in semi-supervised learning and unsupervised learning, mainly used for data compression and feature learning; the diffusion model is a type of deep learning method based on a probabilistic generation model, which gradually converts data into noise by simulating a physical diffusion process, and then learns the reverse process to gradually recover the original data from the noise to achieve a high-quality generation effect.

[0024] As an optional implementation, in step S200, the autoencoder is pre-trained based on perceptual loss and adversarial loss, and the pre-training loss function of the autoencoder is The expression is: ,in, is the perceived loss, To combat losses; ,in, is the input image, is the reconstructed image, Represents a specific layer from the input image The extracted feature vectors, Represents a specific layer from the reconstructed image Extracted feature vectors.

[0025] As an optional implementation, in step S200, if Figure 4As shown in the figure, in the forward denoising process, the diffusion model gradually adds noise to the latent space data of the input second modality medical image, uses the result of the gradual denoising to predict the noise, calculate the loss function, and uses the loss function for back propagation until the diffusion model training is completed; the loss function of the diffusion model is: ,in The latent space representation of the medical image of the second modality is , conditional input ,noise and diffusion time The joint expectation of is a uniform sampling on the time series {1,2,...,T}, is the noise estimated by the diffusion model, Enter the condition The embedding representation of The representation of the medical image of the second modality after adding noise in the latent space is input.

[0026] As an optional implementation, in step S200, in reverse denoising, a UNet network is formed with the medical image of the first modality as a condition. The UNet network is a convolutional neural network for image segmentation, which is characterized by adopting a U-shaped network structure. The latent space data of the medical image of the second modality predicted under the input condition of the medical image of the first modality is decoded into the second modality conversion image in the pixel domain by a decoder. .

[0027] As an optional implementation, in step S300, an autoencoder is used to compress the target image data of the second modality into a latent space representation to generate latent space variables of a floating image; the latent space variables of the floating image are diffused by a diffusion model to generate latent space variables of the second modality; the latent space variables of the second modality are decoded into a pixel space by a decoder to generate an image of the second modality, so that the floating image is converted from the first modality to the second modality.

[0028] As an optional implementation, in step S400, if Figure 3 As shown in FIG. 1 , the image registration network adopts a UNet network structure including an encoder and a decoder; the floating image and the target image are input into the registration network, and the registration deformation field is output; the floating image and the second modal conversion image are distorted and deformed respectively using the spatial transformation network and the registration deformation field to obtain the registered images , .

[0029] As an optional implementation, in step S500, the loss between the registered image and the target image is calculated, including the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after conversion image registration and the target image, and the smoothness loss of the deformation field. The total loss function of the image registration process is for: ,in , represents the weight parameter, represents the reconstruction loss between the image after floating image registration and the target image, represents the pixel index in the image, represents the total number of all pixels in the image, , Respectively represent the floating image and target image after registration, represents the similarity loss between the image after the second modality conversion image registration and the target image; represents the smoothness loss of the deformation field, where is the spatial domain of the image, is the deformation field, is the gradient of the deformation field, Represents a point (voxel or pixel) in the spatial domain Ω of the image.

[0030] As an optional implementation, in step S600, the floating image and the obtained optimal deformation field are input into a spatial transformation network to obtain a registered image.

[0031] The embodiment is only a specific example and does not represent only one implementation mode of the present invention.

[0032] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A cross-modal medical image registration method based on latent space diffusion model, characterized in that: The following steps are involved: S100: Acquire a first modality original medical image and a second modality original medical image, slice and resize them respectively to obtain a first modality slice image and a second modality slice image, perform preprocessing operations on the first modality slice image and the second modality slice image to obtain a floating image and a target image, and form a floating image and a target image registration image pair; S200: constructing a cross-modal conversion network, using a pre-trained autoencoder to compress the target image into a latent space representation and input it into a diffusion model, and training it through forward denoising and reverse denoising; S300: The floating image is compressed by an autoencoder and input into a diffusion model of the cross-modal conversion network to generate a second modal conversion image; S400: constructing an image registration network, inputting the floating image and the target image into the registration network, outputting a registration deformation field, and distorting the floating image and the second modal conversion image based on the spatial transformation network and the registration deformation field to obtain respective registered images; S500: calculating the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after second modal conversion image registration and the target image, and the smoothness loss of the registration deformation field, training the registration network based on minimizing the reconstruction loss, similarity loss, and smoothness loss, and repeatedly executing the steps S200 to S500; S600: The floating image and the target image execute step S400 again to obtain an image after the floating image is registered.

2. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In the step S100, the first modality slice image and the second modality slice image are obtained by slicing each volume of the first modality original medical image and the second modality original medical image and adjusting the size to 256*256 pixels respectively; the preprocessing operation includes affine alignment and normalization.

3. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In the step S200, the cross-modal conversion network includes an autoencoder of a perceptual image compression module and a diffusion model of a latent variable space; the autoencoder includes an encoder and a decoder, and is trained using the target image after a preprocessing operation.

4. The cross-modal medical image registration method based on latent space diffusion model according to claim 3, characterized in that: In the step S200, the autoencoder is pre-trained based on perceptual loss and adversarial loss, and the pre-training loss function of the autoencoder is The expression is: ,in, is the perceived loss, To combat losses; ,in, is the input image, is the reconstructed image, Represents a specific layer from the input image The extracted feature vectors, Represents a specific layer from the reconstructed image Extracted feature vectors.

5. The cross-modal medical image registration method based on latent space diffusion model according to claim 4, characterized in that: In the step S200, in the forward noisy process, the diffusion model gradually adds noise to the latent space data of the input second modality medical image, predicts noise and calculates the loss function using the result of the gradual noisy process, and performs back propagation using the loss function until the diffusion model training is completed; the loss function of the diffusion model is: ,in The latent space representation of the medical image of the second modality is , conditional input ,noise and diffusion time The joint expectation of is a uniform sampling on the time series {1,2,...,T}, is the noise estimated by the diffusion model, Enter the condition The embedding representation of The representation of the medical image of the second modality after adding noise in the latent space is input.

6. The cross-modal medical image registration method based on latent space diffusion model according to claim 5, characterized in that: In the step S200, in the reverse denoising, the UNet network is formed with the medical image of the first modality as a condition, and the latent space data of the medical image of the second modality predicted under the input condition of the medical image of the first modality is decoded into the second modality conversion image in the pixel domain by the decoder. .

7. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In the step S300, the target image data of the second modality is compressed into a latent space representation using the autoencoder to generate latent space variables of the floating image; the latent space variables of the floating image are diffused by a diffusion model to generate latent space variables of the second modality; the latent space variables of the second modality are decoded into a pixel space by a decoder to generate an image of the second modality, so that the floating image is converted from the first modality to the second modality.

8. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In the step S400, the image registration network adopts a UNet network structure including an encoder and a decoder; the floating image and the target image are input into the registration network, and the registration deformation field is output; the floating image and the second modal conversion image are distorted and deformed respectively by using the spatial transformation network and the registration deformation field to obtain respective registered images.

9. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In the step S500, the loss between the registered image and the target image is calculated, including the reconstruction loss between the image after floating image registration and the target image, the similarity loss between the image after conversion image registration and the target image, and the smoothness loss of the deformation field. The total loss function of the image registration process is: ,in , represents the weight parameter, represents the reconstruction loss between the image after floating image registration and the target image, represents the pixel index in the image, represents the total number of all pixels in the image, , Respectively represent the floating image and target image after registration, represents the similarity loss between the image after the second modality conversion image registration and the target image; represents the smoothness loss of the deformation field, where is the spatial domain of the image, is the deformation field, is the gradient of the deformation field, represents a point in the spatial domain Ω of the image.

10. The cross-modal medical image registration method based on latent space diffusion model according to claim 1, characterized in that: In step S600, the floating image and the obtained optimal deformation field are input into a spatial transformation network to obtain a registered image.

Images