Multi-modal medical image elastic registration system based on CyclGan

Abstract

The invention relates to the technical field of medical image processing, and discloses a CyclGan-based multi-modal medical image elastic registration system, which comprises a multi-modal medical image data preprocessing module, a CyclGan-based elastic registration module, a CyclGan-based elastic registration module and a CyclGan-based elastic registration module, a module for training the CyclGan model on the basis of training strategies of medical image data of different modalities; a module for performing elastic transformation and resampling on the preprocessed medical image based on an output grid of the model; and a post-processing module is used for post-processing the deformed medical image. Through the above operations, automatic elastic registration between medical images of different modalities and fusion of medical images of different modalities after registration and alignment can be realized, a doctor is helped to obtain more comprehensive information and more accurately position a lesion area, and thus the accuracy and efficiency of diagnosis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multimodal medical image elastic registration system based on CycleGan. Background Art

[0002] In medical diagnosis, medical images of different modalities (such as CT, MRI, PET, etc.) are usually used to obtain different information about the patient's body. However, due to the different imaging principles and parameter settings of different modal images, these images often have inconsistencies in spatial position. In order to make comprehensive use of this information, it is necessary to align images of different modalities, that is, elastic registration. Traditional registration systems usually require manual intervention, are inefficient and prone to errors. Therefore, it is of great application value to develop a fully automated elastic registration system for multimodal medical images. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal medical image elastic registration system based on CycleGan, which can realize automatic elastic registration between medical images of different modalities. The fusion of medical images of different modalities after registration and alignment helps doctors obtain more comprehensive information and locate the lesion area more accurately, thereby improving the accuracy and efficiency of diagnosis.

[0004] The present invention provides the following technical solution: a multimodal medical image elastic registration system based on CycleGan, comprising the following modules:

[0005] Data preprocessing module: preprocess the input multimodal medical images, including automatic calculation of the optimal window position, rigid registration, image contrast enhancement, cropping, and normalization operations to improve image quality and consistency;

[0006] CycleGan model training module: adopts training strategies based on medical imaging data of different modalities to train the CycleGan model and perform registration between images of different modalities;

[0007] Elastic transformation and resampling module: Based on the output grid of the trained CycleGan model, the preprocessed medical image is elastically transformed and resampled. The deformation information of the output grid is elastically transformed with the input image to align it with the image of another modality in space. The resampling interpolates or resamples the original image according to the transformed image coordinates to generate a registered image with the same resolution and size as the image of another modality.

[0008] Post-processing module: Post-process the deformed medical images, including smoothing and denoising operations, to improve the quality and consistency of the registered images.

[0009] Preferably, the method for automatically calculating the window width and window level is as follows:

[0010] ① Analyze the histogram of medical images, that is, count the frequency of occurrence of each pixel value;

[0011] ② Based on the histogram data, calculate the cumulative distribution function of the image, which shows the probability that the pixel value is less than or equal to a certain value;

[0012] ③ Locate two specific points on the cumulative distribution function. The function values ​​of these two points are close to 0.01 and 0.97, respectively, representing the pixel values ​​corresponding to the low-brightness and high-brightness features in the image;

[0013] ④Determine the pixel value indexes corresponding to these two points. These indexes indicate the dynamic range of image brightness on the histogram;

[0014] ⑤ Use these index values ​​to set the window width and window level to ensure that most important features in the image can be clearly presented on the display device.

[0015] Preferably, the method of automatic rigid registration is as follows:

[0016] ① Extraction of bone contour mask images of medical images: Based on the pixel threshold limiting method, the pixel values ​​greater than 150 are set to 0, and the pixels outside this range are set to 0, that is, the mask image of the bone is obtained. Then, the holes in the mask image are automatically filled by dilation and corrosion;

[0017] ② Noise reduction of the skeleton mask image: First, count the number and area of ​​the connected domains of the skeleton mask image, then only retain the largest connected domain in the 3D space, and delete the foreground corresponding to other connected domains to obtain the denoised skeleton mask image;

[0018] ③ Rigid registration of bone mask images: A pyramid multi-resolution registration framework is used, which consists of three modules: a rigid space transformation model, a conventional gradient descent optimization algorithm, and a DICE similarity measure. The rigid registration of bone mask images can be performed to obtain rigid transformation parameters.

[0019] ④ Based on the above rigid transformation parameters, the original medical image is rigidly transformed to obtain the medical image after rigid registration.

[0020] Preferably, the image contrast enhancement method, for CBCT / CT images, adopts a range truncation method, that is, limiting the Hu value to between a given minimum and maximum value; for MR images, a histogram equalization contrast automatic enhancement method is adopted.

[0021] Preferably, the CycleGan model contains two generators (G_A and G_B), where generator G_A is optimized specifically for the conversion from modality A to modality B, while generator G_B focuses on the conversion from modality B to modality A. Each generator adopts a deep convolutional neural network architecture and adds 9 deep residual network units between key layers. The model is trained using a combination of the modality-independent neighborhood descriptor loss function MIND and cycle consistency loss, and deformable mesh smoothing loss;

[0022] The choice of the target loss function determines the result of network training. The target loss function of the entire network model includes the following three items: similarity measurement loss MIND, regularization loss, and cycle loss:

[0023] MIND can compare the local blocks of image voxels with the local blocks of their surrounding neighborhoods to extract image structural information. This structural information shows the relationship between image voxels and their surrounding voxels, helping to determine whether the voxels are in a continuous area or a boundary area. The MIND feature extraction formula is as follows:

[0024] ;

[0025] ;

[0026] ;

[0027] Where I represents the image, x represents the spatial position of the image voxel, R represents the spatial search area, n is a constant used for normalization, D represents the L2 norm distance between two image blocks in image I, and the two image blocks are searched with x and x+r as the search center respectively. The calculation formula of D is, where P represents the search area of ​​the image block, V is the variance estimate of voxel x, and N refers to the spatial neighborhood area of ​​voxel x;

[0028] The regularization loss term is to prevent unreasonable deformation during network training and increase the objective function loss term. At the same time, it can make the deformed mesh smoother. The specific form of the regularization loss term is as follows: ,in represents the displacement vector field of the mesh control points, represents the L2 norm;

[0029] The purpose of the cyclic loss term is to transform the deformed floating image obtained after the deformation transformation back to the original image, that is, floating image-deformed floating image-floating image; this loss term ensures that the deformation registration result has cyclic consistency. The specific form of the cyclic loss term is as follows:

[0030] ;

[0031] in and represents two deformation generation networks, represents image A, represents image B, Represents the L1 norm.

[0032] Preferably, the training strategies for medical imaging data of different modalities are as follows:

[0033] ① Data collection and preprocessing: Collect medical imaging data from different hospitals, different modalities and different anatomical parts, and preprocess the original medical imaging data through batch rigid registration, denoising, contrast enhancement, cropping and filling, and normalization steps;

[0034] ② Multimodal data fusion: This training strategy first uses multimodal medical imaging data (such as CT, MRI, CBCT) for fusion processing; through appropriate fusion algorithms (multimodal fusion networks based on deep learning), image information of different modalities is effectively integrated to extract richer features and improve the model's ability to recognize target structures;

[0035] ③Model selection and optimization: According to the task requirements and data characteristics, convolutional neural network CNN and generative adversarial network GAN are selected as the training basis. During the training process, the model performance is optimized by adjusting the model's hyperparameters and optimization algorithms;

[0036] ④ Transfer learning and fine-tuning: This training strategy uses a transfer learning method, using a model pre-trained on a large dataset for initialization, and then fine-tuning it on the target dataset to adapt to the needs of a specific task;

[0037] ⑤ Loss function and evaluation indicators: In order to measure the performance of the model, select appropriate loss functions (such as deformable mesh gradient smoothing loss, MIND image measure, cycle loss) and evaluation indicators (such as MI mutual information measure, recall rate, registration time, etc.), evaluate the performance of the model regularly, and adjust the training strategy according to the evaluation results to ensure that the model can be continuously optimized.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] High degree of automation: Through the trained CycleGan model, the system can automatically complete the elastic registration of medical images of different modalities without manual intervention, greatly improving the efficiency and accuracy of registration;

[0040] Strong generalization ability: The CycleGan model used in the present invention has strong generalization ability and can adapt to medical imaging data of different modalities, different parts, and different patients, and realize a wide range of registration applications;

[0041] Good fusion effect: Through elastic registration and image fusion technology, the present invention can fuse medical imaging information of different modalities together to provide doctors with more comprehensive and accurate diagnostic support;

[0042] Easy to implement and promote: The algorithms and technologies used in this invention are based on mature deep learning frameworks and image processing libraries, which are easy to implement and promote; at the same time, the system can be easily integrated into existing medical image processing software to provide doctors with more convenient and efficient services. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a flow chart of a fully automated elastic registration system for multimodal medical images;

[0044] FIG2 is a flowchart of automatic rigid registration based on bone mask images;

[0045] Figure 3 is a flowchart of multimodal elastic registration based on cycle consistency CycleGan network. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of known functions and known components to avoid unnecessary confusion of the concepts of the present invention.

[0047] In a first aspect, the present invention provides a fully automated elastic registration system for multimodal medical images, the implementation steps of which are shown in FIG1 . The system mainly includes the following modules:

[0048] Data preprocessing module: preprocess the input multimodal medical images, including automatic calculation of the optimal window position, rigid registration, image contrast enhancement, cropping, and normalization operations to improve image quality and consistency;

[0049] CycleGan model training module: The CycleGan model is trained using a training strategy based on medical imaging data of different modalities. The CycleGan model is a generative adversarial network (GAN) that can learn the mapping relationship between images of one modality and images of another modality. Through training, the model can generate images of another modality corresponding to the input image, thereby achieving registration between images of different modalities;

[0050] Elastic transformation and resampling module: Based on the output grid of the trained CycleGan model, the preprocessed medical image is elastically transformed and resampled. The deformation information of the output grid is elastically transformed with the input image to align it with the image of another modality in space. The resampling interpolates or resamples the original image according to the transformed image coordinates to generate a registered image with the same resolution and size as the image of another modality.

[0051] Post-processing module: Post-process the deformed medical images, including smoothing and denoising operations, to improve the quality and consistency of the registered images.

[0052] In a second aspect, the present invention provides a method for automatically calculating window width and window position, the implementation steps comprising:

[0053] ① Analyze the histogram of medical images, that is, count the frequency of occurrence of each pixel value.

[0054] ② Based on the histogram data, calculate the cumulative distribution function of the image, which shows the probability that the pixel value is less than or equal to a certain value.

[0055] ③ Locate two specific points on the cumulative distribution function. The function values ​​of these two points are close to 0.01 and 0.97, respectively, representing the pixel values ​​corresponding to the low-brightness and high-brightness features in the image.

[0056] ④ Determine the pixel value indexes corresponding to these two points. These indexes indicate the dynamic range of image brightness on the histogram.

[0057] ⑤ Use these index values ​​to set the window width and window level to ensure that most important features in the image can be clearly presented on the display device.

[0058] Through the above steps, the system can automatically calculate the optimal window width and window level, optimize the display effect of medical images, and highlight the important features in the diagnosis process.

[0059] In a third aspect, the present invention provides a method for automatic rigid registration (as shown in FIG2 ), the implementation steps comprising:

[0060] ① Extraction of bone contour mask images from medical images: Based on the pixel threshold limiting method, the pixel values ​​greater than 150 are set to 0, and the pixels outside this range are set to 0, thus obtaining the bone mask image. Then, the holes in the mask image are automatically filled using dilation and erosion.

[0061] ② Denoising of the bone mask image: Since medical images contain artifacts, dose contrast and other noises, these noises will be extracted as bones. In order to reduce the noise of the bone mask image, the number and area of ​​the connected domains of the bone mask image are first counted, and then only the largest connected domain in the 3D space is retained, and the foreground corresponding to other connected domains is deleted to obtain the denoised bone mask image.

[0062] ③ Rigid registration of bone mask image: The registration system used for rigid registration is based on the pyramid multi-resolution registration framework, and is composed and implemented by three modules: rigid space transformation model, conventional gradient descent optimization algorithm, and DICE similarity measure. Using the rigid registration system, the bone mask image is rigidly registered to obtain the rigid transformation parameters.

[0063] ④ Based on the above rigid transformation parameters, the original medical image is rigidly transformed to obtain the medical image after rigid registration.

[0064] Through the above steps, the system can automatically perform batch rigid registration on medical image data sets. It has been verified that the automatic rigid registration system has high registration accuracy and registration pass rate.

[0065] Different data enhancement methods are used for data of different modalities. For CBCT / CT images, the range truncation method is used, that is, the Hu value is limited to a given minimum and maximum value; for MR images, the histogram equalization contrast automatic enhancement method is used.

[0066] In a fourth aspect, the present invention provides a CycleGan network model for elastic registration of medical images of different modalities (as shown in FIG3). The CycleGan network model is characterized in that it comprises two generators (G_A and G_B), wherein the generator G_A is specifically optimized for the conversion from modality A to modality B, while the generator G_B focuses on the conversion from modality B to modality A. This bidirectional conversion design ensures the consistency and accuracy of the conversion through cycle consistency loss. Each generator adopts a deep convolutional neural network architecture and adds 9 deep residual network units between key layers to improve the convergence ability of the neural network. The training of the model adopts a combination of the modality-independent neighborhood descriptor loss function MIND and cycle consistency loss, and deformable mesh smoothing loss, aiming to simultaneously optimize the realism and consistency of image conversion and improve model performance.

[0067] The choice of the target loss function determines the result of network training. The target loss function of the entire network model includes the following three items: similarity measurement loss MIND, regularization loss, and cycle loss.

[0068] In terms of the choice of similarity measure loss, MI is one of the most common similarity measures for multimodal image registration. However, since MI ignores the correlation between a single voxel in the image and its adjacent spatial region, the present invention adopts a similarity measure called modality-independent neighborhood descriptor (MIND) to achieve elastic registration of multimodal medical images. MIND can compare the local blocks of image voxels with the local blocks of their surrounding neighborhoods to extract image structural information. This structural information indicates the relationship between image voxels and their surrounding voxels, helping to determine whether the voxels are in a continuous region or a boundary region. The MIND feature extraction formula is as follows:

[0069] ;

[0070] ;

[0071] ;

[0072] Where I represents the image, x represents the spatial position of the image voxel, R represents the spatial search area, n is a constant used for normalization, D represents the L2 norm distance between two image blocks in image I, and the two image blocks are searched with x and x+r as the search center respectively. The calculation formula of D is, where P represents the search area of ​​the image block, V is the variance estimate of voxel x, and N refers to the spatial neighborhood area of ​​voxel x;

[0073] The regularization loss term is to prevent unreasonable deformation during network training and increase the objective function loss term. At the same time, it can make the deformed mesh smoother. The specific form of the regularization loss term is as follows: ,in represents the displacement vector field of the mesh control points, represents the L2 norm;

[0074] The purpose of the cyclic loss term is to transform the deformed floating image obtained after the deformation transformation back to the original image, that is, floating image-deformed floating image-floating image; this loss term ensures that the deformation registration result has cyclic consistency. The specific form of the cyclic loss term is as follows:

[0075] ;

[0076] in and represents two deformation generation networks, represents image A, represents image B, Represents the L1 norm.

[0077] Fifth, in order to maintain the topological structure and reversibility of deformation of medical images during deformation registration, a diffeomorphic integration layer is added to the CycleGan network model to ensure that the unsupervised end-to-end learning is diffeomorphic. Diffeomorphisms are differentiable and reversible, thus preserving topology. We directly regard the output of the registration network as a velocity field , rather than the displacement field By normalizing the time period Integrating the inner part, we can get the displacement field: .

[0078] Fifth, the training strategies for medical imaging data of different modalities are as follows:

[0079] ① Data collection and preprocessing: Collect medical imaging data from different hospitals, different modalities and different anatomical parts, and preprocess the original medical imaging data through batch rigid registration, denoising, contrast enhancement, cropping and filling, and normalization steps;

[0080] ② Multimodal data fusion: This training strategy first uses multimodal medical imaging data (such as CT, MRI, CBCT) for fusion processing; through appropriate fusion algorithms (multimodal fusion networks based on deep learning), image information of different modalities is effectively integrated to extract richer features and improve the model's ability to recognize target structures;

[0081] ③Model selection and optimization: According to the task requirements and data characteristics, convolutional neural network CNN and generative adversarial network GAN are selected as the training basis. During the training process, the model performance is optimized by adjusting the model's hyperparameters and optimization algorithms;

[0082] ④ Transfer learning and fine-tuning: This training strategy uses a transfer learning method, using a model pre-trained on a large dataset for initialization, and then fine-tuning it on the target dataset to adapt to the needs of a specific task;

[0083] ⑤ Loss function and evaluation indicators: In order to measure the performance of the model, select appropriate loss functions (such as deformable mesh gradient smoothing loss, MIND image measure, cycle loss) and evaluation indicators (such as MI mutual information measure, recall rate, registration time, etc.), evaluate the performance of the model regularly, and adjust the training strategy according to the evaluation results to ensure that the model can be continuously optimized.

[0084] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A multimodal medical image elastic registration system based on CycleGan, characterized in that: Includes the following modules: Data preprocessing module: preprocess the input multimodal medical images, including automatic calculation of the optimal window position, rigid registration, image contrast enhancement, cropping, and normalization operations to improve image quality and consistency; CycleGan model training module: adopts training strategies based on medical imaging data of different modalities to train the CycleGan model and perform registration between images of different modalities; Elastic transformation and resampling module: Based on the output grid of the trained CycleGan model, the preprocessed medical image is elastically transformed and resampled. The deformation information of the output grid is elastically transformed with the input image to align it with the image of another modality in space. The resampling interpolates or resamples the original image according to the transformed image coordinates to generate a registered image with the same resolution and size as the image of another modality. Post-processing module: Post-process the deformed medical images, including smoothing and denoising operations, to improve the quality and consistency of the registered images.

2. The multimodal medical image elastic registration system based on CycleGan according to claim 1, characterized in that: The method for automatically calculating the window width and window level is as follows: ① Analyze the histogram of medical images, that is, count the frequency of occurrence of each pixel value; ② Based on the histogram data, calculate the cumulative distribution function of the image, which shows the probability that the pixel value is less than or equal to a certain value; ③ Locate two specific points on the cumulative distribution function. The function values ​​of these two points are close to 0.01 and 0.97, respectively, representing the pixel values ​​corresponding to the low-brightness and high-brightness features in the image; ④Determine the pixel value indexes corresponding to these two points. These indexes indicate the dynamic range of image brightness on the histogram; ⑤ Use these index values ​​to set the window width and window level to ensure that most important features in the image can be clearly presented on the display device.

3. The multimodal medical image elastic registration system based on CycleGan according to claim 1, characterized in that: The method of automatic rigid registration is as follows: ① Extraction of bone contour mask images of medical images: Based on the pixel threshold limiting method, the pixel values ​​greater than 150 are set to 0, and the pixels outside this range are set to 0, that is, the mask image of the bone is obtained. Then, the holes in the mask image are automatically filled by dilation and corrosion; ② Noise reduction of the skeleton mask image: First, count the number and area of ​​the connected domains of the skeleton mask image, then only retain the largest connected domain in the 3D space, and delete the foreground corresponding to other connected domains to obtain the denoised skeleton mask image; ③ Rigid registration of bone mask images: A pyramid multi-resolution registration framework is used, which consists of three modules: a rigid space transformation model, a conventional gradient descent optimization algorithm, and a DICE similarity measure. The rigid registration of bone mask images can be performed to obtain rigid transformation parameters. ④ Based on the above rigid transformation parameters, the original medical image is rigidly transformed to obtain the medical image after rigid registration.

4. The multimodal medical image elastic registration system based on CycleGan according to claim 1, characterized in that: The image contrast enhancement method, for CBCT / CT images, uses the value range truncation method, that is, limiting the Hu value to a given minimum and maximum value; for MR images, the histogram equalization contrast automatic enhancement method is used.

5. The multimodal medical image elastic registration system based on CycleGan according to claim 1, characterized in that: The CycleGan model consists of two generators G_A and G_B. Generator G_A is optimized for the conversion from modality A to modality B, while generator G_B focuses on the conversion from modality B to modality A. Each generator adopts a deep convolutional neural network architecture and adds 9 deep residual network units between key layers. The model is trained using a combination of the modality-independent neighborhood descriptor loss function MIND, cycle consistency loss, and deformable mesh smoothing loss. The choice of the target loss function determines the result of network training. The target loss function of the entire network model includes the following three items: similarity measurement loss MIND, regularization loss, and cycle loss: MIND can compare the local blocks of image voxels with the local blocks of their surrounding neighborhoods to extract image structural information. This structural information shows the relationship between image voxels and their surrounding voxels, helping to determine whether the voxels are in a continuous area or a boundary area. The MIND feature extraction formula is as follows: ； ； ； Where I represents the image, x represents the spatial position of the image voxel, R represents the spatial search area, n is a constant used for normalization, D represents the L2 norm distance between two image blocks in image I, and the two image blocks are searched with x and x+r as the search center respectively. The calculation formula of D is, where P represents the search area of ​​the image block, V is the variance estimate of voxel x, and N refers to the spatial neighborhood area of ​​voxel x; The regularization loss term is to prevent unreasonable deformation during network training and increase the objective function loss term. At the same time, it can make the deformed mesh smoother. The specific form of the regularization loss term is as follows: ,in represents the displacement vector field of the mesh control points, represents the L2 norm; The purpose of the cyclic loss term is to transform the deformed floating image obtained after the deformation transformation back to the original image, that is, floating image-deformed floating image-floating image; this loss term ensures that the deformation registration result has cyclic consistency. The specific form of the cyclic loss term is as follows: ； in and represents two deformation generation networks, represents image A, represents image B, Represents the L1 norm.

6. The multimodal medical image elastic registration system based on CycleGan according to claim 1, characterized in that: The training strategies for medical imaging data of different modalities are as follows: ① Data collection and preprocessing: Collect medical imaging data from different hospitals, different modalities and different anatomical parts, and preprocess the original medical imaging data through batch rigid registration, denoising, contrast enhancement, cropping and filling, and normalization steps; ② Multimodal data fusion: This training strategy first uses multimodal medical imaging data for fusion processing; through appropriate fusion algorithms, image information of different modalities is effectively integrated to extract richer features and improve the model's ability to recognize target structures; ③Model selection and optimization: According to the task requirements and data characteristics, convolutional neural network CNN and generative adversarial network GAN are selected as the training basis. During the training process, the model performance is optimized by adjusting the model's hyperparameters and optimization algorithms; ④ Transfer learning and fine-tuning: This training strategy uses a transfer learning method, using a model pre-trained on a large dataset for initialization, and then fine-tuning it on the target dataset to adapt to the needs of a specific task; ⑤ Loss function and evaluation indicators: In order to measure the performance of the model, select appropriate loss functions and evaluation indicators, evaluate the performance of the model regularly, and adjust the training strategy based on the evaluation results to ensure that the model can be continuously optimized.

Images