Medical image registration method and system based on image-to-image translation

Abstract

The application provides a medical image registration method and system based on image-to-image translation, and belongs to the technical field of medical image registration. The method comprises the following steps: acquiring a linear X-ray image; inputting the linear X-ray image into an image-to-image translation module after processing to obtain a simulated DRR image; inputting the acquired linear X-ray image into a pose initialization module to obtain an initial pose; setting the obtained initial pose as a global variable and inputting the global variable into an optimizer; calculating the image similarity between the simulated DRR image and a standard DRR image after down-sampling the simulated DRR image and the standard DRR image to a certain scale, and inputting the image similarity into the optimizer; constantly updating the pose output by the optimizer; obtaining a final pose after optimization at different scales; and calculating a target registration error based on the final pose.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image registration, and particularly relates to a medical image registration method and system based on image-to-image translation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] 2D / 3D medical image registration is a key technology in image-guided surgery (IGS), and its core goal is to align the preoperative high-resolution 3D computed tomography (CT) image with the intraoperative 2D X-ray image in real time, thereby providing accurate surgical navigation for surgeons.

[0004] However, due to imaging modality differences, initial pose estimation errors, and strict requirements for real-time and high-precision in clinical practice, existing registration methods still face many challenges.

[0005] For example, imaging modality differences make registration difficult: traditional registration methods, such as optimization methods based on normalized cross-correlation (NCC) or mutual information (MI), rely on the intensity similarity between X-ray images and digitally reconstructed radiographs (DRR images) generated by CT. However, X-ray imaging is affected by scattering effects, noise, and low contrast, while DRR images are simulated images generated based on idealized physical models, and there are significant differences in gray scale distribution, noise pattern, and structural details between the two. In actual training, the X-ray image is preprocessed first by applying a logarithmic transformation (logarithm of maximum pixel value minus logarithm of each pixel value) to change the attenuation mode of the X-ray image from nonlinear to linear, and then cropping 50 pixels from the periphery to eliminate the effects of the collimator to ensure compatibility with the DRR image projector. The preprocessed X-ray image is collectively referred to as a linear X-ray image. At the same time, the original DRR image is collectively referred to as a standard DRR image, and the DRR image obtained through the image-to-image translation network is collectively referred to as a simulated DRR image. The image obtained by data augmentation from the standard DRR image is collectively referred to as a simulated X-ray image.

[0006] Regarding unsupervised image-to-image translation methods such as CycleGAN, which do not rely on paired data, structural distortion or artifacts may be introduced during translation, destroying key anatomical features such as bone edges, leading to subsequent registration failure.

[0007] Supervised image-to-image translation methods such as pix2pix rely on paired data, but high-quality X-ray image and DRR image paired dataset in the medical field is scarce, and the cost of manual annotation is extremely high. The difference between imaging modalities makes intensity-based registration algorithms perform poorly in low-contrast areas such as soft tissue or complex anatomical structures such as the pelvis and spine, and the registration error increases.

[0008] There is also the problem of inaccurate initial pose estimation affecting optimization convergence, for example: marker-based methods such as PnP rely on manually annotated 2D-3D marker matching, but intraoperative X-ray images may have occlusions, metal artifacts or missing markers, leading to matching failure. Marker positioning errors will directly affect the pose estimation and affect the registration accuracy.

[0009] Random sampling point-based methods such as SCRNet and RayEmb establish 2D-3D correspondence by regressing scene coordinates or learning ray embedding space, avoiding marker dependence, but have high computational complexity, requiring a large number of sampling points, and are sensitive to image quality. Pose regression-based methods such as PoseNet and DiffPose use convolutional networks to directly regress 6D poses from images, but due to the difference between imaging modalities, models trained using DRR images are difficult to adapt to X-ray images, and the initial pose error is large. Initial pose deviation can cause gradient optimization to fall into local optimum, especially in complex anatomical structures or large pose offset conditions, the registration failure rate increases significantly. Although xvr uses data augmentation to improve the generalization ability of the initial pose estimation of X-ray images in the pose regression process, it performs segmentation and projection on CT in the pose optimization process, increasing the modal difference between DRR images and X-ray images, and the registration accuracy needs to be further improved.

[0010] In addition, there is a problem of low optimization efficiency, which is difficult to meet the real-time demand. Traditional optimization, such as CMA-ES, adopts a random search strategy, which can expand the capture range, but the convergence speed is slow, and the time consumption of single registration can reach several minutes, which cannot meet the real-time demand in the operation. Gradient optimization, such as DiffDRR, accelerates gradient calculation through differentiable rendering, but is still limited by local optimization problems and is sensitive to the initial pose. Specifically, the Trilinear projection method used by DiffDRR is a DRR image projection method based on trilinear interpolation. By uniformly sampling three-dimensional space points along the ray and interpolating the adjacent voxel values, the cumulative attenuation effect of X-rays passing through the CT volume is simulated. This method discretizes the ray into multiple sampling points, uses the grid_sample function of PyTorch for differentiable trilinear interpolation, and finally integrates (sums or takes the maximum value) along the ray direction to generate a DRR image. Compared with accurate ray tracing such as Siddon method, it has higher computational efficiency and supports gradient propagation, which is suitable for joint optimization with deep learning models, but the accuracy is slightly lower than the physical-based ray tracing method.

[0011] There is also a problem of low registration accuracy. Single-scale registration relies only on a single resolution or fixed receptive field, making it difficult to capture both global spatial structure consistency (such as large-scale displacement or rotation) and local fine deformation accuracy (such as tissue edges or texture details). This limitation leads to deficiencies in global consistency or local accuracy in the registration results, affecting the overall registration accuracy.

[0012] Therefore, the current 2D / 3D medical image registration has the problems of inaccurate registration due to imaging modality differences, inaccurate initial pose estimation of images due to markers, and low optimization efficiency and registration accuracy in the pose optimization process. SUMMARY

[0013] To overcome the shortcomings of the prior art, the present application provides a medical image registration method and system based on image-to-image translation, which integrates image-to-image translation, pose initialization and optimization into a unified framework, reduces the time-consuming problem of random search in traditional methods, and realizes real-time and high-precision registration.

[0014] To achieve the above purpose, one or more embodiments of the present application provide the following technical solutions:

[0015] In a first aspect, a medical image registration method based on image-to-image translation is disclosed, comprising:

[0016] Obtaining a linear X-ray image, processing the linear X-ray image and inputting it into an image-to-image translation module to obtain a simulated DRR image;

[0017] Inputting the linear X-ray image into a pose initialization module to obtain an initial pose;

[0018] The obtained initialization pose is set as a global variable and input into the optimizer, the simulated DRR image obtained by translation and the standard DRR image obtained by projection are down-sampled to a certain scale, the image similarity between the images is calculated and input into the optimizer, the pose output by the optimizer is continuously updated, and finally the pose is obtained after optimization at different scales, and the target registration error is calculated based on the final pose.

[0019] As a further technical solution, the image-to-image translation module adopts a generative adversarial network, including a generator and a discriminator.

[0020] In the training stage, the input of the generator is the linear X-ray image processed by the mask, and the input of the discriminator is the standard DRR image obtained by segmenting and projecting the CT volume under the real pose.

[0021] The generator translates the input linear X-ray image into a simulated DRR image, and the discriminator judges the true or false of the simulated DRR image translated by the generator according to the standard DRR image.

[0022] As a further technical solution, the image-to-image translation module adopts multi-scale discriminator joint optimization, and through adversarial loss, feature matching loss and VGG perception loss, it ensures that the translated simulated DRR image retains key anatomical structure features.

[0023] As a further technical solution, the pose initialization module samples the real pose in the pre-defined pose range in the training stage, then obtains the standard DRR image by projecting the real pose, and then obtains a large number of simulated X-ray images by data augmentation operation on the standard DRR image, and then inputs the simulated X-ray images into the pose regressor to obtain the predicted pose, and then obtains the standard DRR image by projecting the predicted pose.

[0024] As a further technical solution, the pose initialization module finally optimizes the multi-scale normalized cross-correlation loss and the manifold-based geometric constraint loss by self-supervised learning in the training stage, so that the pose regressor outputs the predicted initialization pose.

[0025] In a second aspect, a medical image registration system based on image-to-image translation is disclosed, comprising:

[0026] An image-to-image translation module configured to: receive an input linear X-ray image, and obtain a simulated DRR image after processing the linear X-ray image.

[0027] A pose initialization module configured to: process the obtained linear X-ray image to obtain an initialization pose.

[0028] The pose optimization module is configured to: set the obtained initial pose as a global variable and input into an optimizer, calculate the image similarity between the simulated DRR image and the standard DRR image after down-sampling to a certain scale, and input into the optimizer, constantly update the pose output by the optimizer, obtain the final pose after optimization at different scales, and calculate the target registration error based on the final pose.

[0029] The above one or more technical solutions have the following beneficial effects:

[0030] In view of the problem that imaging modal difference leads to registration difficulty: the image-to-image translation module is designed in the technical solution of the embodiment, the neural network used by the module is an improved high-resolution image-to-image translation network, the core of which is composed of a generator and a discriminator, and is specially used for translating linear X-ray images into simulated DRR images. The global generator improves the detail fidelity, and multiple sub-discriminators judge the image authenticity at different resolutions, forcing the generator to retain the spatial consistency of the anatomical structure, which helps to achieve accurate registration subsequently.

[0031] The technical solution of the embodiment compensates for the imaging modal difference between the linear X-ray image and the standard DRR image through image-to-image translation, thereby improving the robustness of the registration algorithm based on intensity similarity, and achieving more accurate registration in low-contrast areas (soft tissue) and complex anatomical structures (pelvis, spine).

[0032] In view of the problem that inaccurate initial pose estimation affects optimization convergence: the pose initialization module is designed in the technical solution of the embodiment, the pose regressor used by the module is obtained by jointly optimizing the image multi-scale normalized cross-correlation loss and the manifold-based geometric constraint loss through self-supervised learning, and does not need to rely on artificial marking points, thereby reducing the matching failure caused by metal artifacts or occlusion.

[0033] In view of the problem that the optimization efficiency and registration accuracy are not high in the pose optimization process: the pose optimization module is designed in the technical solution of the embodiment, the multi-scale optimization strategy used by the module realizes fast global pose estimation at a coarse scale, and then accurately optimizes high-frequency details such as bone microstructure at a fine scale, so as to significantly improve the registration efficiency while ensuring higher registration accuracy.

[0034] The technical solution of the embodiment integrates the image-to-image translation, pose initialization and optimization modules into a unified framework, reduces the time-consuming of random search of traditional methods such as CMA-ES, and realizes real-time and high-precision registration.

[0035] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated herein by reference. The illustrations are shown schematically and are not intended to limit the application.

[0037] Figure 1 CT visualization structure chart and label under 3D Slicer;

[0038] Figure 2 Network structure chart of PixPose of the embodiment of the application;

[0039] Figure 3 Simulation X-ray image example after data enhancement of the pose initialization module of the PixPose network structure;

[0040] Figure 4 Example chart of each module of the PixPose network structure;

[0041] Figure 5 TRE visualization chart of different methods on the DeepFluoro dataset;

[0042] Figure 6 Registration time visualization chart of different methods on the DeepFluoro dataset;

[0043] Figure 7 Real landmark and predicted landmark visualization chart of different methods on the DeepFluoro dataset. DETAILED DESCRIPTION

[0044] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0045] It should be noted that the terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the application.

[0046] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0047] Embodiment one

[0048] The key technical improvement of the embodiment is to construct a 2D / 3D medical image registration framework PixPose. Specifically, a medical image registration method based on image-to-image translation is disclosed, which comprises:

[0049] Step one: obtain a linear X-ray image, input the processed linear X-ray image into an image-to-image translation module to obtain a simulated DRR image;

[0050] Step two: input the obtained linear X-ray image into a pose initialization module to obtain an initialized pose;

[0051] Step three: set the initialized pose as a global variable and input it into an optimizer, downsample the simulated DRR image and the standard DRR image to a certain scale, calculate the image similarity between the images, and input it into the optimizer, constantly update the pose output by the optimizer, after optimization at different scales, obtain a final pose, and calculate the target registration error based on the final pose.

[0052] In an embodiment, regarding the imaging modality difference problem in step one, an image-to-image translation module based on a pix2pixHD network is designed. The network used in this module is an improved high-resolution image-to-image translation network, which consists of a generator and a discriminator. The generator is used to translate linear X-ray images into simulated DRR images. The generator adopts a multi-scale global-local structure. The global generator extracts overall features through a U-Net structure of down-sampling-residual block-up-sampling, while the local enhancer refines the features layer by layer based on low-resolution features to improve the detail fidelity. The discriminator adopts a multi-scale PatchGAN architecture, which judges the authenticity of the image at different resolutions through multiple sub-discriminators, forcing the generator to retain the spatial consistency of anatomical structures.

[0053] In the training phase, as shown in module I black line, Figure 2 The input of this module is a linear X-ray image and a standard DRR image with the same pose. In order to adapt to the input of the pix2pixHD network, the image size is down-sampled from 1436 x 1436 to 512 x 512.

[0054] In order to fully utilize the characteristics of the DeepFluoro dataset, this module further performs mask processing on the linear X-ray image and segment projection on the CT volume, reducing the influence of the pelvic anatomical structure on image-to-image translation. Since the labels of the linear X-ray image mask are 1 to 6, the labels reserved by the segmentation (Segment, S) are also set to 1 to 6, as shown in Figure 1 .

[0055] The generator (Generator, G) translates the input linear X-ray image into a simulated DRR image G fakeThe discriminator (D) is used to deceive the G-factor. The discriminator determines the G-factor based on the segmented CT volume (V) and the standard DRR image generated by the true pose transformation (T) projection. fake Is it true or false?

[0056] In terms of specific implementation details, this module adopts multi-scale discriminator joint optimization. Through the adversarial loss corresponding to the following formula (1), the feature matching loss corresponding to the formula (2), and the VGG perceptual loss corresponding to the formula (3), it ensures that the final generated simulated DRR image has both high resolution and structural accuracy, providing high-quality input for subsequent registration.

[0057] During the testing phase, see the red solid line in Module I for details, such as... Figure 2 As shown, a simulated DRR image obtained by directly inputting the masked linear X-ray image and translating it into a standard DRR image can be generated. This design reduces the imaging modal difference between the linear X-ray image and the standard DRR image during pose optimization, significantly improving registration accuracy.

[0058] Regarding the losses incurred in combat:

[0059] ;

[0060] in, It is a linear X-ray image. It is a standard DRR image. It is a simulated DRR image. This represents the last layer of different discriminators. This indicates the probability that the discriminator classifies a standard DRR image as true. This indicates the probability that the discriminator classifies a simulated DRR image as true. Indicates the number of discriminators. Indicates the weights of different discriminators. This represents the average loss over a set of images. This loss function is trained adversarially between the generator and discriminator to ensure that the generated simulated DRR images are visually indistinguishable from standard DRR images.

[0061] Regarding feature matching loss:

[0062] ;

[0063] in, It is a linear X-ray image. It is a standard DRR image. It is a simulated DRR image. The first distinct discriminator represents the second distinct discriminator. layer, This indicates that the discriminator judges the standard DRR image as true. Feature map of the layer This indicates that the discriminator classifies the simulated DRR image as true. Feature map of the layer This represents the weights of different layers of the discriminator. This represents the total number of layers in each discriminator. Indicates the number of discriminators. Indicates the weights of different discriminators. This represents the average loss over a set of images. This loss function constrains the generator to preserve the structural features of the images and avoids deviations between simulated DRR images and standard DRR images in deep features by matching feature maps (such as edges and textures) in the intermediate layers of the discriminator.

[0064] Regarding VGG perceived loss:

[0065] ;

[0066] in, It is a linear X-ray image. It is a standard DRR image. It is a simulated DRR image. This indicates the VGG network's... layer, This indicates the simulated X-ray image in the VGG network. Feature map of the layer This indicates the standard DRR image in the VGG network. Feature map of the layer This represents the different weights of each layer in the VGG network. This indicates the total number of layers in the VGG network. This represents the average loss over a set of images. This loss function uses a pre-trained VGG network to extract high-level semantic features, further aligning the global semantic content, such as organ shape and spatial layout, between simulated DRR images and standard DRR images.

[0067] In one implementation example, to address the issue of inaccurate initial pose estimation in step two, a pose initialization module based on a pose regressor was designed. This pose regressor uses ResNet-18, a classic deep convolutional neural network architecture primarily used to solve the vanishing gradient problem during deep network training. This network contains 18 weighted layers, and its core innovation is the introduction of residual connections (skip connections), allowing gradients to propagate directly back to shallower layers, thus enabling efficient training of deeper networks. ResNet-18 serves as the backbone network for feature extraction. Features extracted from its last layer are decoded into rotation (3 degrees of freedom) and translation (3 degrees of freedom) parameters by two linear layers, achieving end-to-end pose regression.

[0068] During the training phase, such as Figure 2 As shown in the black solid line of module II, this module first samples the three translational components and three rotational components within a predefined pose range to obtain the true pose. Then, the DRR image is obtained through projection. Then, a large number of simulated X-ray images are obtained through data augmentation (DA) operations. ,like Figure 3 As shown, the predicted pose is then obtained by inputting the pose regressor (P). Then, the DRR image is obtained through projection. Finally, the multi-scale normalized cross-correlation loss corresponding to formula (4) and the geometric constraint loss based on the SE(3) manifold corresponding to formula (5) are jointly optimized through self-supervised learning.

[0069] During the testing phase, such as Figure 2 As shown in the red solid line of Module II, the initial pose can be obtained by directly inputting a linear X-ray image. This design avoids the reliance on traditional marker point methods, generates synthetic training data through differentiable rendering, and simultaneously performs random data augmentation operations on the synthetic training data to obtain simulated X-ray images, thereby improving the generalization ability and robustness of the initial pose estimation.

[0070] Regarding multi-scale normalized cross-correlation loss:

[0071] ;

[0072] in, and These represent the normalized simulated DRR image patch and the standard DRR image patch at a specific scale, respectively. Pixel value at pixel coordinates Indicates the size of the image patch. Represents weights at different scales. This represents the average loss over a set of images. This loss function measures the local similarity between simulated DRR images and standard DRR images at different scales, and optimizes the pose regressor through normalized cross-correlation.

[0073] Geometric constraint loss based on SE(3) manifold:

[0074] ;

[0075] in, and These are the true poses The rotation and translation components, and are the rotational and translational components of the predicted pose respectively, denotes the focal length of the projection, denotes the similarity between a set of poses. This loss function directly constrains the distance between the predicted pose and the ground truth pose on the SE(3) manifold, ensuring the physical plausibility of the rotational and translational components.

[0076] In one embodiment, to address the problem of low optimization efficiency and registration accuracy in the pose optimization process in step three, a pose optimization module based on a multi-scale strategy is designed. The core of this strategy is to balance the efficiency and accuracy of the registration process by gradually reducing the image resolution: fast and rough alignment at high resolution levels (small scale) to capture large-scale transformations; fine adjustment at low resolution levels (large scale) to optimize local details. This strategy effectively expands the optimization convergence domain, avoids local optimal solutions, and significantly improves computational efficiency. In specific implementation, the learning rate is dynamically adjusted at each scale level, and an early stopping mechanism is adopted to ensure optimal registration results under limited computational resources.

[0077] This module is directly used in the test phase, as shown in Figure 2 , see module III red solid line, the linear X-ray image is input into the trained pose regressor in step two to obtain the initial pose , then set as a global variable and input it into the Adam optimizer. The masked linear X-ray image in step one is translated into a simulated DRR image by the trained generator, and the DRR image generated by the projection of the segmented CT volume in step three is downsampled to a certain scale, and then the image similarity (Similarity, Sim) between the corresponding images according to formula (7) is calculated to continuously update the pose After optimization at different scales, the final pose is obtained, and the target registration error (Target Registration Error, TRE) corresponding to formula (8) is calculated.

[0078] In terms of specific implementation details, since the preoperative CT and intraoperative X-ray images are obtained independently, the patient's femur may move slightly during this process, so the segmented femur blocks of label 5 and label 6 are excluded when using segmentation projection.

[0079] Gradient normalization cross-correlation similarity calculation is introduced in the pose optimization module:

[0080] ;

[0081] wherein, and​ respectively represent the normalized simulated DRR image and the standard DRR image after gradient operator processing in pixel value in pixel coordinates, and represent the size of the image, represent the similarity of a group of images. The loss function enhances the alignment of image edges through the gradient operator, improving the accuracy of pose optimization.

[0082] Regarding the similarity calculation between images:

[0083] ;

[0084] wherein, and respectively represent the simulated DRR image and the standard DRR image of different scales, represent the size of the image block, represent the weight of different scales, represent the loss of a group of images. This similarity calculation combines multi-scale NCC and gradient NCC to construct a pyramid optimization target, achieving fine-to-coarse pose adjustment.

[0085] Regarding the target registration error calculation:

[0086] ;

[0087] wherein, represent the 3D coordinates of the marker points, and respectively represent the true pose and the predicted pose, represent the number of marker points. This evaluation index calculates the Euclidean distance of the marker points under the transformation of the true pose and the predicted pose.

[0088] In the technical solution of the present embodiment, the entire process integrates three sub-modules: image-to-image translation (Module I), pose initialization (Module II), and optimization (Module III), ultimately maintaining sub-millimeter-level accuracy (5% quantile error <1mm) while controlling the registration time to 3.2 seconds.

[0089] The technical solution of the present embodiment uses pix2pixHD network for image-to-image translation, combining multi-scale discriminators, adversarial loss, feature matching loss, and VGG perceptual loss to ensure that the generated DRR image retains key anatomical structures such as bone edges, avoiding the structural distortion problem of unsupervised image-to-image translation methods such as CycleGAN.

[0090] The technical solution of the embodiment introduces a differentiable rendering technique (such as DiffDRR) to generate standard DRR images, and improves the generalization ability of the pose regressor to estimate the pose of the linear X-ray image through data enhancement, thereby avoiding the initial pose error caused by the difference in imaging modalities in the traditional method.

[0091] In view of the problems of low optimization efficiency and registration accuracy in the pose optimization process, the technical solution of the embodiment proposes a multi-scale optimization strategy to perform fast global registration at a coarse scale and accurate optimization of high-frequency details such as bone microstructure at a fine scale, thereby improving the registration efficiency and accuracy.

[0092] In an embodiment, through hardware acceleration (such as GPU parallel computing) and lightweight network design, efficient deployment in a clinical environment (such as a surgical navigation system) is ensured, and process interruption caused by manual intervention is avoided.

[0093] In an embodiment, a high-quality linear X-ray image and standard DRR image pairing dataset construction method is first proposed: the existing DeepFluoro dataset is 6 pelvis CT scans, corresponding segmentation, and each CT has a number of original X-ray images, corresponding mask images and real poses. On this basis, a standard DRR image is generated using the real pose corresponding to each linear X-ray image and the CT volume projection. Since linear X-ray images are relatively scarce, first, all linear X-ray images and standard DRR images of each CT are separately formed into a test set, and the remaining all linear X-ray images and standard DRR images of all CTs are formed into a training set.

[0094] The PixPose framework proposed by the technical solution of the embodiment is compared with the prior art as follows: Figure 4 The visual effect comparison of the dataset samples in the preprocessing, image-to-image translation and registration stages is shown. The images registered using the PixPose method are almost completely aligned with the linear X-ray images. Figure 5 Table 1 shows the 2D / 3D registration performance comparison of various methods on the DeepFluoro dataset, covering various accuracy indicators (including 5th, 50th, 95th percentile errors, success rates at different thresholds) and registration time. PixPose shows obvious advantages in many key indicators, especially in tasks with high accuracy requirements. For example, on all sub-datasets, the 5th percentile error and failure rate at a 1mm threshold of PixPose are significantly lower than those of other methods, showing excellent robustness and accuracy. In addition, the method also achieves or approaches the best results in terms of 50th, 95th percentile errors and failure rates at 10mm and 5mm thresholds, while maintaining a relatively low running time, such as Figure 6As shown, it is usually about 3.2 seconds, which demonstrates the efficiency and real-time performance in actual deployment. These results highlight the significant advantages of image translation technology in solving the problem of differences between linear X-ray images and standard DRR image imaging modalities. Figure 7 The visualization comparison of the predicted landmark points and the real landmark points after registration of each method shows that the predicted landmark points obtained by PixPose are closest to the real landmark points.

[0095] Table 1

[0096]

[0097] Embodiment Two

[0098] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0099] Embodiment Three

[0100] The purpose of this embodiment is to provide a computer-readable storage medium.

[0101] A computer-readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to perform the steps of the above method.

[0102] Embodiment Four

[0103] The purpose of this embodiment is to provide a medical image registration system based on image-to-image translation, which includes:

[0104] An image-to-image translation module configured to receive an input linear X-ray image and obtain a simulated DRR image by processing the linear X-ray image.

[0105] A pose initialization module configured to process the obtained linear X-ray image to obtain an initialization pose.

[0106] A pose optimization module configured to set the obtained initialization pose as a global variable, input it into an optimizer, calculate the image similarity between the simulated DRR image and the standard DRR image after down-sampling them to a certain scale, and input it into the optimizer to constantly update the pose output by the optimizer, obtain a final pose after optimization at different scales, and calculate a target registration error based on the final pose.

[0107] Embodiment Five

[0108] The purpose of the embodiment is to provide a computer program product containing instructions, which, when running on a computer, causes the computer to perform the method and functions involved in any of the above embodiments.

[0109] The steps involved in the device of the above embodiment correspond to the method embodiment one, and the specific implementation can refer to the relevant description part of embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0110] Those skilled in the art should understand that the above modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0111] Although the specific embodiments of the present application are described above in combination with the drawings, it is not a limitation on the scope of protection of the present application, and those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A method of medical image registration based on image-to-image translation, characterized in that, Comprising: A linear X-ray image is acquired, processed, and then input into an image-to-image translation module to obtain a simulated DRR image. The image-to-image translation module employs a multi-scale discriminator PatchGAN architecture for joint optimization, using adversarial loss, feature matching loss, and VGG perceptual loss to ensure that the generated simulated DRR image retains key anatomical features. Adversarial loss: Feature matching loss: VGG perceived loss: ;in, It is a linear X-ray image. It is a standard DRR image. It is a simulated DRR image. This represents the last layer of different discriminators. This represents the probability that the discriminator classifies a simulated DRR image as true. This indicates that the discriminator judges the standard DRR image as true. Feature map of the layer This represents the total number of layers in each discriminator; The first distinct discriminator represents the third distinct discriminator. layer, This indicates the probability that the discriminator classifies a standard DRR image as true. This indicates that the discriminator classifies the simulated DRR image as true. Feature maps of layers Indicates the number of discriminators. Indicates the weights of different discriminators. This represents the average loss over a set of images; This indicates the VGG network's... layer, This indicates the simulated X-ray image in the VGG network. Feature map of the layer This indicates the standard DRR image in the VGG network. Feature map of the layer This represents the different weights of each layer in the VGG network. This indicates the total number of layers in the VGG network; The acquired linear X-ray image is input into a pose initialization module to obtain an initialized pose; in the training stage, the pose initialization module samples a real pose in a predefined pose range, and then obtains a DRR image by projecting the real pose The DRR image is compared with the acquired linear X-ray image A simulation X-ray image is obtained through a data enhancement operation, and then the simulation X-ray image is input into a pose regressor to obtain a predicted pose, and then a standard DRR image is obtained by projecting the predicted pose; The obtained initialization pose is set as a global variable and input into the optimizer, the simulated DRR image and the standard DRR image are down-sampled to a certain scale, the image similarity between the images is calculated and input into the optimizer, the pose output by the optimizer is constantly updated, the final pose is obtained after optimization at different scales, and the target registration error is calculated based on the final pose; the pose optimization module based on the multi-scale strategy is adopted to perform fast and rough alignment at a high resolution level; and fine adjustment is performed at a low resolution level.

2. The image-to-image translation based medical image registration method of claim 1, wherein, The image-to-image translation module adopts a generative adversarial network, including a generator and a discriminator; In the training stage, the input of the generator is a linear X-ray image processed by a mask, and the input of the discriminator is a standard DRR image obtained by segmenting and projecting a CT volume under a real pose; The generator translates the input X-ray image into a simulated DRR image, and the discriminator judges the authenticity of the simulated DRR image translated by the generator according to the standard DRR image.

3. The image-to-image translation based medical image registration method of claim 1, wherein, The pose initialization module jointly optimizes the multi-scale normalized cross-correlation loss and the manifold-based geometric constraint loss through self-supervised learning at the end of the training stage, so that the pose regressor outputs a predicted initialization pose.

4. A medical image registration system based on image-to-image translation, characterized in that, Comprising: The image-to-image translation module is configured to: receive an input linear X-ray image, process the linear X-ray image to obtain a simulated DRR image; the image-to-image translation module employs a multi-scale discriminator PatchGAN architecture for joint optimization, using adversarial loss, feature matching loss, and VGG perceptual loss to ensure that the generated simulated DRR image retains key anatomical features; adversarial loss: Feature matching loss: VGG perceived loss: ;in, It is a linear X-ray image. It is a standard DRR image. It is a simulated DRR image. This represents the last layer of different discriminators. This represents the probability that the discriminator classifies a simulated DRR image as real. This indicates that the discriminator judges the standard DRR image as true. Feature map of the layer This represents the total number of layers in each discriminator; The first distinct discriminator represents the second distinct discriminator. layer, This indicates the probability that the discriminator classifies a standard DRR image as true. This indicates that the discriminator classifies the simulated DRR image as true. Layer feature map, Indicates the number of discriminators. Indicates the weights of different discriminators. This represents the average loss over a set of images; This indicates the VGG network's... layer, This indicates the simulated X-ray image in the VGG network. Feature map of the layer This indicates the standard DRR image in the VGG network. Feature map of the layer This represents the different weights of each layer in the VGG network. This indicates the total number of layers in the VGG network; The pose initialization module is configured to process the acquired linear X-ray image to obtain an initialization pose; in the training stage, the pose initialization module samples a predefined pose range to obtain a real pose, and then projects the real pose to obtain a DRR image The DRR image is projected The simulation X-ray image is obtained through a data enhancement operation, and then the simulation X-ray image is input into the pose regressor to obtain a predicted pose, and then the predicted pose is projected to obtain a standard DRR image; The pose optimization module is configured to: set the obtained initialization pose as a global variable, input into the optimizer, down-sample the simulated DRR image and the standard DRR image to a certain scale, calculate the image similarity between the images and input into the optimizer, constantly update the pose output by the optimizer, obtain the final pose after optimization at different scales, and calculate the target registration error based on the final pose; adopt the pose optimization module based on the multi-scale strategy to perform fast and rough alignment at a high resolution level; and perform fine adjustment at a low resolution level.

5. The image-to-image translation based medical image registration system of claim 4, wherein, The image-to-image translation module adopts a generative adversarial network, including a generator and a discriminator; In the training stage, the input of the generator is a linear X-ray image processed by a mask, and the input of the discriminator is a standard DRR image obtained by segmenting and projecting a CT volume under a real pose; The generator translates the input X-ray image into a simulated DRR image, and the discriminator judges the authenticity of the simulated DRR image translated by the generator according to the standard DRR image.

6. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 3.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the method of any one of claims 1 to 3.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to perform the steps of the method of any one of claims 1 to 3.

Images