Ankle joint image registration method and system based on fused three-dimensional CT data

Abstract

The present invention discloses a method and system for ankle joint image registration based on fused three-dimensional CT data. The method comprises: performing grayscale uniform processing on the ankle joint X-ray image based on a trained CycleGAN network model and in combination with an ankle joint DRR image with label information to obtain an ankle joint X-ray image after grayscale uniformization; performing inference prediction on the ankle joint X-ray image after grayscale uniformization based on a trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image; performing two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters; and realizing ankle joint image registration based on the predicted affine transformation parameters. The present invention can realize fast and accurate ankle joint image registration. As an ankle joint image registration method and system based on fused three-dimensional CT data, the present invention can be widely used in the field of image registration technology.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular to an ankle joint image registration method and system based on fused three-dimensional CT data. Background Art

[0002] Combining computer technology and imaging data, the patient's bone structure is 3D modeled to help doctors plan the surgical process, so that a large amount of X-ray transmission is not required during the operation, thereby greatly reducing the radiation dose. The basic workflow of the orthopedic surgical navigation system is to obtain 3D data of the patient's corresponding bones through CT scanning before the operation and build a complete 3D model. During the operation, the system collects 2D X-ray data in real time and aligns it with the 3D CT data obtained before the operation to establish a correspondence between the digital space and the real space. Subsequently, the system will track the position of the navigation point (i.e., the surgical tool) in real time and render it in the digital space to help doctors accurately locate and operate. However, the image feature-based method in the related art uses artificial markers or inherent features inside and outside the patient's body for positioning. For example, skin markers or titanium nails are attached to the human body before surgery. While this method is fast and accurate, it is extremely damaging to the human body and does not conform to the principles of minimally invasive surgery. Other external marker methods also share this issue, making them difficult to meet the demands of clinical surgery. 2D / 3D registration methods based on image grayscale first employ digital radiographic reconstruction (DRR) to project 3D images into 2D images to unify the dimensions. Similarity measures between the DRR and the X-ray are then calculated, and the optimal registration result is achieved through iterative optimization. However, purely grayscale-based iterative methods are significantly slower than clinical surgical requirements. Furthermore, deep learning-based medical image registration can be broadly categorized into three types: deep iterative, supervised, and unsupervised, depending on the type and training scheme. Deep iterative methods utilize only neural networks to extract image features and learn similarities, still using traditional iterative calculation methods, but they also suffer from slow speed. Supervised learning methods use the true transformation parameters as a gold standard to train the network and regress the transformation parameters, requiring a large amount of labeled data for training. Unsupervised learning optimizes registration results through its own loss function or adversarial networks, and can adapt to different types and modalities of problems. However, the lack of labels makes the results difficult to evaluate and interpret, and requires a large amount of data to learn effective feature representations, while the amount of medical data is often insufficient to support training. In summary, deep learning-based methods in the context of orthopedic surgical navigation tasks generally suffer from insufficient effectiveness and generalization due to the lack of large labeled datasets. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide an ankle joint image registration method and system based on fusion of three-dimensional CT data, which can achieve fast and accurate ankle joint image registration.

[0004] The first technical solution adopted by the present invention is: a method for ankle joint image registration based on fusion of three-dimensional CT data, comprising the following steps:

[0005] Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is grayscale unified to obtain the ankle joint X-ray image after grayscale unification;

[0006] Based on the trained image segmentation network model, the ankle joint X-ray image with unified grayscale is inferred and predicted to obtain the segmentation label information of the ankle joint X-ray image;

[0007] Based on the segmentation label information of the ankle joint X-ray image, a two-stage image registration process is performed to obtain the predicted affine transformation parameters;

[0008] Ankle joint image registration based on predicted affine transformation parameters.

[0009] Furthermore, the step of performing grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image to obtain the ankle joint X-ray image after grayscale uniformization specifically includes:

[0010] Obtain ankle X-rays;

[0011] Acquire ankle joint CT images and segment them according to the various bone parts to obtain ankle joint CT images with label information;

[0012] The ankle joint CT image with label information is projected by digital reconstruction radiographic imaging method to construct the ankle joint DRR image with label information;

[0013] Combine ankle joint DRR images with label information and ankle joint X-ray images to construct an ankle joint training dataset;

[0014] The improved CycleGAN network model is trained based on the ankle joint training dataset to obtain a trained CycleGAN network model;

[0015] Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is grayscale unified to obtain the ankle joint X-ray image with unified grayscale.

[0016] Furthermore, the improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module, and the loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function, wherein the output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, the output end of the attention mechanism module is connected to the input end of the first discriminator and the input end of the second discriminator, and the output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module.

[0017] Furthermore, the step of training the improved CycleGAN network model based on the ankle joint training dataset to obtain the trained CycleGAN network model specifically includes:

[0018] Input the ankle joint training dataset into the improved CycleGAN network model;

[0019] Based on the first generator of the improved CycleGAN network model, the ankle joint X-ray image is processed by image style conversion to obtain an ankle joint X-ray image with DRR style;

[0020] Based on the second generator of the improved CycleGAN network model, the ankle joint DRR images with label information are processed by image style conversion to obtain ankle joint DRR images with X-ray style;

[0021] Based on the attention mechanism module of the improved CycleGAN network model, the features of the ankle joint X-ray image with DRR style and the ankle joint DRR image with X-ray style are enhanced to obtain the enhanced ankle joint X-ray image with DRR style and the enhanced ankle joint DRR image with X-ray style;

[0022] Based on the first discriminator of the improved CycleGAN network model, the enhanced ankle joint X-ray image with DRR style is judged to obtain the first judgment result;

[0023] Based on the second discriminator of the improved CycleGAN network model, the enhanced ankle joint DRR image with X-ray style is judged to obtain the second judgment result;

[0024] Based on the loss function module of the improved CycleGAN network model, the first judgment result and the second judgment result are restored, and the trained CycleGAN network model is output.

[0025] Furthermore, the step of performing inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image specifically includes:

[0026] The image segmentation network model is pre-trained using ankle joint DRR images with label information to obtain a trained image segmentation network model;

[0027] The ankle joint X-ray image with unified grayscale is input into the trained image segmentation network model for inference and prediction to obtain the segmentation label information of the ankle joint X-ray image.

[0028] Furthermore, the step of performing a two-stage image registration process based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters specifically includes:

[0029] Perform coarse registration of the segmented label information of the ankle X-ray image with the ankle CT image with label information to obtain preliminary predicted affine transformation parameters;

[0030] Apply the preliminary predicted affine transformation parameters to the ankle joint CT image for DDR projection processing to obtain the predicted ankle joint DRR image;

[0031] Calculate the similarity between the predicted ankle joint DRR image and the ankle joint X-ray image, and judge the calculation results;

[0032] If the calculation result does not meet the preset accuracy requirements, the preliminary predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization registration;

[0033] Until the calculation result meets the preset accuracy requirement, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output.

[0034] Furthermore, the step of performing grayscale iterative registration on the preliminary predicted affine transformation parameters specifically includes:

[0035] Extract the rotation angle and displacement value based on the preliminary predicted affine transformation parameters and construct the transformation matrix;

[0036] Perform inverse transformation on the transformation matrix, transform the virtual point light source, and perform matrix multiplication calculation with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array;

[0037] Generate a new ankle joint DRR image according to the transformed virtual light source point position and the new DRR physical coordinate array;

[0038] Convert the ankle joint X-ray image into an image array and calculate the pixel gradient value with the new ankle joint DRR image to obtain the gradient direction measurement value and gradient median value;

[0039] If the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirement, the grayscale iterative registration is performed cyclically until the gradient direction measurement value and the gradient median value meet the preset similarity value requirement.

[0040] The second technical solution adopted by the present invention is: an ankle joint image registration system based on fusion of three-dimensional CT data, comprising:

[0041] The first module is used to perform grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and the ankle joint DRR image with label information to obtain a grayscale uniform ankle joint X-ray image;

[0042] The second module is used to perform inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain the segmentation label information of the ankle joint X-ray image;

[0043] The third module is used to perform two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain the predicted affine transformation parameters;

[0044] The fourth module is used to realize ankle joint image registration based on predicted affine transformation parameters.

[0045] The beneficial effects of the method and system of the present invention are as follows: the present invention performs grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information, adopts DRR combined with image style uniformity method to replace X-ray for training the network of related deep learning process, does not require the large-scale collection of expensive real clinical image data, greatly reduces the data cost and annotation cost, further based on the trained image segmentation network model, performs inference prediction on the ankle joint X-ray image after grayscale uniformity, obtains the segmentation label information of the ankle joint X-ray image, and finally performs two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image, by performing image style uniformity on the input X-ray image so that its intensity distribution is the same as that of the DRR image, in the first stage registration process, it is beneficial to predict the label information of the X-ray image and reduce the error caused by contrast difference; in the second stage registration, it is beneficial to the grayscale iteration optimization process, reduces the possibility of falling into the local optimal solution, and ultimately improves the image registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the steps of an ankle joint image registration method based on fusion of three-dimensional CT data according to the present invention;

[0047] Figure 2 This is a structural block diagram of an ankle joint image registration system based on fusion of three-dimensional CT data according to the present invention;

[0048] Figure 3 1 is a schematic diagram of a process framework for ankle joint image registration according to a specific embodiment of the present invention;

[0049] Figure 4 Schematic diagram of CycleGAN network training according to a specific embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of a result of unified grayscale style provided by a specific embodiment of the present invention;

[0051] Figure 6 1 is a schematic diagram of pre-training of an image segmentation network model provided by a specific embodiment of the present invention;

[0052] Figure 7 is a schematic diagram of a two-stage image registration provided by a specific embodiment of the present invention;

[0053] Figure 8 It is a schematic diagram of grayscale iterative stage registration provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0055] First, it should be noted that existing methods, such as a 4D joint standing position 3D imaging method and apparatus, utilize a pre-set joint kinematic 2D image sequence and standing position 3D reconstruction data to train a segmentation model for the 3D data and segment the kinematic 2D data sequence to perform 2D / 3D registration and predict the corresponding transformation parameters for each frame. However, given the high cost of medically annotated data, directly predicting the affine transformation parameters for the registration results through deep learning is not feasible, while the speed of grayscale iterative methods is insufficient to meet clinical requirements. Therefore, some studies have attempted to combine the advantages of both methods, using deep learning methods to predict a good initial pose (coarse registration) and then using grayscale iterative processes to optimize the pixel-level pose to achieve accurate results (fine registration). For example, Robert B. Grupp et al. proposed a solution for hip joint registration. However, all of these methods require automatic label detection of intraoperative X-rays, a difficult task that can easily introduce additional errors. Furthermore, generating the DRR is the most time-consuming step in the grayscale iterative algorithm, accounting for approximately 60% of the time.

[0056] Based on this, the embodiment of the present invention provides a two-stage 2D / 3D registration strategy combining deep learning and grayscale iteration. An improved CycleGan image style unification method is designed, such as Figure 3 As shown in the figure, it is used to assist in obtaining 2D image segmentation labels, based on the acquired CT image segmentation labels. An acceleration strategy for the grayscale iteration process in the fine registration phase is designed. This strategy pre-projects the preoperative CT data within a certain range and stores the DRR array instead of the image. During surgery, the corresponding DRR array is retrieved and the similarity measure is calculated, thus omitting the DRR generation step and accelerating the grayscale iteration process. This provides reliable technical support for accurate and rapid 2D / 3D ankle registration.

[0057] Reference Figure 1 The present invention provides an ankle joint image registration method based on fusion of three-dimensional CT data, the method comprising the following steps:

[0058] S100, based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information, performing grayscale uniform processing on the ankle joint X-ray image to obtain a grayscale uniform ankle joint X-ray image;

[0059] Specifically, an ankle joint X-ray image is obtained; an ankle joint CT image is obtained and segmented according to various bone parts to obtain an ankle joint CT image with label information; the ankle joint CT image with label information is projected through a digital reconstruction radiographic imaging method to construct an ankle joint DRR image with label information; the ankle joint DRR image with label information is combined with the ankle joint X-ray image to construct an ankle joint training data set; the improved CycleGAN network model is trained based on the ankle joint training data set to obtain a trained CycleGAN network model; based on the trained CycleGAN network model, the ankle joint X-ray image is grayscale unified in combination with the ankle joint DRR image with label information to obtain an ankle joint X-ray image with unified grayscale.

[0060] In this example, a CT image of the ankle joint is first obtained preoperatively. 3D Slicer software is used to segment the various skeletal components, generating 3D image data of the tibia, fibula, and talus. The segmentation labels are saved, and digitally reconstructed radiographs (DRRs) are used to obtain a large number of DRRs containing segmentation labels. A training dataset is constructed, and intraoperative X-ray images are obtained. These images are then combined with the DRR images for image style unification using an improved CycleGAN network.

[0061] It should be further explained that the training dataset consists of two parts: ankle DRR images obtained by DRR projection of ankle CT images of real clinical patients, and intraoperative X-ray images of real clinical patients. Since intraoperative X-ray images are used, the network model should be trained using ankle imaging data of other patients before surgery. The training dataset includes image pairings of X-ray images and DRR images, as follows:

[0062]

[0063] Each of these is the ankle X-ray image corresponding to the i-th sample.

[0064]

[0065] Each of these is the CT projection image (DRR) corresponding to the i-th sample, generated by the corresponding CT data.

[0066] Therefore, the expression of the total training data set is as follows:

[0067]

[0068] It should be noted that the improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module. The loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function. The output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, and the output end of the attention mechanism module is connected to the input end of the first discriminator and the input end of the second discriminator. The output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module.

[0069] Furthermore, the network pre-training process in the embodiment of the present invention is described. The ankle joint training data set is input into the improved CycleGAN network model. Based on the first generator of the improved CycleGAN network model, the ankle joint X-ray image is subjected to image style conversion processing to obtain an ankle joint X-ray image with a DRR style. Based on the second generator of the improved CycleGAN network model, the ankle joint DRR image with label information is subjected to image style conversion processing to obtain an ankle joint DRR image with an X-ray style. Based on the attention mechanism module of the improved CycleGAN network model, the ankle joint X-ray image with a DRR style is compared with the ankle joint X-ray image with an X-ray style. The joint DRR image is feature enhanced to obtain an enhanced ankle joint X-ray image with DRR style and an enhanced ankle joint DRR image with X-ray style; based on the first discriminator of the improved CycleGAN network model, the enhanced ankle joint X-ray image with DRR style is judged to obtain a first judgment result; based on the second discriminator of the improved CycleGAN network model, the enhanced ankle joint DRR image with X-ray style is judged to obtain a second judgment result; based on the loss function module of the improved CycleGAN network model, the first judgment result and the second judgment result are restored, and the trained CycleGAN network model is output.

[0070] like Figure 4 As shown in the figure, the input stage inputs an ankle X-ray image as the source image for image style unification. The DRR image is input as the target image for image style reference. This is then input into the generator to generate an appropriate output image based on the image content and the target style.

[0071] The X-ray image is input to the generator G. The function of the generator G is to convert the X-ray image into a DRR style image, preserving the structural information and adjusting the style. The generator G outputs a DRR style X-ray image.

[0072] The DRR image is input into the generator F. The function of the generator F is to convert the DRR style image back to the X-ray image style to assist the training process. The generator F outputs the X-ray style DRR image.

[0073] The image output by the generator is processed by an attention mechanism. This module aims to enhance key information areas in the image (such as ankle joint details). It focuses on important parts of the image, retaining these key information while avoiding unnecessary loss of details. This includes both spatial and channel attention mechanisms.

[0074] The role of the spatial attention mechanism is to enhance attention to key areas in the image (such as the center area of ​​the ankle joint). Its input is the generated image, and its output is an image that focuses on important areas.

[0075] The role of the channel attention mechanism is to learn the importance of different channel features and optimize the quality of image conversion. Its input is the generated image features, and its output is the optimized image features.

[0076] After the attention mechanism processes the image, the discriminator evaluates it. It determines whether the generated image matches the target style and compares it with the real image. This allows adversarial training to continuously optimize the generator and produce more realistic images. The discriminators D_A and D_B, respectively, determine whether the styles of the generated X-ray image and DRR image are consistent with the real style. Their inputs are the generated image and the corresponding real image, and their output is the judgment result.

[0077] The loss function module in the figure is an improved loss function module, which includes the following losses:

[0078] Cycle consistency loss: ensures that the generated style conversion image can be restored to the original image. Its input is the generated image and its inverse conversion result. Specifically, it requires that when an image goes from the source style (such as X-ray image) to the target style (such as CT projection image) and then back to the source style, the original image should be restored as much as possible. Assuming that G is a generator from the source style (X-ray image) to the target style (DRR image), F is a generator from the target style (DRR image) to the source style (X-ray image), x is the source image, and y is the target style image, then the cycle consistency loss is defined as:

[0079] L cycle (G,F)=E x～X [||F(G(x))-x||1]+E y～Y [||G(F(y))-y||1]

[0080] Here, ||·||1 represents the L1 norm (absolute error), which is commonly used to measure the pixel-wise difference between images. The first term ensures that the image transformed by the generator G and the inverse generator F is similar to the original image. The second term ensures that the target image transformed by the generator F and the inverse generator G is similar to the original target image.

[0081] Adversarial loss: This is derived from the adversarial training framework of Generative Adversarial Networks (GANs). The discriminator helps the generator produce more realistic images by distinguishing between generated images and real images. For the generator G and the discriminator D, the adversarial loss consists of the generator loss and the discriminator loss. The goal of the generator is to convince the discriminator that the generated images are realistic. The adversarial loss for the generator is as follows:

[0082]

[0083] The goal of the discriminator is to distinguish between generated images and real images. The adversarial loss of the discriminator is as follows:

[0084]

[0085] Where D(x) represents the output of the discriminator for image x, which represents the probability that image x is a real image.

[0086] Style loss: The Gram matrix is ​​used to represent the style features of the image, ensuring that the generated image has consistent texture and tonal features in the target style. The style loss measures the difference between the generated image and the target image at the style level, usually measured using the Euclidean distance (L2 norm).

[0087] Assuming that the features of the generated image G(x) and the target image y are extracted as feature map F through a layer of convolutional neural network, the style loss is defined as:

[0088]

[0089] Where Gram(F) means extracting the style features of the image by calculating the Gram matrix of the feature map F. The Gram matrix is ​​obtained by calculating the inner product between the features of each layer in the feature map F.

[0090] Content loss: Ensures that the generated image is consistent with the original image at the content level. Usually, the VGG network is used to extract high-level features of the image, and then the difference between the generated image and the target image in the feature space is calculated. Assuming φ is the feature map of a layer in the VGG network, the content loss of the generated image G(x) and the target image y is defined as:

[0091]

[0092] Where φ(G(x)) and φ(y) represent the features extracted from the generated and target images at a certain VGG layer, respectively. Content loss measures the difference between the structure and content of an image by calculating the difference between the high-level features of the image, usually using the L2 norm (Euclidean distance).

[0093] The weighting strategy of the loss function is: during the training process, by dynamically adjusting the weights of cycle consistency loss, adversarial loss, style loss and content loss, the model can take into account the fidelity of image content and visual style at the same time during style conversion.

[0094] Through the above methods, CycleGAN can not only accurately unify the visual style of the image in the style transfer task, but also effectively maintain the structural information of the image, especially in the protection of details such as ankle bones in medical images, and improve the effect of unifying the style of ankle X-ray images and DRR images. Figure 5shown.

[0095] S200, based on the trained image segmentation network model, performing inference prediction on the ankle joint X-ray image after grayscale unification to obtain segmentation label information of the ankle joint X-ray image;

[0096] Specifically, if Figure 6 As shown in the figure, the image segmentation network model is pre-trained by the ankle joint DRR image with label information to obtain the trained image segmentation network model; the ankle joint X-ray image with unified grayscale is input into the trained image segmentation network model for inference and prediction to obtain the segmentation label information of the ankle joint X-ray image.

[0097] The ankle CT image is acquired during preoperative scanning and is segmented into four parts using the 3D Slicer software: the tibia, fibula, talus, and the remaining bones (collectively referred to as the sole of the foot). For the 2D / 3D registration task scenario of ankle replacement surgery, the focus is on the tibia area and the area near the tibia-talar junction. The segmentation label information is saved and can be used in subsequent steps to perform coarse registration in combination with the X-ray label information. The CT image is then projected to obtain several DRRs containing segmentation labels to construct a training dataset. The image segmentation network model is trained. When an X-ray image is acquired during surgery, its label information can be inferred. Since there are differences in grayscale and image style between the intraoperative X-ray image and the DRR image used during training, in order to improve the accuracy of reasoning, the X-ray image and the DRR image are first unified in image style as described in step 200. After that, the segmentation label information of each bone corresponding to the X-ray image can be accurately predicted.

[0098] In this embodiment, the DRR technology used is implemented based on the Siddon algorithm and uses a GPU for accelerated calculation. The principle of the Siddon algorithm to generate DRR images is to simulate the projection process of X-rays through ray tracing. Specifically, the algorithm calculates the path of each ray emitted from the projection source as it passes through a three-dimensional volume (such as CT scan data), and accurately calculates the intersection of the ray and each voxel and the length of the ray passed through. Then, based on the density value of each voxel, the attenuation effect of the light in the volume is accumulated, and finally these attenuation values ​​are projected onto a two-dimensional plane to generate a DRR image similar to the actual X-ray projection. This method takes advantage of the efficiency of the Siddon algorithm to quickly determine which voxels are passed by the ray and the contribution of the ray in the voxel, thereby accelerating the generation of DRR.

[0099] S300, performing a two-stage image registration process based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters;

[0100] Specifically, the segmentation label information of the ankle joint X-ray image is roughly aligned with the ankle joint CT image with label information to obtain preliminary predicted affine transformation parameters; the preliminary predicted affine transformation parameters are applied to the ankle joint CT image for DDR projection processing to obtain a predicted ankle joint DRR image; the predicted ankle joint DRR image and the ankle joint X-ray image are calculated for similarity values, and the calculation results are judged; if the calculation results do not meet the preset accuracy requirements, the preliminary predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization alignment; until the calculation results meet the preset accuracy requirements, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output.

[0101] Further, if Figure 7 As shown in the figure, the initial pose parameters of the CT image are set to the result of the first stage of registration, that is, pose parameters that are very close to the correct result. Based on these pose parameters, the position of the initial virtual point light source is set, and the DRR is projected. The similarity measure is calculated and judged with the intraoperative X-ray. When sufficient similarity is found, the registration is stopped and the final pose parameters are output, namely the predicted affine transformation parameters.

[0102] Further, the grayscale iterative registration in this embodiment is explained. The rotation angle and displacement value are extracted according to the preliminary predicted affine transformation parameters to construct a transformation matrix; the transformation matrix is ​​inversely transformed, and the virtual point light source is transformed and matrix multiplication is performed with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array; a new ankle joint DRR image is generated according to the transformed virtual light source point position and the new DRR physical coordinate array; the ankle joint X-ray image is converted into an image array form and the pixel gradient value is calculated with the new ankle joint DRR image to obtain the gradient direction measurement value and the gradient median value; if the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirements, the grayscale iterative registration is performed in a loop until the gradient direction measurement value and the gradient median value meet the preset similarity value requirements.

[0103] The optimization algorithm uses the CMA-ES evolutionary algorithm. The parameter search space is partitioned using the KD tree partitioning algorithm to enable multi-start parallel search, avoiding getting stuck in local optima. The similarity measure uses the gradient orientation measure (GO). Since the first-stage registration already finds pose parameters with approximately accurate values, the parameter search space can be set very small, accelerating registration.

[0104] Each iteration generally performs two tasks: After searching for new affine transformation parameters, a new DRR is generated. This is referred to as the Update module. The Goal Measure (GO) is calculated and evaluated against the new DRR and the intraoperative X-ray. This is referred to as the Metric module.

[0105] The Update module takes more than 60% of the entire second stage registration time. The main steps are as follows: The algorithm extracts the latest affine transformation parameters [rot X ,rot Y ,rot Z ,trans X ,trans Y ,trans Z ], that is, the three rotation angles around the X, Y, and Z axes and the three translation values ​​along the X, Y, and Z axes. Calculate the transformation matrix Tr based on the input rotation angle and displacement value, and calculate its inverse matrix inv T , and convert it to float32 type and assign it to GPU variable; use inv T Transform the virtual point light source self.source to obtain the transformed virtual light source point position source_forGpu; GPU combines the old DRR array with inv T Perform matrix multiplication and other processing with Tn to finally obtain a new DRR physical coordinate array DRRPhy_array, which is flattened into one dimension; generate a new DRR image based on the new virtual light source point position source_forGpu and the new DRR array DRRPhy_array, adjust it to a two-dimensional shape and return it.

[0106] In the Metric stage, the X-ray input is converted into an image array. GPU-accelerated calculations are used to calculate the gradient map (the gradient value of each pixel) of the DRR and X-ray image. The median value is used as the calculation threshold to calculate the GO metric. This stage takes less than 40% of the time.

[0107] This solution designs an acceleration strategy for the grayscale iteration process. During the preoperative preparation phase, the affine transformation parameters are changed in advance with a fine-grained step size. The resulting DRR physical coordinate arrays and their corresponding normalized affine transformation parameters are saved as key-value pairs. The key is the affine parameter, and the value is the DRR physical coordinate array.

[0108] like Figure 8 As shown in the figure, the left side of the dotted line represents the preoperative preparation phase, and the right side of the dotted line represents the grayscale iteration phase during the operation.

[0109] After acquiring the preoperative CT image, a DRR sufficient to cover the affine transformation parameter space is generated based on the planned intraoperative X-ray viewing angle (generally 0° anteroposterior (AP) or 90° lateral (LAT)), using the camera position at that viewing angle as the reference point. The two modules on the left of this figure, Moving the Point Light Source and Generating the DRR, are optional; only the DRR array needs to be generated and stored, not the DRR image, to save space.

[0110] The range of the affine transformation depends on the size and resolution of the CT image. For example, the rotation range of the three rotation axes is set to [-5°, 5°[, with a step size of 2°, meaning each rotation is 2 degrees. The translation range of the three translation axes is set to [-10mm, 10mm], with a step size of 2mm. This example will generate 125,000 different affine transformation parameters and DRR arrays, storing 125,000 key-value pairs. The smaller the step size, the more key-value pairs are stored. The selection of the step size needs to take into account the error tolerance of general registration methods. Based on current research, it is generally considered that for ankle joint tasks, registration is successful when the rotation difference of manual registration is less than 2° and the translation difference is less than 3mm.

[0111] After all DRR arrays are generated, when the intraoperative registration reaches the grayscale iteration stage, the most approximate Key value is searched from the key-value pair array according to the affine transformation parameters currently selected by the CMA-ES evolutionary algorithm, and the DRR array stored in its Value value is extracted. The gradient map and gradient median are calculated together with the X-ray, and the similarity measure GO is calculated. If the threshold is reached, the registration is stopped. Otherwise, the evolutionary algorithm continues to select new affine transformation parameters and continues to select new DRR arrays to repeat the process.

[0112] S400: Implement ankle joint image registration based on predicted affine transformation parameters.

[0113] In summary, the embodiments of the present invention have the following beneficial effects:

[0114] 1) Using DRR combined with an image style unification method instead of X-rays to train the network of related deep learning processes does not require the collection of a large amount of expensive real clinical imaging data, greatly reducing data costs and annotation costs.

[0115] 2) Using the form of CT image projection DRR, the CT image can be directly projected into the two-dimensional DRR, which does not require manual verification and reduces the verification cost.

[0116] 3) Through a two-stage process from coarse to fine registration, this approach combines the advantages of deep learning and grayscale iteration methods: the rapid prediction of deep learning and the pixel-level optimization of grayscale iteration. This achieves 2D / 3D ankle joint registration tasks with the speed and accuracy required for clinical surgery.

[0117] 4) The proposed image style unification method utilizes CycleGAN combined with multi-task learning, attention mechanisms, data augmentation, and improved loss function design to enhance performance in the style unification task of ankle X-rays and DRR. Compared to traditional histogram equalization, this method offers greater flexibility and enables sophisticated transitions between different styles. It not only enhances contrast but also preserves key image information and anatomical structure, ensuring that the style-unified images provide accurate input data for subsequent 2D / 3D registration. This approach is adaptable to complex application scenarios.

[0118] 5) By unifying the image style of the input X-ray image so that its intensity distribution is the same as that of the DRR image, it is beneficial to predict the label information of the X-ray image and reduce the error caused by contrast differences in the first stage of registration. In the second stage of registration, it is beneficial to the grayscale iteration optimization process and reduce the possibility of falling into the local optimal solution.

[0119] 6) Through the multi-start strategy and the acceleration strategy of pre-storing DRR array key-value pairs, the time is further optimized and the possibility of falling into the local optimal solution is reduced.

[0120] Reference Figure 2 , an ankle joint image registration system based on fusion of three-dimensional CT data, comprising:

[0121] The first module 201 is configured to perform grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information to obtain a grayscale uniformed ankle joint X-ray image;

[0122] The second module 202 is used to perform inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image;

[0123] The third module 203 is used to perform a two-stage image registration process based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters;

[0124] The fourth module 204 is configured to implement ankle joint image registration based on the predicted affine transformation parameters.

[0125] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for ankle joint image registration based on fused three-dimensional CT data, characterized in that: The following steps are involved: Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is grayscale unified to obtain the unified ankle joint X-ray image, which includes: The improved CycleGAN network model is trained based on the ankle joint training dataset to obtain a trained CycleGAN network model; The improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module, wherein the loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function, wherein the output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, the output end of the attention mechanism module is both connected to the input end of the first discriminator and the input end of the second discriminator, and the output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module; Based on the trained image segmentation network model, the ankle joint X-ray image with unified grayscale is inferred and predicted to obtain the segmentation label information of the ankle joint X-ray image; Perform coarse registration of the segmented label information of the ankle X-ray image with the ankle CT image with label information to obtain preliminary predicted affine transformation parameters; Apply the preliminary predicted affine transformation parameters to the ankle joint CT image for DDR projection processing to obtain the predicted ankle joint DRR image; Calculate the similarity between the predicted ankle joint DRR image and the ankle joint X-ray image, and judge the calculation results; If the calculated result does not meet the preset accuracy requirements, the initial predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization registration, including: Extract the rotation angle and displacement value based on the preliminary predicted affine transformation parameters and construct the transformation matrix; Perform inverse transformation on the transformation matrix, transform the virtual point light source, and perform matrix multiplication calculation with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array; Generate a new ankle joint DRR image according to the transformed virtual light source point position and the new DRR physical coordinate array; Convert the ankle joint X-ray image into an image array and calculate the pixel gradient value with the new ankle joint DRR image to obtain the gradient direction measurement value and gradient median value; If the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirement, the grayscale iterative registration is performed cyclically until the gradient direction measurement value and the gradient median value meet the preset similarity value requirement; Until the calculation result meets the preset accuracy requirement, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output; Ankle joint image registration based on predicted affine transformation parameters.

2. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 1, characterized in that: The step of performing grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image to obtain the ankle joint X-ray image after grayscale uniformization specifically includes: Obtain ankle X-rays; Acquire ankle joint CT images and segment them according to the various bone parts to obtain ankle joint CT images with label information; The ankle joint CT image with label information is projected by digital reconstruction radiographic imaging method to construct the ankle joint DRR image with label information; Combine ankle joint DRR images with label information and ankle joint X-ray images to construct an ankle joint training dataset; The improved CycleGAN network model is trained based on the ankle joint training dataset to obtain a trained CycleGAN network model; Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is grayscale unified to obtain the ankle joint X-ray image with unified grayscale.

3. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 2, characterized in that: The step of training the improved CycleGAN network model based on the ankle joint training dataset to obtain the trained CycleGAN network model specifically includes: Input the ankle joint training dataset into the improved CycleGAN network model; Based on the first generator of the improved CycleGAN network model, the ankle joint X-ray image is processed by image style conversion to obtain an ankle joint X-ray image with DRR style; Based on the second generator of the improved CycleGAN network model, the ankle joint DRR images with label information are processed by image style conversion to obtain ankle joint DRR images with X-ray style; Based on the attention mechanism module of the improved CycleGAN network model, the features of the ankle joint X-ray image with DRR style and the ankle joint DRR image with X-ray style are enhanced to obtain the enhanced ankle joint X-ray image with DRR style and the enhanced ankle joint DRR image with X-ray style; Based on the first discriminator of the improved CycleGAN network model, the enhanced ankle joint X-ray image with DRR style is judged to obtain the first judgment result; Based on the second discriminator of the improved CycleGAN network model, the enhanced ankle joint DRR image with X-ray style is judged to obtain the second judgment result; Based on the loss function module of the improved CycleGAN network model, the first judgment result and the second judgment result are restored, and the trained CycleGAN network model is output.

4. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 3, characterized in that: The step of performing inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image specifically includes: The image segmentation network model is pre-trained using ankle joint DRR images with label information to obtain a trained image segmentation network model; The ankle joint X-ray image with unified grayscale is input into the trained image segmentation network model for inference and prediction to obtain the segmentation label information of the ankle joint X-ray image.

5. An ankle joint image registration system based on fusion of three-dimensional CT data, characterized in that: Includes the following modules: The first module is used to perform grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and the ankle joint DRR image with label information to obtain the ankle joint X-ray image after grayscale uniformization, which includes: The improved CycleGAN network model is trained based on the ankle joint training dataset to obtain a trained CycleGAN network model; The improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module, wherein the loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function, wherein the output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, the output end of the attention mechanism module is both connected to the input end of the first discriminator and the input end of the second discriminator, and the output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module; The second module is used to perform inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain the segmentation label information of the ankle joint X-ray image; The third module is used to perform coarse registration between the segmented label information of the ankle X-ray image and the ankle CT image with label information to obtain preliminary predicted affine transformation parameters; Apply the preliminary predicted affine transformation parameters to the ankle joint CT image for DDR projection processing to obtain the predicted ankle joint DRR image; Calculate the similarity between the predicted ankle joint DRR image and the ankle joint X-ray image, and judge the calculation results; If the calculated result does not meet the preset accuracy requirements, the initial predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization registration, including: Extract the rotation angle and displacement value based on the preliminary predicted affine transformation parameters and construct the transformation matrix; Perform inverse transformation on the transformation matrix, transform the virtual point light source, and perform matrix multiplication calculation with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array; Generate a new ankle joint DRR image according to the transformed virtual light source point position and the new DRR physical coordinate array; Convert the ankle joint X-ray image into an image array and calculate the pixel gradient value with the new ankle joint DRR image to obtain the gradient direction measurement value and gradient median value; If the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirement, the grayscale iterative registration is performed cyclically until the gradient direction measurement value and the gradient median value meet the preset similarity value requirement; Until the calculation result meets the preset accuracy requirement, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output; The fourth module is used to realize ankle joint image registration based on predicted affine transformation parameters.

Images