Ankle joint image registration method and system based on fused three-dimensional CT data

Abstract

The invention discloses an ankle joint image registration method and system based on fused three-dimensional CT data, and the method comprises the steps: carrying out the gray-scale unified processing of an ankle joint X-ray image based on a trained CycleGAN network model in combination with an ankle joint DRR image with label information, and obtaining an ankle joint X-ray image after the gray-scale unified; based on the trained image segmentation network model, performing reasoning prediction on the ankle joint X-ray image after gray scale unification to obtain segmentation label information of the ankle joint X-ray image; based on the segmentation label information of the ankle joint X-ray image, performing two-stage image registration processing to obtain a predicted affine transformation parameter; and realizing ankle joint image registration based on prediction affine transformation parameters. According to the invention, the ankle joint image registration can be realized quickly and accurately. The ankle joint image registration method and system based on fused three-dimensional CT data can be widely applied to the technical field of image registration.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular to an ankle joint image registration method and system based on fusion of three-dimensional CT data. Background Art

[0002] Combining computer technology and imaging data, the patient's bone structure is 3D modeled to help doctors plan the surgical process, so that a large amount of X-ray transmission is not required during the operation, thereby greatly reducing the radiation dose. The basic workflow of the orthopedic surgical navigation system is to obtain the 3D data of the patient's corresponding bones through CT scanning before the operation and build a complete 3D model. During the operation, the system collects 2D X-ray data in real time, and aligns it with the 3D CT data obtained before the operation to establish a correspondence between the digital space and the real space. Subsequently, the system will track the position of the navigation point (i.e., the surgical tool) in real time and render it in the digital space to help doctors accurately locate and operate. However, the image feature-based method in the related art uses artificial markers or inherent features inside and outside the patient's body for positioning. For example, skin markers or titanium nails are attached to the human body before surgery. Although this method is fast and accurate, it is too damaging to the human body and does not conform to the principle of minimally invasive surgery. Other external marker methods have this problem and are difficult to meet the needs of clinical surgery. The 2D / 3D registration method based on image grayscale first uses digital reconstruction of radiographic images (DRR) to project 3D images into 2D images to achieve dimensionality unification, and then measures the similarity between DRR and X-rays, and iterates and optimizes to obtain the best registration result. However, the speed of the simple grayscale iteration method is significantly different from the clinical needs of surgery. In addition, medical image registration based on deep learning can be roughly divided into three types: deep iteration, supervised, and unsupervised according to the type and training scheme. The deep iteration method only uses neural networks to extract image features and learn similarity, and still uses traditional methods for iterative calculation, but it also has the problem of slow speed; the supervised learning method uses real transformation parameters as the gold standard to train the network and regress the transformation parameters. This method requires a large amount of labeled data for training. Unsupervised learning optimizes the registration results through its own loss function or adversarial network, which can adapt to problems of different types and modalities. However, the lack of labels makes the results difficult to evaluate and poorly interpretable. In addition, a large amount of data is required to learn effective feature representations, while the amount of medical data is often small and insufficient to support training. In summary, deep learning-based methods in the context of orthopedic surgical navigation tasks generally have problems such as insufficient effectiveness and insufficient generalization due to the lack of a large number of labeled datasets. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide an ankle joint image registration method and system based on fused three-dimensional CT data, which can realize ankle joint image registration quickly and accurately.

[0004] The first technical solution adopted by the present invention is: a method for ankle joint image registration based on fusion of three-dimensional CT data, comprising the following steps:

[0005] Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is processed in grayscale to obtain the ankle joint X-ray image with unified grayscale;

[0006] Based on the trained image segmentation network model, the ankle joint X-ray image with unified grayscale is inferred and predicted to obtain the segmentation label information of the ankle joint X-ray image;

[0007] Based on the segmentation label information of the ankle joint X-ray image, a two-stage image registration process is performed to obtain the predicted affine transformation parameters;

[0008] Ankle joint image registration based on predicted affine transformation parameters.

[0009] Further, the step of performing grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image to obtain the ankle joint X-ray image after grayscale uniformization specifically includes:

[0010] Get ankle X-rays;

[0011] Acquire an ankle joint CT image and perform segmentation processing according to various bone parts to obtain an ankle joint CT image with label information;

[0012] The ankle joint CT image with label information is projected by digital reconstruction radiographic imaging method to construct the ankle joint DRR image with label information;

[0013] Combine ankle joint DRR images with label information and ankle joint X-ray images to construct an ankle joint training dataset;

[0014] The improved CycleGAN network model is trained based on the ankle joint training data set to obtain a trained CycleGAN network model;

[0015] Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is processed with unified grayscale to obtain the ankle joint X-ray image with unified grayscale.

[0016] Furthermore, the improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module, and the loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function, wherein the output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, the output end of the attention mechanism module is both connected to the input end of the first discriminator and the input end of the second discriminator, and the output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module.

[0017] Furthermore, the step of training the improved CycleGAN network model based on the ankle joint training data set to obtain the trained CycleGAN network model specifically includes:

[0018] Input the ankle joint training dataset into the improved CycleGAN network model;

[0019] Based on the first generator of the improved CycleGAN network model, the ankle joint X-ray image is processed by image style conversion to obtain an ankle joint X-ray image with DRR style;

[0020] Based on the second generator of the improved CycleGAN network model, the ankle joint DRR image with label information is processed by image style conversion to obtain an ankle joint DRR image with X-ray style;

[0021] Based on the attention mechanism module of the improved CycleGAN network model, the ankle joint X-ray image with DRR style and the ankle joint DRR image with X-ray style are feature enhanced to obtain the enhanced ankle joint X-ray image with DRR style and the enhanced ankle joint DRR image with X-ray style;

[0022] Based on the first discriminator of the improved CycleGAN network model, the enhanced ankle joint X-ray image with DRR style is judged to obtain a first judgment result;

[0023] Based on the second discriminator of the improved CycleGAN network model, the enhanced ankle joint DRR image with X-ray style is judged to obtain the second judgment result;

[0024] Based on the loss function module of the improved CycleGAN network model, the first judgment result and the second judgment result are restored, and the trained CycleGAN network model is output.

[0025] Furthermore, the step of performing inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image specifically includes:

[0026] The image segmentation network model is pre-trained by using the ankle joint DRR images with label information to obtain the trained image segmentation network model;

[0027] The ankle joint X-ray image with unified grayscale is input into the trained image segmentation network model for inference and prediction to obtain the segmentation label information of the ankle joint X-ray image.

[0028] Furthermore, the step of performing two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters specifically includes:

[0029] Roughly align the segmented label information of the ankle joint X-ray image with the ankle joint CT image with label information to obtain preliminary predicted affine transformation parameters;

[0030] Apply the preliminary predicted affine transformation parameters to the ankle joint CT image for DDR projection processing to obtain the predicted ankle joint DRR image;

[0031] Calculate the similarity between the predicted ankle joint DRR image and the ankle joint X-ray image, and judge the calculation result;

[0032] If the calculation result does not meet the preset accuracy requirement, the preliminary predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization registration;

[0033] Until the calculation result meets the preset accuracy requirement, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output.

[0034] Furthermore, the step of performing grayscale iterative registration on the preliminary predicted affine transformation parameters specifically includes:

[0035] Extract the rotation angle and displacement value according to the preliminary predicted affine transformation parameters and construct the transformation matrix;

[0036] Perform inverse transformation on the transformation matrix, transform the virtual point light source, and perform matrix multiplication calculation with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array;

[0037] Generate a new ankle joint DRR image according to the transformed virtual light source point position and the new DRR physical coordinate array;

[0038] The ankle joint X-ray image is converted into an image array form and pixel gradient values ​​are calculated with the new ankle joint DRR image to obtain the gradient direction measurement value and the gradient median value;

[0039] If the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirement, grayscale iterative registration is performed cyclically until the gradient direction measurement value and the gradient median value meet the preset similarity value requirement.

[0040] The second technical solution adopted by the present invention is: an ankle joint image registration system based on fusion of three-dimensional CT data, comprising:

[0041] The first module is used to perform grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information to obtain the ankle joint X-ray image after grayscale uniformization;

[0042] The second module is used to perform inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain the segmentation label information of the ankle joint X-ray image;

[0043] The third module is used to perform two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain the predicted affine transformation parameters;

[0044] The fourth module is used to realize ankle joint image registration based on predicted affine transformation parameters.

[0045] The beneficial effects of the method and system of the present invention are as follows: the present invention performs grayscale uniform processing on ankle joint X-ray images based on the trained CycleGAN network model and in combination with ankle joint DRR images with label information, adopts DRR combined with image style uniform method to replace X-ray for training the network of related deep learning processes, does not need to collect a large amount of expensive real clinical image data, greatly reduces data cost and annotation cost, further based on the trained image segmentation network model, performs inference prediction on the ankle joint X-ray images after grayscale uniformization, obtains segmentation label information of the ankle joint X-ray images, and finally performs two-stage image registration processing based on the segmentation label information of the ankle joint X-ray images, and performs image style uniformity on the input X-ray images to make their intensity distribution the same as that of the DRR images. In the first stage registration process, it is beneficial to predict the label information of the X-ray images and reduce the errors caused by contrast differences; in the second stage registration, it is beneficial to the optimization process of grayscale iteration, reduces the possibility of falling into the local optimal solution, and finally improves the image registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the steps of an ankle joint image registration method based on fusion of three-dimensional CT data of the present invention;

[0047] Figure 2 It is a structural block diagram of an ankle joint image registration system based on fusion of three-dimensional CT data of the present invention;

[0048] Figure 3 It is a schematic diagram of the process framework of ankle joint image registration provided by a specific embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of CycleGAN network training provided by a specific embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of a result of unified grayscale style provided by a specific embodiment of the present invention;

[0051] Figure 6 is a schematic diagram of pre-training of an image segmentation network model provided by a specific embodiment of the present invention;

[0052] Figure 7 is a schematic diagram of a two-stage image registration provided by a specific embodiment of the present invention;

[0053] Figure 8 It is a schematic diagram of grayscale iterative stage registration provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] First of all, it should be noted that existing methods such as a 4D joint standing position three-dimensional imaging method and device use a preset joint motion two-dimensional image sequence and standing position three-dimensional reconstruction data to train a segmentation model for 3D data and segmentation data for a motion 2D data sequence to perform 2D / 3D registration to predict the transformation parameters corresponding to each frame. However, in the context of expensive medical annotation data, it is not feasible to directly predict the affine transformation parameters of the registration result through deep learning, and the speed of the grayscale iteration method is difficult to meet clinical requirements. Therefore, some studies have tried to combine the advantages of both, using deep learning methods to predict a good initial pose (coarse registration), and then using a grayscale iteration process to optimize at the pixel level to obtain accurate results (fine registration). For example, Robert B. Grupp et al. have solutions for hip joint registration tasks, but without exception, these methods require automatic label information detection of intraoperative X-rays, which is a difficult task and is prone to additional errors. In addition, in the grayscale iteration algorithm, generating DRR is the most time-consuming step, taking up about 60% of the time.

[0056] Based on this, the embodiment of the present invention provides a two-stage 2D / 3D registration strategy combining deep learning and grayscale iteration. An improved CycleGAN image style unification method is designed, such as Figure 3 As shown in the figure, it is used to assist in further obtaining the segmentation labels of 2D images on the premise of obtaining the segmentation labels of CT images. An acceleration strategy for the grayscale iteration process in the fine registration stage is designed, which pre-projects the preoperative CT data within a certain range and stores the DRR array instead of the image. During the operation, the corresponding DRR array is taken out to calculate the similarity measure, thereby omitting the step of generating DRR and speeding up the grayscale iteration process, providing reliable technical support for achieving accurate and fast ankle 2D / 3D registration.

[0057] Reference Figure 1 The present invention provides an ankle joint image registration method based on fusion of three-dimensional CT data, the method comprising the following steps:

[0058] S100, based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information, performing grayscale uniform processing on the ankle joint X-ray image to obtain a grayscale uniform ankle joint X-ray image;

[0059] Specifically, an ankle X-ray image is acquired; an ankle CT image is acquired and segmented according to various bone parts to obtain an ankle CT image with label information; the ankle CT image with label information is projected by a digital reconstruction radiographic method to construct an ankle DRR image with label information; an ankle DRR image with label information is combined with an ankle X-ray image to construct an ankle training data set; an improved CycleGAN network model is trained based on the ankle training data set to obtain a trained CycleGAN network model; based on the trained CycleGAN network model and combined with the ankle DRR image with label information, the ankle X-ray image is grayscale unified to obtain an ankle X-ray image with unified grayscale.

[0060] In this embodiment, first, a CT image of the ankle joint is obtained before surgery. The 3D Slicer software is used to segment the various bone parts to obtain three-dimensional image data of bones such as the tibia, fibula, and talus. The segmentation label information is saved, and a large amount of DRR containing segmentation label information is obtained by applying digital reconstruction radiography (DRR) technology, a training data set is constructed, and X-ray images are obtained during surgery, and the image style is unified with the DRR images based on the improved CycleGAN network.

[0061] It should be further explained that the training data set consists of two parts: ankle joint DRR images obtained by DRR projection of ankle joint CT images of real clinical patients, and intraoperative X-ray images of real clinical patients. Since intraoperative X-ray images are used, the network model should be trained using ankle joint image data of other patients before surgery. The training data set includes image pairing data of X-ray images and DRR images, as follows:

[0062]

[0063] Each of these is the ankle joint X-ray image corresponding to the i-th sample.

[0064]

[0065] Each of these is the CT projection image (DRR) corresponding to the i-th sample, generated by the corresponding CT data.

[0066] Therefore, the expression of the total training data set is as follows:

[0067]

[0068] It should be noted that the improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module, and the loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function, wherein the output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, the output end of the attention mechanism module is both connected to the input end of the first discriminator and the input end of the second discriminator, and the output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module.

[0069] Furthermore, the network pre-training process in the embodiment of the present invention is described, the ankle joint training data set is input into the improved CycleGAN network model; based on the first generator of the improved CycleGAN network model, the ankle joint X-ray image is subjected to image style conversion processing to obtain an ankle joint X-ray image with a DRR style; based on the second generator of the improved CycleGAN network model, the ankle joint DRR image with label information is subjected to image style conversion processing to obtain an ankle joint DRR image with an X-ray style; based on the attention mechanism module of the improved CycleGAN network model, the ankle joint X-ray image with a DRR style is compared with the ankle joint X-ray image with an X-ray style. The joint DRR image is feature enhanced to obtain an enhanced ankle joint X-ray image with DRR style and an enhanced ankle joint DRR image with X-ray style; based on the first discriminator of the improved CycleGAN network model, the enhanced ankle joint X-ray image with DRR style is judged to obtain a first judgment result; based on the second discriminator of the improved CycleGAN network model, the enhanced ankle joint DRR image with X-ray style is judged to obtain a second judgment result; based on the loss function module of the improved CycleGAN network model, the first judgment result and the second judgment result are restored, and the trained CycleGAN network model is output.

[0070] like Figure 4 As shown in the figure, the input stage inputs the ankle joint X-ray image as the source image for image style unification. The DRR image is input as the target image for image style reference. Then it is input into the generator to generate a suitable output image according to the image content and target style.

[0071] The X-ray image is input into the generator G, which converts the X-ray image into a DRR style image, maintains the structural information, and adjusts the style. Generator G outputs a DRR style X-ray image.

[0072] The DRR image is input into the generator F, which converts the DRR style image back to the X-ray image style to assist the training process. Generator F outputs the X-ray style DRR image.

[0073] The image output by the generator will be processed by the attention mechanism. The function of this module is to enhance the key information areas in the image (such as the details of the ankle joint). It focuses on the important parts of the image, retains these key information, and avoids unnecessary loss of details. It includes spatial attention mechanism and channel attention mechanism.

[0074] The role of the spatial attention mechanism is to enhance the focus on key areas in the image (such as the center area of ​​the ankle joint). Its input is the generated image, and its output is an image that focuses on important areas.

[0075] The role of the channel attention mechanism is to learn the importance of different channel features and optimize the quality of image conversion. Its input is the generated image features, and its output is the optimized image features.

[0076] The image processed by the attention mechanism will be evaluated by the discriminator. It determines whether the generated image conforms to the target style and compares it with the real image, so as to promote the continuous optimization of the generator through adversarial training and generate more realistic images. The discriminators D_A and D_B respectively determine whether the styles of the generated X-ray image and DRR image are consistent with the real style. Its input is the generated image and the corresponding real image, and the output is the judgment result.

[0077] The loss function module in the figure is an improved loss function module, which includes the following losses:

[0078] Cycle consistency loss: ensures that the generated style transfer image can be restored to the original image. Its input is the generated image and its reverse conversion result. Specifically, it requires that when an image goes from the source style (such as an X-ray image) to the target style (such as a CT projection image) and then back to the source style, the original image should be restored as much as possible. Assuming that G is a generator from the source style (X-ray image) to the target style (DRR image), F is a generator from the target style (DRR image) to the source style (X-ray image), x is the source image, and y is the target style image, then the cycle consistency loss is defined as:

[0079] L cycle (G,F)=E x～X [||F(G(x))-x||1]+E y～Y [||G(F(y))-y||1]

[0080] Where ||·||1 represents the L1 norm (absolute error), which is often used to measure the pixel difference between images. The first term ensures that the image transformed by the generator G and the reverse generator F is similar to the original image. The second term ensures that the target image transformed by the generator F and the reverse generator G is similar to the original target image.

[0081] Adversarial loss: It comes from the adversarial training framework of Generative Adversarial Network (GAN). The discriminator helps the generator generate more realistic images by distinguishing between generated images and real images. For the generator G and the discriminator D, the adversarial loss includes the loss of the generator and the loss of the discriminator. The goal of the generator is to make the discriminator think that the generated image is real. The adversarial loss of the generator is as follows:

[0082]

[0083] The goal of the discriminator is to distinguish between generated images and real images. The adversarial loss of the discriminator is as follows:

[0084]

[0085] Where D(x) represents the output of the discriminator for image x, indicating the probability that image x is a real image.

[0086] Style loss: The style features of the image are represented based on the Gram matrix to ensure that the generated image has consistent texture and tone features in the target style. The style loss measures the difference between the generated image and the target image at the style level, usually measured using the Euclidean distance (L2 norm).

[0087] Assuming that the features of the generated image G(x) and the target image y are extracted as feature map F through a layer of convolutional neural network, the style loss is defined as:

[0088]

[0089] Where Gram(F) means extracting the style features of the image by calculating the Gram matrix for the feature map F. The Gram matrix is ​​obtained by calculating the inner product between the features of each layer in the feature map F.

[0090] Content loss: Ensure that the generated image is consistent with the original image at the content level. Usually, the VGG network is used to extract the high-level features of the image, and then the difference between the generated image and the target image in the feature space is calculated. Assuming φ is the feature map of a layer in the VGG network, the content loss of the generated image G(x) and the target image y is defined as:

[0091]

[0092] Where φ(G(x)) and φ(y) represent the features extracted from the generated image and the target image at a certain VGG layer. Content loss measures the difference in structure and content of an image by calculating the difference between high-level features of the image, usually measured using the L2 norm (Euclidean distance).

[0093] The weighting strategy of the loss function is: during the training process, by dynamically adjusting the weights of cycle consistency loss, adversarial loss, style loss and content loss, the model can take into account the fidelity of both image content and visual style during style transfer.

[0094] Through the above methods, CycleGAN can not only accurately unify the visual style of the image in the style transfer task, but also effectively maintain the structural information of the image, especially in the protection of details such as ankle bones in medical images, and improve the effect of unifying the style of ankle X-ray images and DRR images. Figure 5shown.

[0095] S200, based on the trained image segmentation network model, inferring and predicting the ankle joint X-ray image after grayscale unification to obtain segmentation label information of the ankle joint X-ray image;

[0096] Specifically, Figure 6 As shown, the image segmentation network model is pre-trained by the ankle joint DRR image with label information to obtain the trained image segmentation network model; the ankle joint X-ray image after grayscale unification is input into the trained image segmentation network model for inference and prediction to obtain the segmentation label information of the ankle joint X-ray image.

[0097] The ankle CT image is acquired during preoperative scanning and is segmented into four parts in the 3D Slicer software: tibia, fibula, talus, and the remaining bones (collectively referred to as the sole of the foot). For the 2D / 3D registration task scenario of ankle replacement surgery, the focus is on the tibia area and the area near the tibia-talus junction. The segmentation label information is saved and can be used in subsequent steps to perform rough registration in combination with the label information of the X-ray. After that, the CT image is projected to obtain several DRRs containing segmentation labels to construct a training data set. Train the image segmentation network model. When the X-ray image is obtained during surgery, its label information can be inferred. Since there are differences in grayscale and image style between the intraoperative X-ray image and the DRR image used during training, in order to improve the accuracy of reasoning, the image style of the X-ray image and the DRR image is unified as described in step 200. After that, the segmentation label information of each bone corresponding to the X-ray image can be accurately predicted.

[0098] In this embodiment, the DRR technology used is implemented based on the Siddon algorithm, and the GPU is used for accelerated calculation. The principle of the Siddon algorithm to generate DRR images is to simulate the projection process of X-rays through ray tracing. Specifically, the algorithm calculates the path of each ray emitted from the projection source when it passes through a three-dimensional volume (such as CT scan data), and accurately calculates the intersection of the ray with each voxel and the length of the ray passing through. Then, based on the density value of each voxel, the attenuation effect of the light in the volume is accumulated, and finally these attenuation values ​​are projected onto a two-dimensional plane to generate a DRR image similar to the actual X-ray projection. This method uses the efficiency of the Siddon algorithm to quickly determine which voxels are passed by the ray and the contribution of the ray in the voxel, thereby accelerating the generation of DRR.

[0099] S300, performing a two-stage image registration process based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters;

[0100] Specifically, the segmented label information of the ankle X-ray image is roughly aligned with the ankle CT image with label information to obtain preliminary predicted affine transformation parameters; the preliminary predicted affine transformation parameters are applied to the ankle CT image for DDR projection processing to obtain a predicted ankle DRR image; the predicted ankle DRR image and the ankle X-ray image are processed for similarity values, and the calculation result is judged; if the calculation result does not meet the preset accuracy requirement, the preliminary predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization alignment; until the calculation result meets the preset accuracy requirement, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output.

[0101] Further, if Figure 7 As shown in the figure, the initial pose parameters of the CT image are set to the result of the first stage of registration, that is, a pose parameter that is very close to the correct result. The position of the initial virtual point light source is set according to this pose parameter, and the DRR is projected, and the similarity measure is calculated and judged with the intraoperative X-ray. When it is similar enough, the registration is stopped and the final pose parameters, that is, the predicted affine transformation parameters, are output.

[0102] Further, the grayscale iterative registration in this embodiment is explained. The rotation angle and displacement value are extracted according to the preliminary predicted affine transformation parameters to construct a transformation matrix; the transformation matrix is ​​inversely transformed, and the virtual point light source is transformed and matrix multiplication is performed with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array; a new ankle joint DRR image is generated according to the transformed virtual light source point position and the new DRR physical coordinate array; the ankle joint X-ray image is converted into an image array form and the pixel gradient value is calculated with the new ankle joint DRR image to obtain the gradient direction measurement value and the gradient median value; if the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirements, the grayscale iterative registration is repeated until the gradient direction measurement value and the gradient median value meet the preset similarity value requirements.

[0103] The optimization algorithm uses the CMA-ES evolutionary algorithm. The parameter search space is divided using the KD tree partitioning algorithm to achieve multi-start parallel search, which can avoid falling into the local optimal solution. The similarity measure uses the gradient direction measure (GO). Since the first stage registration has found the pose parameters with approximate accurate values, the parameter search space can be set very small to speed up the registration speed.

[0104] Each iteration generally performs two tasks: after searching for new affine transformation parameters, a new DRR is generated. This is called the Update module. The new DRR and the intraoperative X-ray are used to calculate and judge the GO measure. This is called the Metric module.

[0105] The Update module takes more than 60% of the time of the entire second stage registration. The main steps are as follows: The algorithm extracts the latest affine transformation parameters [rot X ,rot Y ,rot Z ,trans X ,trans Y ,trans Z ], that is, the three rotation angles around the X, Y, and Z axes and the three translation values ​​along the X, Y, and Z axes. Calculate the transformation matrix Tr based on the input rotation angle and displacement value, and calculate its inverse matrix inv T , and convert it to float32 type and assign it to GPU variable; use inv T Transform the virtual point light source self.source to obtain the transformed virtual light source point position source_forGpu; GPU combines the old DRR array with inv T Perform matrix multiplication and other processing with Tn to finally obtain a new DRR physical coordinate array DRRPhy_array, which is flattened into one dimension; generate a new DRR image based on the new virtual light source point position source_forGpu and the new DRR array DRRPhy_array, adjust it to a two-dimensional shape and return it.

[0106] In the Metric stage, the X-rays are input and converted into image arrays, and the gradient map (gradient value of each pixel) of the DRR and X-ray images is calculated using GPU acceleration, and the median value is calculated as the calculation threshold to obtain the GO metric value. This stage takes less than 40% of the time.

[0107] This scheme designs an acceleration strategy for the grayscale iteration process. In the preoperative preparation stage, the affine transformation parameters are changed in advance according to a certain fine-grained step size, and all the generated DRR physical coordinate arrays and the normalized values ​​of their corresponding affine transformation parameters are saved as key-value pairs. The key is the affine parameter and the value is the DRR physical coordinate array.

[0108] like Figure 8 As shown in the figure, the left side of the dotted line is the work of the preoperative preparation stage, and the right side of the dotted line is the grayscale iteration stage during the operation.

[0109] After obtaining the CT image before surgery, according to the X-ray viewing angle planned during surgery (generally 0° anteroposterior, i.e., AP viewing angle; or 90° lateral, i.e., LAT viewing angle), the camera position of the shooting angle is used as the reference point to generate a DRR that is sufficient to cover the affine transformation parameter space. The two modules of moving the point light source and generating DRR on the left side of this figure are not necessary. Only the DRR array needs to be generated and stored, rather than the DRR image, to save space.

[0110] The range of the affine transformation depends on the size and resolution of the CT image. An example is that the rotation range of the three rotation axes is set to [-5°, 5°[, with a step size of 2°, that is, 2 degrees each rotation. The translation range of the three translation axes is [-10mm, 10mm], with a step size of 2mm. This example will generate 125,000 different affine transformation parameters and DRR arrays, that is, 125,000 key-value pairs are stored. The smaller the step size, the more key-value pairs are stored. The selection of the step size needs to take into account the allowable error range of the general registration method. According to current relevant research, it is generally believed that for ankle joint tasks, when the rotation difference of the manual registration is less than 2° and the translation difference is less than 3mm, the registration is considered successful.

[0111] After all DRR arrays are generated, when the intraoperative registration reaches the grayscale iteration stage, the most approximate Key value is searched from the key-value pair array according to the affine transformation parameters currently selected by the CMA-ES evolutionary algorithm, and the DRR array stored in its Value value is extracted. The gradient map and gradient median are calculated together with the X-ray, and the similarity measure GO is calculated. If the threshold is reached, the registration is stopped, otherwise the evolutionary algorithm continues to select new affine transformation parameters and continue to select new DRR arrays to repeat the process.

[0112] S400, realizing ankle joint image registration based on predicted affine transformation parameters.

[0113] In summary, the embodiments of the present invention have the following beneficial effects:

[0114] 1) Using DRR combined with the image style unification method instead of X-rays to train the network of related deep learning processes does not require the collection of a large amount of expensive real clinical imaging data, greatly reducing data costs and annotation costs.

[0115] 2) Using the form of CT image projection DRR, the CT image can be directly projected into the two-dimensional DRR, without the need for manual verification, thus reducing the verification cost.

[0116] 3) Through the two-stage process of coarse registration to fine registration, the advantages of deep learning and grayscale iteration are combined, namely the fast prediction advantage of deep learning and the pixel-level optimization advantage of grayscale iteration. The ankle joint 2D / 3D registration task is achieved in terms of speed and accuracy to meet the needs of clinical surgery.

[0117] 4) The proposed image style unification processing method uses CycleGAN combined with multi-task learning, attention mechanism, data enhancement, improved loss function design and other means to improve the effect in the ankle joint X-ray and DRR style unification task. Compared with traditional histogram equalization, it has higher flexibility and can perform fine conversion between different styles. It not only enhances contrast, but also maintains the key information and anatomical structure of the image, ensuring that the image after style unification can provide accurate input data for subsequent 2D / 3D registration. Adapt to complex application scenarios.

[0118] 5) By unifying the image style of the input X-ray image and making its intensity distribution the same as that of the DRR image, in the first stage of the registration process, it is beneficial to predict the label information of the X-ray image and reduce the error caused by contrast differences; in the second stage of the registration, it is beneficial to the optimization process of grayscale iteration and reduce the possibility of falling into the local optimal solution.

[0119] 6) Through the multi-start strategy and the acceleration strategy of pre-storing DRR array key-value pairs, the time is further optimized and the possibility of falling into the local optimal solution is reduced.

[0120] Reference Figure 2 , an ankle joint image registration system based on fused three-dimensional CT data, comprising:

[0121] The first module 201 is used to perform grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information to obtain a grayscale uniform ankle joint X-ray image;

[0122] The second module 202 is used to perform inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image;

[0123] The third module 203 is used to perform two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters;

[0124] The fourth module 204 is used to implement ankle joint image registration based on the predicted affine transformation parameters.

[0125] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for ankle joint image registration based on fused three-dimensional CT data, characterized in that: The following steps are involved: Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is processed in grayscale to obtain the ankle joint X-ray image with unified grayscale; Based on the trained image segmentation network model, the ankle joint X-ray image with unified grayscale is inferred and predicted to obtain the segmentation label information of the ankle joint X-ray image; Based on the segmentation label information of the ankle joint X-ray image, a two-stage image registration process is performed to obtain the predicted affine transformation parameters; Ankle joint image registration based on predicted affine transformation parameters.

2. According to claim 1, a method for ankle joint image registration based on fusion of three-dimensional CT data, characterized in that: The step of performing grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image to obtain the ankle joint X-ray image after grayscale uniformization specifically includes: Get ankle X-rays; Acquire an ankle joint CT image and perform segmentation processing according to various bone parts to obtain an ankle joint CT image with label information; The ankle joint CT image with label information is projected by digital reconstruction radiographic imaging method to construct the ankle joint DRR image with label information; Combine ankle joint DRR images with label information and ankle joint X-ray images to construct an ankle joint training dataset; The improved CycleGAN network model is trained based on the ankle joint training data set to obtain a trained CycleGAN network model; Based on the trained CycleGAN network model and combined with the ankle joint DRR image with label information, the ankle joint X-ray image is processed with unified grayscale to obtain the ankle joint X-ray image with unified grayscale.

3. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 2, characterized in that: The improved CycleGAN network model specifically includes a first generator, a second generator, an attention mechanism module, a first discriminator, a second discriminator and a loss function module, wherein the loss function module includes a cycle consistency loss function, an adversarial loss function, a style loss function, a content loss function and a weighted loss function, wherein the output end of the first generator and the output end of the second generator are both connected to the input end of the attention mechanism module, the output end of the attention mechanism module is both connected to the input end of the first discriminator and the input end of the second discriminator, and the output end of the first discriminator and the output end of the second discriminator are both connected to the input end of the loss function module.

4. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 3, characterized in that: The step of training the improved CycleGAN network model based on the ankle joint training data set to obtain the trained CycleGAN network model specifically includes: Input the ankle joint training dataset into the improved CycleGAN network model; Based on the first generator of the improved CycleGAN network model, the ankle joint X-ray image is processed by image style conversion to obtain an ankle joint X-ray image with DRR style; Based on the second generator of the improved CycleGAN network model, the ankle joint DRR image with label information is processed by image style conversion to obtain an ankle joint DRR image with X-ray style; Based on the attention mechanism module of the improved CycleGAN network model, the ankle joint X-ray image with DRR style and the ankle joint DRR image with X-ray style are feature enhanced to obtain the enhanced ankle joint X-ray image with DRR style and the enhanced ankle joint DRR image with X-ray style; Based on the first discriminator of the improved CycleGAN network model, the enhanced ankle joint X-ray image with DRR style is judged to obtain a first judgment result; Based on the second discriminator of the improved CycleGAN network model, the enhanced ankle joint DRR image with X-ray style is judged to obtain the second judgment result; Based on the loss function module of the improved CycleGAN network model, the first judgment result and the second judgment result are restored, and the trained CycleGAN network model is output.

5. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 4, characterized in that: The step of performing inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain segmentation label information of the ankle joint X-ray image specifically includes: The image segmentation network model is pre-trained by using the ankle joint DRR images with label information to obtain the trained image segmentation network model; The ankle joint X-ray image with unified grayscale is input into the trained image segmentation network model for inference and prediction to obtain the segmentation label information of the ankle joint X-ray image.

6. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 5, characterized in that: The step of performing two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain predicted affine transformation parameters specifically includes: Roughly align the segmentation label information of the ankle joint X-ray image with the ankle joint CT image with label information to obtain preliminary predicted affine transformation parameters; Apply the preliminary predicted affine transformation parameters to the ankle joint CT image for DDR projection processing to obtain the predicted ankle joint DRR image; Calculate the similarity between the predicted ankle joint DRR image and the ankle joint X-ray image, and judge the calculation result; If the calculation result does not meet the preset accuracy requirement, the preliminary predicted affine transformation parameters are used as the initial pose for grayscale iterative optimization registration; Until the calculation result meets the preset accuracy requirement, the preliminary predicted affine transformation parameters are pruned and the predicted affine transformation parameters are output.

7. The ankle joint image registration method based on fusion of three-dimensional CT data according to claim 6, characterized in that: The step of performing grayscale iterative registration on the preliminary predicted affine transformation parameters specifically includes: Extract the rotation angle and displacement value according to the preliminary predicted affine transformation parameters and construct the transformation matrix; Perform inverse transformation on the transformation matrix, transform the virtual point light source, and perform matrix multiplication calculation with the predicted ankle joint DRR image to obtain the transformed virtual light source point position and the new DRR physical coordinate array; Generate a new ankle joint DRR image according to the transformed virtual light source point position and the new DRR physical coordinate array; The ankle joint X-ray image is converted into an image array form and pixel gradient values ​​are calculated with the new ankle joint DRR image to obtain the gradient direction measurement value and the gradient median value; If the gradient direction measurement value and the gradient median value do not meet the preset similarity value requirement, grayscale iterative registration is performed cyclically until the gradient direction measurement value and the gradient median value meet the preset similarity value requirement.

8. An ankle joint image registration system based on fusion of three-dimensional CT data, characterized in that: Includes the following modules: The first module is used to perform grayscale uniform processing on the ankle joint X-ray image based on the trained CycleGAN network model and in combination with the ankle joint DRR image with label information to obtain the ankle joint X-ray image after grayscale uniformization; The second module is used to perform inference prediction on the ankle joint X-ray image after grayscale unification based on the trained image segmentation network model to obtain the segmentation label information of the ankle joint X-ray image; The third module is used to perform two-stage image registration processing based on the segmentation label information of the ankle joint X-ray image to obtain the predicted affine transformation parameters; The fourth module is used to realize ankle joint image registration based on predicted affine transformation parameters.

Images