A 2D-3D medical image registration method, device, computer equipment and storage medium

Abstract

The present application relates to a 2D-3D medical image registration method, device, computer equipment and storage medium. The method includes: obtaining a preoperative CT image and an intraoperative X-ray image of a target bone block, inputting the preoperative CT image and the intraoperative X-ray image into a regression network based on deep learning, and roughly estimating the initial spatial pose of the target bone block through the regression network; adjusting the projection of the preoperative CT image based on the initial spatial pose to generate a DRR image; inputting the DRR image, the intraoperative X-ray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating the feature point correspondence between the DRR image and the Xray image through the corresponding point relationship estimation network, optimizing the initial spatial pose of the target bone block according to the feature point correspondence, and obtaining the optimized spatial pose. The present application does not require manual annotation of a large amount of data, greatly reduces the radiation damage caused by repeated perspectives, and can achieve relatively efficient and accurate registration.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular, relates to a 2D-3D medical image registration method, device, computer equipment and storage medium. Background Art

[0002] In pelvic fracture reduction surgery, X-ray fluoroscopic images are key data to assist doctors in judging the treatment situation during the operation. During the operation, doctors need to predict the spatial posture of the broken bone through multiple X-rays in order to formulate the next reduction operation. The lack of visual field information of the fracture area poses a great challenge to the doctor's spatial reasoning ability. At the same time, multiple X-rays during the operation will also cause serious radiation damage to the patient. The spatial posture of the real X-ray image can be obtained through 2D-3D image registration. In the prior art, the commonly used 2D-3D image registration methods mainly include:

[0003] 1) Iterative optimization based on image similarity: This method uses the gradient, intensity and features of the preoperative CT (Computed Tomography) image simulated C-arm perspective projection image and the intraoperative X-ray image as the optimization function, and obtains the optimal parameters through multiple iterations to achieve registration of the two images, and obtains the spatial position of the current real X-ray image through the optimal parameters. However, when the initial registration error is large, the optimization problem is usually a non-convex problem, which is easy to fall into the local optimal solution and is not suitable for pelvic fracture reduction surgery.

[0004] 2) 2D-3D registration based on reinforcement learning; this method regards the process of finding the best external parameters of the simulated C-arm to align the simulated projection image (DRR) with the real X-ray image as a Markov process, defines the six-degree-of-freedom state, action function and corresponding reward function, and optimizes it in the environment through artificial agents. However, this algorithm requires a large amount of parameter space and occupies a large amount of computing resources. At the same time, the high-dimensional state space and large action range bring challenges to the training of reinforcement learning, and it takes a long time to train to stabilize.

[0005] 3) 2D-3D registration algorithm based on anatomical landmark detection; this algorithm detects the 2D coordinates and 3D spatial coordinates of multiple groups of anatomical annotation points on X-ray, and uses the pnp (Perspective-n-Point, a method for solving 3D to 2D point pair motion) algorithm to solve the current spatial position of the pelvis. This algorithm requires a large number of manual labels, which is complex and time-consuming to produce.

[0006] 4) 2D-3D registration based on convolutional neural network: This method directly predicts the pelvic spatial position corresponding to a single X-ray image through convolutional neural network. However, due to the specificity of each pelvic fracture patient, the registration accuracy of directly predicting the bone position through convolutional neural network is low, and it is only suitable as an initial registration scheme. Summary of the invention

[0007] The present application provides a 2D-3D medical image registration method, apparatus, computer device and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0008] In order to solve the above problems, this application provides the following technical solutions:

[0009] A 2D-3D medical image registration method, comprising:

[0010] Obtaining a preoperative CT image and an intraoperative X-ray image of the target bone block, inputting the preoperative CT image and the intraoperative X-ray image into a deep learning-based regression network, and roughly estimating the initial spatial position of the target bone block through the regression network;

[0011] Adjusting and projecting the preoperative CT image based on the initial spatial posture to generate a DRR image;

[0012] The DRR image, intraoperative X-ray image and preoperative CT image are input into a pre-trained corresponding point relationship estimation network, the corresponding point relationship between the DRR image and the X-ray image is estimated by the corresponding point relationship estimation network, and the initial spatial pose of the target bone block is optimized and updated according to the corresponding feature point relationship to obtain the optimized spatial pose.

[0013] The technical solution adopted in the embodiment of the present application also includes: the regression network includes a CNN module, a transformer module and a feature fusion module; the CNN module includes a feature extractor, two multi-layer perceptrons and an SVD decomposition module, the feature extractor uses the first 6 layers of the efficientnet-B0 network as the baseline method, the two multi-layer perceptrons are respectively a rotation regression head and a translation regression head, and the rotation regression head and the translation regression head are both three-layer multi-layer perceptrons.

[0014] The technical solution adopted by the embodiment of the present application also includes: the initial spatial position of the target bone block is roughly estimated by the regression network, specifically:

[0015] Inputting the intraoperative X-ray image into a CNN module for first feature extraction;

[0016] Inputting the first feature into two multilayer perceptrons, the two multilayer perceptrons respectively outputting a rotation component and a translation component in the spatial posture of the target bone block through a rotation regression head and a translation regression head;

[0017] The output of the rotation regression head is transformed into a matrix M of a set size, and the matrix M is input into the SVD decomposition module for SVD decomposition to obtain UΣV T , map the matrices U and V to the SO(3) space to obtain the rotation component R of the target bone spatial posture:

[0018] R=UΣ′U T ,whereΣ′=diag(1,...,1,det(UV T ))

[0019] Input the intraoperative X-ray image into a Tranformer module, perform 3D position coding and image block coding on the input image through the Tranformer module, and add the 3D position coding and image block coding results to obtain a second feature of the intraoperative X-ray image;

[0020] The first feature and the second feature are input into a feature fusion module for modulation, and the initial spatial position of the target bone block is output.

[0021] The technical solution adopted in the embodiment of the present application also includes: the 3D position encoding method of the Tranformer module is: using sinusoidal position encoding to encode the vertical position, horizontal position and edge information of the intraoperative X-ray image respectively, and by adding edge information, the two-dimensional spatial coordinates (x, y) of each pixel are expanded to three-dimensional coordinates (x, y, e); the encoding formula is as follows:

[0022]

[0023] Among them, PE is the 3D encoder information, pos D is the position of the image block in dimension D, and the variable i represents the i-th position encoding information.

[0024] The technical solution adopted by the embodiment of the present application also includes: the first feature and the second feature are input into the feature fusion module for modulation, specifically:

[0025] The modulation formula is as follows:

[0026]

[0027] in represents the i+1th input feature of the CNN module, Represents the features output by the i-th swim-block, represents the i-th output feature of the CNN module, G(·) represents the convolution operation, BN(·) represents batch normalization, represents the multiplication operation, Represents the addition operation.

[0028] The technical solution adopted in the embodiment of the present application also includes: the loss function L of the regression network based on deep learning includes the rotation component loss L in SO(3) space o And the translation component loss L in Euclidean space t , the rotational component loss L o Represents the predicted value R and true value Rg of the rotation component t The geodesic distance between the two, the translation component loss L t Represents the predicted translation component t and the true value tg t The translation error between them; the loss function L is defined as follows:

[0029]

[0030] The technical solution adopted by the embodiment of the present application also includes: estimating the feature point correspondence between the DRR image and the Xray image through the corresponding point relationship estimation network, optimizing and updating the initial spatial pose of the target bone block according to the feature point correspondence relationship, and obtaining the optimized spatial pose, specifically:

[0031] The corresponding point relationship estimation network uses the global matching-based optical flow algorithm GMflow to predict the offset dp at the feature point p in the DRR image, and finds the feature point p'=p+dp matching the feature point p in the intraoperative X-ray image through the offset dp, so as to obtain the feature corresponding point relationship between the DRR image and the intraoperative X-ray image;

[0032] Based on the corresponding relationship of the feature points, the relative posture relationship matrix dT between the initial spatial posture and the target posture is calculated by the PPC slover algorithm, and the initial spatial posture of the target bone block is optimized and updated according to the relative posture relationship matrix dT to obtain the optimized spatial posture of the bone block; wherein, the PPC slover algorithm uses {p, p'} between the DRR image and the intraoperative X-ray image and the three-dimensional point w and its gradient g to establish the relationship between the two-dimensional motion and the three-dimensional differential motion, and the relative relationship between the initial spatial posture and the target spatial posture is obtained by the following formula δv=(dw T dt T ) T :

[0033] (g×w T -nT )δv=g T w

[0034]

[0035] Among them, dw T is the rotation vector, dv T The rotation vector dw is converted into an axis-angle representation, where the axis r = dw / ||dw||, the angle θ = ||dw||, and the three-dimensional point w is extracted from the preoperative CT image by a 3D sobel edge extractor.

[0036] Another technical solution adopted by the embodiment of the present application is: a 2D-3D medical image registration device, comprising:

[0037] Rough estimation module: used to obtain preoperative CT images and intraoperative X-ray images of the target bone block, input the preoperative CT images and intraoperative X-ray images into a regression network based on deep learning, and roughly estimate the initial spatial position of the target bone block through the regression network;

[0038] DRR projection module: used to adjust the projection of the preoperative CT image based on the initial spatial posture to generate a DRR image;

[0039] Refined estimation module: used to input the DRR image, intraoperative X-ray image and preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate the feature point correspondence between the DRR image and the X-ray image through the corresponding point relationship estimation network, optimize and update the initial spatial pose of the target bone block according to the feature point correspondence, and obtain the optimized spatial pose.

[0040] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0041] The memory stores program instructions for implementing the 2D-3D medical image registration method;

[0042] The processor is used to execute the program instructions stored in the memory to control a 2D-3D medical image registration method.

[0043] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the 2D-3D medical image registration method.

[0044] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the 2D-3D medical image registration method, device, computer equipment and storage medium of the embodiments of the present application provide a two-stage registration framework combining coarse registration and fine registration to predict the spatial pose of bones of patients with pelvic fractures. In the coarse registration stage, a regression network based on CNN-Transformer is used to roughly estimate the initial spatial pose of the target bone, and the network convergence speed and accuracy are improved by estimating the X-ray light source rotation representation method; in the fine registration stage, a point-to-plane 2D-3D registration algorithm based on deep learning is used to further optimize the initial spatial pose predicted by the regression network, and the initial spatial pose of the target bone is updated through a global optical flow network and PPC, and fine adjustment is completed with a smaller number of iterations. The embodiment of the present application can train the model only based on the patient's preoperative CT image, and a single X-ray image can be used during the operation to predict the spatial position of the bone block. There is no need to manually label a large amount of data, and the running time is fast, which greatly reduces the radiation damage caused by repeated fluoroscopy to doctors and patients. It gets rid of the difficulty of image similarity-based optimization alignment that easily falls into local optimality under large displacement, and can achieve relatively efficient and accurate alignment under a large range of initial alignment errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a 2D-3D medical image registration method according to an embodiment of the present application;

[0046] Figure 2 This is a diagram of a regression network architecture based on deep learning according to an embodiment of the present application;

[0047] Figure 3 This is a schematic diagram of the structure of a 2D-3D medical image registration device according to an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the computer device structure of an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or computer device that includes a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or computer devices.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] Specifically, see Figure 1 , is a flow chart of the 2D-3D medical image registration method of the embodiment of the present application. The 2D-3D medical image registration method of the embodiment of the present application comprises the following steps:

[0054] S100: preoperative CT image and intraoperative X-ray image of the target bone block;

[0055] S110: inputting the preoperative CT image and the intraoperative X-ray image into a deep learning-based regression network, and roughly estimating the initial spatial position of the target bone block through the regression network;

[0056] In this step, if Figure 2As shown, it is a regression network architecture diagram based on deep learning of an embodiment of the present application, and the regression network includes a CNN (Convolutional Neural Networks) module, a transformer module, and a feature fusion module. Among them, the CNN module includes a feature extractor, two multi-layer perceptrons, and an SVD decomposition module. The feature extractor uses the first 6 layers of the efficientnet-B0 network as the baseline method. The two multi-layer perceptrons are a rotation regression head and a translation regression head, respectively. The rotation regression head and the translation regression head are both three-layer multi-layer perceptrons.

[0057] Based on the above network structure, the process of roughly estimating the initial spatial pose of the bone block in the intraoperative X-ray image by the regression network includes:

[0058] S111: Input the intraoperative X-ray image into the CNN module for first feature extraction;

[0059] S112: inputting the extracted first feature into two multilayer perceptrons, and the two multilayer perceptrons respectively output the rotation component and the translation component of the spatial posture of the target bone block through the rotation regression head and the translation regression head; wherein the output dimension of the rotation regression head is 9, and the output dimension of the translation regression head is 3, and the specific dimension value can be set according to the actual application scenario;

[0060] S113: transform the output of the rotation regression head into a matrix M of a set size (3×3, which can be set according to the actual application scenario), and input the matrix M into the SVD decomposition module for SVD decomposition to obtain UΣV T , map the matrices U and V to the SO(3) space to obtain the rotation component R of the target bone spatial posture:

[0061] R=UΣ′V T ,where Σ′=diag(1,…,1,det(UV T )) (1)

[0062] S114: Input the intraoperative X-ray image into the Tranformer module, perform 3D position encoding and image block encoding on the input image through the Tranformer module, and add the 3D position encoding and image block encoding results to obtain the second feature of the intraoperative X-ray image, and pass the second feature map to the CNN module; wherein the 3D position encoding method of the Tranformer module is: use sinusoidal position encoding to encode the vertical position, horizontal position and edge information of the intraoperative X-ray image respectively, and by adding edge information, expand the two-dimensional spatial coordinates (x, y) of each pixel into three-dimensional coordinates (x, y, e). The encoding formula is as follows:

[0063]

[0064] In formula (2), PE is the 3D encoder information, pos D is the position of the image block in dimension D (i.e., vertical position, horizontal position, and edge information), and variable i represents the i-th position encoding information. In the embodiment of the present application, d of each dimension is set to 32, thereby forming 96 positions consisting of spatial position and edge information. Specifically, dimension d can also be set to other values ​​according to actual application scenarios.

[0065] S115: The first feature extracted by the CNN module and the second feature extracted by the transformer module are input into the feature fusion module for modulation, and the initial spatial pose of the target bone is output; wherein the output of each transformer module is modulated with the output of the i∈{1,3,4} layer of the CNN module, and the modulation formula is as follows:

[0066]

[0067] in represents the i+1th input feature of the CNN module, Represents the features output by the i-th swim-block, represents the i-th output feature of the CNN module, G(·) represents the convolution operation, BN(·) represents batch normalization, represents the multiplication operation, Represents the addition operation.

[0068] In the embodiment of the present application, the loss function L of the regression network based on deep learning consists of two parts, namely, the rotation component loss L in SO(3) space and o And the translation component loss L in Euclidean space t , rotational component loss L o Represents the predicted value R and true value Rg of the rotation component t The geodesic distance between them, the translation component loss L t Represents the predicted translation component t and the true value tg t The translation error between ; Specifically, the loss function L is defined as follows:

[0069]

[0070] S120: adjusting and projecting the preoperative CT image based on the initial spatial posture roughly estimated by the regression network to generate a DRR image;

[0071] S130: inputting the DRR image, the X-ray image and the CT image into a pre-trained corresponding point relationship estimation network, and estimating the corresponding relationship of feature points between the DRR image and the X-ray image through the corresponding point relationship estimation network;

[0072] In this step, the corresponding point relationship estimation network is essentially a feature matching network, which uses the global matching-based optical flow algorithm GMflow to estimate the feature corresponding point relationship between the DRR image and the intraoperative X-ray image. The offset dp at the feature point p in the DRR image is obtained by optical flow prediction, and the feature point p'=p+dp matching the feature point p is found in the intraoperative X-ray image through dp to obtain the feature corresponding point relationship between the DRR image and the intraoperative X-ray image. The loss function L of the corresponding point relationship estimation network is flow for:

[0073]

[0074] where f i is the optical flow prediction value of the ith refinement, f gt is the distance between the corresponding points in the two-dimensional projection of the DRR image and the intraoperative X-ray image, γ is the scale factor, and N represents the number of optical flow updates.

[0075] S140: Based on the corresponding relationship of the feature points, the relative posture relationship matrix dT between the initial spatial posture and the target posture is calculated by the PPC slover algorithm, and the initial spatial posture of the target bone block is optimized and updated according to the relative posture relationship matrix dT to obtain the optimized spatial posture of the bone block;

[0076] In this step, a point-to-plane (PPC) 2D-3D registration algorithm based on deep learning is proposed to further optimize the initial spatial pose predicted by the regression network. Based on the correspondence between the feature points of the DRR image and the intraoperative X-ray image, the relative pose relationship matrix dT between the current initial spatial pose T and the target spatial pose is calculated by the PPC slover algorithm. Finally, the initial spatial pose [R′|t′]=dT·[R0|t0] is updated according to the relative relationship, and the above process is repeated multiple times to continuously update the spatial pose of the target bone block, so as to achieve a better registration effect between the DRR image and the intraoperative X-ray image.

[0077] Specifically, the PPCslover algorithm uses {p, p'} between the DRR image and the intraoperative X-ray image as well as the three-dimensional point w and its gradient g to establish the relationship between the two-dimensional motion and the three-dimensional differential motion. The relative relationship between the initial spatial pose and the target spatial pose is obtained by formula (6): δv = (dw T dt T ) T :

[0078] (g×w T -n T )δv=g T w (6)

[0079]

[0080] Among them, dw T is the rotation vector, dv T The rotation vector dw is converted into an axis-angle representation, where the axis r = dw / ||dw||, the angle θ = ||dw||, and the three-dimensional point w is extracted from the preoperative CT image by a 3D sobel edge extractor.

[0081] Based on the above, the 2D-3D medical image registration method of the embodiment of the present application provides a two-stage registration framework combining coarse registration and fine registration to predict the spatial position of the bone blocks of patients with pelvic fractures. In the coarse registration stage, the initial spatial position of the target bone block is roughly estimated by using a regression network based on CNN-Transformer, and the convergence speed and accuracy of the network are improved by estimating the X-ray light source rotation amount representation method; in the fine registration stage, the initial spatial position predicted by the regression network is further optimized using a point-to-plane 2D-3D registration algorithm based on deep learning, and the initial spatial position of the target bone block is updated by a global optical flow network and PPC, and fine adjustment is completed using a small number of iterations. The embodiment of the present application can train the model only based on the patient's preoperative CT image, and a single Xray image can be used during the operation to predict the spatial position of the bone block, without manually annotating a large amount of data, and the running time is fast, which greatly reduces the radiation damage caused to doctors and patients by repeated perspectives, and gets rid of the difficulty of being easily trapped in the local optimum under large displacement based on image similarity optimization registration, and can achieve more efficient and accurate registration under a large range of initial registration errors.

[0082] See also Figure 3 , is a schematic diagram of the structure of a 2D-3D medical image registration device according to an embodiment of the present application. The 2D-3D medical image registration device 40 according to an embodiment of the present application comprises:

[0083] A rough estimation module 41 is used to obtain a preoperative CT image and an intraoperative X-ray image of a target bone block, input the preoperative CT image and the intraoperative X-ray image into a deep learning-based regression network, and roughly estimate the initial spatial position of the target bone block through the regression network;

[0084] DRR projection module 42: used to adjust and project the preoperative CT image based on the initial spatial posture to generate a DRR image;

[0085] Refined estimation module 43: used for inputting the DRR image, intraoperative X-ray image and preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating the feature point correspondence between the DRR image and the X-ray image through the corresponding point relationship estimation network, optimizing and updating the initial spatial pose of the target bone block according to the feature point correspondence, and obtaining the optimized spatial pose.

[0086] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0087] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0088] See also Figure 4 , is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes:

[0089] A memory 51 storing executable program instructions;

[0090] A processor 52 connected to the memory 51;

[0091] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: obtain a preoperative CT image and an intraoperative X-ray image of the target bone block, input the preoperative CT image and the intraoperative X-ray image into a regression network based on deep learning, and roughly estimate the initial spatial pose of the target bone block through the regression network; adjust the projection of the preoperative CT image based on the initial spatial pose to generate a DRR image; input the DRR image, the intraoperative X-ray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate the feature point correspondence relationship between the DRR image and the Xray image through the corresponding point relationship estimation network, optimize the initial spatial pose of the target bone block according to the feature point correspondence relationship, and obtain the optimized spatial pose.

[0092] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] See also Figure 5 , is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program instruction 61 that can implement the following steps: obtain the preoperative CT image and intraoperative Xray image of the target bone block, input the preoperative CT image and the intraoperative Xray image into a regression network based on deep learning, and roughly estimate the initial spatial pose of the target bone block through the regression network; adjust the projection of the preoperative CT image based on the initial spatial pose to generate a DRR image; input the DRR image, the intraoperative Xray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate the feature point correspondence between the DRR image and the Xray image through the corresponding point relationship estimation network, and optimize the initial spatial pose of the target bone block according to the feature point correspondence relationship to obtain the optimized spatial pose. Among them, the program instruction 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network computer device, etc.) or a processor to perform all or part of the steps of the various implementation methods of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program instructions, or terminal computer devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0095] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A 2D-3D medical image registration method, characterized in that: include: Obtaining a preoperative CT image and an intraoperative X-ray image of the target bone block, inputting the preoperative CT image and the intraoperative X-ray image into a deep learning-based regression network, and roughly estimating the initial spatial position of the target bone block through the regression network; Adjusting and projecting the preoperative CT image based on the initial spatial posture to generate a DRR image; Inputting the DRR image, the intraoperative X-ray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating the feature point correspondence relationship between the DRR image and the X-ray image through the corresponding point relationship estimation network, optimizing and updating the initial spatial pose of the target bone block according to the feature point correspondence relationship, and obtaining an optimized spatial pose; The regression network includes a CNN module, a transformer module and a feature fusion module; the CNN module includes a feature extractor, two multi-layer perceptrons and an SVD decomposition module, the feature extractor uses the first 6 layers of the efficientnet-B0 network as the baseline method, the two multi-layer perceptrons are respectively a rotation regression head and a translation regression head, and the rotation regression head and the translation regression head are both three-layer multi-layer perceptrons; The initial spatial position of the target bone block is roughly estimated by the regression network, specifically: Inputting the intraoperative X-ray image into a CNN module for first feature extraction; Inputting the first feature into two multilayer perceptrons, the two multilayer perceptrons respectively outputting a rotation component and a translation component in the spatial posture of the target bone block through a rotation regression head and a translation regression head; The output of the rotation regression head is transformed into a matrix M of a set size, and the matrix M is input into the SVD decomposition module for SVD decomposition to obtain UΣV T , map the matrices U and V to the SO(3) space to obtain the rotation component R of the target bone spatial posture: R=UΣ′V T ,whereΣ′=diag(1,…,1,det(UV T )) Input the intraoperative X-ray image into a Tranformer module, perform 3D position coding and image block coding on the input image through the Tranformer module, and add the 3D position coding and image block coding results to obtain a second feature of the intraoperative X-ray image; Inputting the first feature and the second feature together into a feature fusion module for modulation, and outputting the initial spatial position of the target bone block; The 3D position encoding method of the Tranformer module is: the vertical position, horizontal position and edge information of the intraoperative X-ray image are encoded respectively by using sinusoidal position encoding, and the two-dimensional spatial coordinates (x, y) of each pixel are expanded into three-dimensional coordinates (x, y, e) by adding edge information; the encoding formula is as follows: Among them, PE is the 3D encoder information, pos D is the position of the image block in dimension D, and the variable i represents the i-th position encoding information.

2. The 2D-3D medical image registration method according to claim 1, characterized in that: The inputting the first feature and the second feature together into a feature fusion module for modulation is specifically as follows: The modulation formula is as follows: in represents the i+1th input feature of the CNN module, Represents the features output by the i-th swim-block, represents the i-th output feature of the CNN module, G(·) represents the convolution operation, BN(·) represents batch normalization, represents the multiplication operation, Represents the addition operation.

3. The 2D-3D medical image registration method according to claim 2, characterized in that: The loss function L of the deep learning-based regression network includes the rotation component loss L in SO(3) space o And the translation component loss L in Euclidean space t , the rotational component loss L o Represents the predicted value R and the true value R of the rotation component gt The geodesic distance between the two, the translation component loss L t Represents the predicted translation component t and the true value t gt The translation error between them; the loss function L is defined as follows:

4. The 2D-3D medical image registration method according to claim 3, characterized in that: The method of estimating the feature point correspondence between the DRR image and the X-ray image through the corresponding point relationship estimation network and optimizing and updating the initial spatial pose of the target bone block according to the feature point correspondence to obtain the optimized spatial pose is as follows: The corresponding point relationship estimation network uses the global matching-based optical flow algorithm GMflow to predict the offset dp at the feature point p in the DRR image, and finds the feature point p'=p+dp matching the feature point p in the intraoperative X-ray image through the offset dp, so as to obtain the feature corresponding point relationship between the DRR image and the intraoperative X-ray image; Based on the corresponding relationship of the feature points, the relative posture relationship matrix dT between the initial spatial posture and the target posture is calculated by the PPC slover algorithm, and the initial spatial posture of the target bone block is optimized and updated according to the relative posture relationship matrix dT to obtain the optimized spatial posture of the bone block; wherein, the PPC slover algorithm uses {p, p'} between the DRR image and the intraoperative X-ray image and the three-dimensional point w and its gradient g to establish the relationship between the two-dimensional motion and the three-dimensional differential motion, and the relative relationship between the initial spatial posture and the target spatial posture is obtained by the following formula δv=(dw T dt T ) T : (g×w T -g T )δv=g T w Among them, dw T is the rotation vector, dt T The rotation vector dw is converted into an axis-angle representation, where the axis r = dw / ||dw||, the angle θ = ||dw||, and the three-dimensional point w is extracted from the preoperative CT image by a 3D sobel edge extractor.

5. A 2D-3D medical image registration device using the 2D-3D medical image registration method of claim 1, characterized in that: include: Rough estimation module: used to obtain preoperative CT images and intraoperative X-ray images of the target bone block, input the preoperative CT images and intraoperative X-ray images into a regression network based on deep learning, and roughly estimate the initial spatial position of the target bone block through the regression network; DRR projection module: used to adjust the projection of the preoperative CT image based on the initial spatial posture to generate a DRR image; Refined estimation module: used to input the DRR image, intraoperative X-ray image and preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate the feature point correspondence between the DRR image and the X-ray image through the corresponding point relationship estimation network, optimize and update the initial spatial pose of the target bone block according to the feature point correspondence, and obtain the optimized spatial pose.

6. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the 2D-3D medical image registration method according to any one of claims 1 to 4; The processor is used to execute the program instructions stored in the memory to control a 2D-3D medical image registration method.

7. A storage medium, characterized in that: The device stores program instructions executable by a processor, wherein the program instructions are used to execute the 2D-3D medical image registration method according to any one of claims 1 to 4.

Images