2d / 3d medical image registration method and apparatus, computer device, and storage medium

Abstract

The present application relates to a 2D / 3D medical image registration method and apparatus, a computer device, and a storage medium. The method comprises: acquiring a preoperative CT image and an intraoperative X-ray image of a target bone block; inputting the preoperative CT image and the intraoperative X-ray image into a deep learning-based regression network, and roughly estimating an initial spatial pose of the target bone block by means of the regression network; adjusting and projecting the preoperative CT image on the basis of the initial spatial pose so as to generate a DRR image; inputting the DRR image, the intraoperative X-ray image, and the preoperative CT image into a pre-trained corresponding point relationship estimation network, and estimating a feature point correspondence between the DRR image and the X-ray image by means of the corresponding point relationship estimation network; and optimizing the initial spatial pose of the target bone block on the basis of the feature point correspondence to obtain an optimized spatial pose. In the present application, a large amount of data does not need to be manually annotated, radiation damage caused by repeated perspective is greatly reduced, and relatively efficient and accurate registration can be achieved.

Description

2D-3D medical image registration method and device, computer equipment and storage medium TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a 2D-3D medical image registration method and device, computer equipment and storage medium. BACKGROUND

[0002] In a pelvic fracture reduction surgery, an Xray fluoroscopy image is key data for a surgeon to judge the treatment situation. During the surgery, the surgeon needs to predict the spatial pose of the fractured bone block through multiple X fluoroscopy to formulate the next reduction operation. Due to the lack of visual information of the fracture area, the spatial reasoning ability of the surgeon is greatly challenged. At the same time, multiple X fluoroscopy during the surgery will cause serious radiation damage to the patient. Through 2D-3D image registration, the spatial pose of the real Xray image can be obtained. In the prior art, the commonly used 2D-3D image registration method mainly includes:

[0003] 1) Iterative optimization based on image similarity; this method takes the gradient, intensity and features of the simulated C-arm fluoroscopy projection image of the preoperative CT (Computed Tomography, electronic computed tomography) image and the intraoperative Xray image as an optimization function, and obtains the optimal parameters through multiple iterations to make the two images reach registration, and the spatial pose of the current real Xray image is obtained through the optimal parameters. However, when the initial registration error is large, the optimization problem is usually a non-convex problem, which is easy to fall into a local optimal solution, and is not suitable for pelvic fracture reduction surgery.

[0004] 2) 2D-3D registration based on reinforcement learning; this method regards the process of finding the best external parameter of the simulated C-arm to make the simulated projection image (DRR) and the real Xray image reach registration as a Markov process, defines the six-degree-of-freedom state, action function and corresponding reward function, and optimizes in the environment through an artificial agent. However, this algorithm requires a large parameter space, occupies a large amount of computing resources, and at the same time, the high-dimensional state space and large action range bring challenges to the training of reinforcement learning, which requires a long training time to stabilize.

[0005] 3) 2D-3D registration algorithm based on anatomical landmark point detection; this algorithm detects the 2D coordinates of multiple groups of anatomical landmarks on the Xray and the 3D spatial coordinates, and uses the pnp (Perspective-n-Point, a method for solving 3D to 2D point motion) algorithm to solve the spatial pose of the current pelvis. This algorithm requires a large amount of artificial labels, which is complex and time-consuming to make.

[0006] 4) 2D-3D registration based on convolutional neural network; this method directly predicts the spatial pose of the pelvis corresponding to the single Xray image through the convolutional neural network. However, due to the specificity of each pelvis fracture patient, the registration accuracy of directly predicting the pose of the bone block through the convolutional neural network is low, and it is only suitable as an initial registration scheme.

[0007] SUMMARY

[0008] The present application provides a 2D-3D medical image registration method, device, computer equipment and storage medium, aiming to at least solve one of the above technical problems in the prior art to some extent.

[0009] In order to solve the above problems, the present application provides the following technical scheme:

[0010] A 2D-3D medical image registration method, comprising:

[0011] Obtaining the preoperative CT image and the intraoperative Xray image of the target bone block, inputting the preoperative CT image and the intraoperative Xray image into a regression network based on deep learning, and roughly estimating the initial spatial pose of the target bone block through the regression network;

[0012] Adjusting and projecting the preoperative CT image based on the initial spatial pose to generate a DRR image;

[0013] Inputting the DRR image, the intraoperative Xray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating the corresponding relationship between the feature points of the DRR image and the Xray image through the corresponding point relationship estimation network, and updating the initial spatial pose of the target bone block according to the corresponding relationship of the feature points to obtain the optimized spatial pose.

[0014] The technical scheme adopted by the embodiment of the present application further comprises: the regression network comprises a CNN module, a transformer module and a feature fusion module; the CNN module comprises a feature extractor, two multilayer perceptrons and an SVD decomposition module, the feature extractor takes the first 6 layers of efficientnet-B0 network as the baseline method, the two multilayer perceptrons are a rotation regression head and a translation regression head respectively, and the rotation regression head and the translation regression head are both three-layer multilayer perceptrons.

[0015] The technical scheme adopted by the embodiment of the present application further comprises: the regression network roughly estimates the initial spatial pose of the target bone block, specifically:

[0016] Inputting the intraoperative Xray image into the CNN module for first feature extraction;

[0017] The first feature is input into two multi-layer perceptrons, and the two multi-layer perceptrons output a rotation component and a translation component in a spatial pose of the target bone block through a rotation regression head and a translation regression head respectively;

[0018] The output of the rotation regression head is transformed into a matrix M of a set size, and the matrix M is input into an SVD decomposition module for SVD decomposition to obtain UΣV T The matrices U and V are mapped to an SO(3) space to obtain a rotation component R of the spatial pose of the target bone block: R=UΣ′V T , where Σ′=diag(1,…,1,det(UV T ))

[0019] The intraoperative Xray image is input into a Tranformer module, 3D position encoding and image block encoding are performed on the input image through the Tranformer module, and the 3D position encoding and the image block encoding result are added to obtain a second feature of the intraoperative Xray image.

[0020] The first feature and the second feature are input into a feature fusion module for modulation, and an initial spatial pose of the target bone block is output.

[0021] The technical scheme adopted by the embodiment of the application further includes that the 3D position encoding manner of the Tranformer module is that the vertical position, the horizontal position and the edge information of the intraoperative Xray image are respectively encoded by using sinusoidal position encoding, and by adding the edge information, the two-dimensional spatial coordinates (x, y) of each pixel are expanded to three-dimensional coordinates (x, y, e); and the encoding formula is as follows:

[0022] wherein PE is three-dimensional encoder information, pos D is the position of the image block in the dimension D, and the variable i represents the i-th position encoding information.

[0023] The technical scheme adopted by the embodiment of the application further includes that the first feature and the second feature are input into a feature fusion module for modulation, and the modulation formula is as follows:

[0024] The modulation formula is as follows:

[0025] wherein represents the i+1 input feature of the CNN module, represents the feature output by the i-th swim-block, represents the i-th output feature of the CNN module, and G(·) represents a convolution operation, and BN(·) represents batch normalization. Represents multiplication operation. This represents an addition operation.

[0026] The technical solution adopted in this application embodiment also includes: the loss function L of the deep learning-based regression network includes the rotation component loss L in SO(3) space. o Translation component loss L in Euclidean space t The rotational component loss L O Represents the predicted value R and the true value Rg of the rotation component. t The geodesic distance between them, the translation component loss L t This represents the difference between the predicted translation component t and the actual value tg. t The translation error between them; the loss function L is defined as follows:

[0027] The technical solution adopted in this application embodiment further includes: estimating the feature point correspondence between the DRR image and the Xray image through the corresponding point relationship estimation network, and optimizing and updating the initial spatial pose of the target bone block according to the feature point correspondence to obtain the optimized spatial pose, specifically as follows:

[0028] The corresponding point relationship estimation network uses the global matching-based optical flow algorithm GMflow to predict the offset dp at feature point p in the DRR image. Then, using the offset dp, a feature point p' = p + dp that matches feature point p is found in the intraoperative X-ray image, thus obtaining the feature correspondence point relationship between the DRR image and the intraoperative X-ray image.

[0029] Based on the correspondence of the feature points, the relative pose relationship matrix dT between the initial spatial pose and the target pose is calculated using the PPC slover algorithm. The initial spatial pose of the target bone block is then optimized and updated according to the relative pose relationship matrix dT to obtain the optimized bone block spatial pose. The PPC slover algorithm utilizes the {p, p'} between the DRR image and the intraoperative X-ray image, as well as the three-dimensional point w and its gradient g, to establish the relationship between two-dimensional motion and three-dimensional differential motion. The relative relationship δv between the initial spatial pose and the target spatial pose is obtained using the following formula: δv = (dw / dt) / dt. T dt T ) T : (g×w T -n T )δv=g T w

[0030] Among them, dw T dv is a rotation vector. TFor the translation vector, the rotation vector dw is converted into an axis-angle representation, where the axis r = dw / ||dw|| and the angle Θ = ||dw||. The 3D point w is extracted from the preoperative CT image by a 3D sobel edge extractor.

[0031] Another technical scheme adopted by the embodiment of the present application is a 2D-3D medical image registration device, comprising:

[0032] The rough estimation module is configured to acquire a preoperative CT image and an intraoperative Xray image of the target bone block, input the preoperative CT image and the intraoperative Xray image into a regression network based on deep learning, and roughly estimate an initial spatial pose of the target bone block through the regression network.

[0033] The DRR projection module is configured to project the preoperative CT image based on the initial spatial pose to generate a DRR image.

[0034] The fine estimation module is configured to input the DRR image, the intraoperative Xray image, and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate a corresponding relationship between feature points of the DRR image and the Xray image through the corresponding point relationship estimation network, and optimize and update the initial spatial pose of the target bone block according to the corresponding relationship between the feature points to obtain an optimized spatial pose.

[0035] Still another technical scheme adopted by the embodiment of the present application is a computer device, comprising a processor and a memory coupled to the processor, wherein

[0036] The memory stores program instructions for implementing the 2D-3D medical image registration method.

[0037] The processor is configured to execute the program instructions stored in the memory to control the 2D-3D medical image registration method.

[0038] Still another technical scheme adopted by the embodiment of the present application is a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the 2D-3D medical image registration method.

[0039] Compared with the prior art, the 2D-3D medical image registration method, device, computer device and storage medium provided by the embodiment of the application have the beneficial effects that: the 2D-3D medical image registration method, device, computer device and storage medium provided by the embodiment of the application provide a two-stage registration framework combining coarse registration and fine registration to predict the spatial pose of the bone block of a pelvic fracture patient, in the coarse registration stage, a CNN-Transformer-based regression network is used to roughly estimate the initial spatial pose of the surgical target bone block, and the estimation of the X-ray light source rotation amount is used to improve the convergence speed and accuracy of the network; in the fine registration stage, a point-to-plane 2D-3D registration algorithm based on deep learning is used to further optimize the initial spatial pose predicted by the regression network, the initial spatial pose of the target bone block is updated through a global optical flow network and PPC, and fine adjustment is completed through a small number of iterations. The embodiment of the application can train the model only according to the preoperative CT image of the patient, and can predict the spatial pose of the bone block only by using a single X-ray image during surgery, without the need for manual labeling of a large amount of data, and the running time is relatively fast, which greatly reduces the radiation damage to doctors and patients caused by repeated perspective, gets rid of the difficulty that the image similarity-based optimization registration is easy to fall into local optimum under large displacement, and can realize relatively efficient and accurate registration under large initial registration error. BRIEF DESCRIPTION OF DRAWINGS

[0040] FIG. 1 is a flowchart of the 2D-3D medical image registration method of the embodiment of the application;

[0041] FIG. 2 is a regression network architecture diagram based on deep learning according to the embodiment of the application;

[0042] FIG. 3 is a structural schematic diagram of the 2D-3D medical image registration device according to the embodiment of the application;

[0043] FIG. 4 is a structural schematic diagram of the computer device according to the embodiment of the application;

[0044] FIG. 5 is a structural schematic diagram of the storage medium according to the embodiment of the application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0046] The terms "first", "second", "third", etc. are used only for descriptive purposes and do not connote or imply relative importance or a quantity of the indicated technical features. Thus, features defined with "first", "second", or "third" can include at least one of the features explicitly or implicitly. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited. All directional references, such as up, down, left, right, front, back, etc., used in the description of the present application are only used for convenience of explanation and are not intended to limit the scope of the present application. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include other steps or units not listed, or can optionally include other steps or units inherent to the process, method, system, product, or computer device.

[0047] Reference herein to "embodiments" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a separate or alternative embodiment. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with each other.

[0048] Specifically, please refer to FIG. 1, which is a flow chart of the 2D-3D medical image registration method of the embodiments of the present application. The 2D-3D medical image registration method of the embodiments of the present application includes the following steps:

[0049] S100: input the preoperative CT image of the target bone block and the intraoperative Xray image respectively;

[0050] S110: input the preoperative CT image and the intraoperative Xray image into the deep learning-based regression network together, and roughly estimate the initial spatial pose of the target bone block through the regression network;

[0051] In this step, as shown in FIG. 2, a regression network architecture based on deep learning of an embodiment of the present application is shown. The regression network includes a CNN (Convolutional Neural Networks) module, a transformer module, and a feature fusion module. The CNN module includes a feature extractor, two multilayer perceptrons, and an SVD decomposition module. The feature extractor uses the first 6 layers of the efficientnet-B0 network as a baseline method. The two multilayer perceptrons are a rotation regression head and a translation regression head, respectively. Both the rotation regression head and the translation regression head are three-layer multilayer perceptrons.

[0052] Based on the above network structure, the rough estimation process of the initial spatial pose of the bone block in the intraoperative Xray image by the regression network includes:

[0053] S111: inputting the intraoperative Xray image into the CNN module for first feature extraction;

[0054] S112: inputting the extracted first feature into two multilayer perceptrons, which respectively output the rotation component and the translation component in the spatial pose of the target bone block through the rotation regression head and the translation regression head. The output dimension of the rotation regression head is 9, and the output dimension of the translation regression head is 3. The specific dimension value can be set according to the actual application scenario.

[0055] S113: transforming the output of the rotation regression head into a matrix M of a set size (3x3, which can be set according to the actual application scenario), and inputting the matrix M into the SVD decomposition module for SVD decomposition to obtain UΣV T Mapping the matrices U and V to the SO(3) space to obtain the rotation component R of the spatial pose of the target bone block: R = UΣ'V T , where Σ' = diag(1,…,1,det(UV T )) (1)

[0056] S114: inputting the intraoperative Xray image into the Tranformer module, performing 3D position encoding and image block encoding on the input image through the Tranformer module, adding the 3D position encoding and the image block encoding result, obtaining the second feature of the intraoperative Xray image, and passing the second feature map to the CNN module. The 3D position encoding method of the Tranformer module is: using sinusoidal position encoding to encode the vertical position, horizontal position, and edge information of the intraoperative Xray image respectively. By adding edge information, the two-dimensional spatial coordinates (x, y) of each pixel are expanded to three-dimensional coordinates (x, y, e). The encoding formula is as follows:

[0057] In formula (2), PE is three-dimensional encoder information, pos D is the position of the image block in dimension D (i.e., the vertical position, the horizontal position, and the edge information), and the variable i represents the i-th position encoding information. Embodiments of the present application set d of each dimension to 32, thereby forming 96 positions composed of spatial positions and edge information. Specifically, the dimension d can also be set to other values according to actual application scenarios.

[0058] S115: input the first feature extracted by the CNN module and the second feature extracted by the transformer module into a feature fusion module for modulation, and output an initial spatial pose of the target bone block; wherein the output of each transformer module is modulated with the output of the i e {1, 3, 4} layer of the CNN module, and the modulation formula is as follows:

[0059] wherein represents the i+1 input feature of the CNN module, represents the feature output by the i-th swim-block, represents the i-th output feature of the CNN module, G(·) represents a convolution operation, and BN(·) represents a batch normalization, represents a multiplication operation, represents an addition operation.

[0060] In embodiments of the present application, the loss function L of the regression network based on deep learning is composed of two parts, namely, the rotation component loss L O in the SO(3) space and the translation component loss L t in the Euclidean space, the rotation component loss L o represents the geodesic distance between the predicted value R of the rotation component and the true value Rg t , and the translation component loss L t represents the translation error between the predicted translation component t and the true value tg t ; specifically, the loss function L is defined as follows:

[0061] S120: adjust and project the preoperative CT image based on the initial spatial pose roughly estimated by the regression network to generate a DRR image;

[0062] S130: input the DRR image, the Xray image, and the CT image into a pre-trained corresponding point relationship estimation network, and estimate the corresponding relationship between the feature points of the DRR image and the Xray image through the corresponding point relationship estimation network;

[0063] In this step, the corresponding point relationship estimation network is essentially a feature matching network, which uses a global matching based optical flow algorithm GMflow to estimate the feature corresponding point relationship between the DRR image and the intraoperative Xray image. The offset dp of the feature point p in the DRR image is obtained by optical flow prediction, and the feature point p' = p + dp matching the feature point p in the intraoperative Xray image is found by dp, to obtain the feature corresponding point relationship between the DRR image and the intraoperative Xray image. The loss function L of the corresponding point relationship estimation network is flow :

[0064] where f i is the i-th refined optical flow prediction value, f gt is the distance between the corresponding points under the two-dimensional projection of the DRR image and the intraoperative Xray image, γ is a scale factor, and N represents the number of optical flow updates.

[0065] S140: Based on the feature point corresponding relationship, the relative pose relationship matrix dT between the initial spatial pose and the target pose is calculated by the PPC slover algorithm, and the initial spatial pose of the target bone block is updated and optimized according to the relative pose relationship matrix dT to obtain the optimized bone block spatial pose;

[0066] In this step, a point-to-plane (PPC) 2D-3D registration algorithm based on deep learning is proposed to further optimize the initial spatial pose predicted by the regression network. Based on the feature point corresponding relationship between the DRR image and the intraoperative Xray image, the relative pose relationship matrix dT between the current initial spatial pose T and the target spatial pose is calculated by the PPC slover algorithm, and finally the initial spatial pose is updated by the relative relationship [R'|t'] = dT·[R0|t0], and the above process is repeated multiple times to continuously update the spatial pose of the target bone block, so that the DRR image and the intraoperative Xray image achieve better registration effect.

[0067] Specifically, the PPC slover algorithm uses {p, p'} between the DRR image and the intraoperative Xray image, and three-dimensional points w and their gradients g to establish the relationship between two-dimensional motion and three-dimensional differential motion, and the relative relationship δv = (dw T dt T ) T : (g×w T -n T )δv=g T w (6)

[0068] where dw T is the rotation vector, and dvT is the translation vector. Convert the rotation vector dwinto an axis-angle representation, where the axis r = dw / ||dw|| and the angle Θ = ||dw||. The 3D point w is extracted from the preoperative CT image by a 3D sobel edge extractor.

[0069] Based on the above, the 2D-3D medical image registration method of the embodiments of the application provides a two-stage registration framework combining coarse registration and fine registration to predict the spatial pose of the bone block of the pelvic fracture patient. In the coarse registration stage, a CNN-Transformer-based regression network is used to roughly estimate the initial spatial pose of the surgical target bone block, and the estimation of the X-ray light source rotation amount is used to improve the convergence speed and accuracy of the network. In the fine registration stage, a point-to-plane 2D-3D registration algorithm based on deep learning is used to further optimize the initial spatial pose predicted by the regression network. The initial spatial pose of the target bone block is updated by a global optical flow network and PPC, and fine adjustment is completed with a small number of iterations. The model of the embodiments of the application can be trained only according to the preoperative CT image of the patient, and the spatial pose of the bone block can be predicted using a single Xray image during surgery. A large amount of data does not need to be manually labeled, the running time is fast, the radiation damage to doctors and patients caused by repeated perspective is greatly reduced, the difficulty of local optimization under large displacement based on image similarity optimization registration is overcome, and efficient and accurate registration can be achieved under large initial registration error.

[0070] Please refer to FIG. 3, which is a structural schematic diagram of the 2D-3D medical image registration device of the embodiments of the application. The 2D-3D medical image registration device 40 of the embodiments of the application comprises:

[0071] The coarse estimation module 41 is configured to obtain the preoperative CT image of the target bone block and the intraoperative Xray image, input the preoperative CT image and the intraoperative Xray image into a deep learning-based regression network, and roughly estimate the initial spatial pose of the target bone block through the regression network.

[0072] The DRR projection module 42 is configured to adjust and project the preoperative CT image based on the initial spatial pose to generate a DRR image.

[0073] The fine estimation module 43 is configured to input the DRR image, the intraoperative Xray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate the corresponding relationship between the feature points of the DRR image and the Xray image through the corresponding point relationship estimation network, and update the initial spatial pose of the target bone block according to the corresponding relationship between the feature points to obtain an optimized spatial pose.

[0074] It should be noted that the information interaction, execution process and the like between the above apparatus / units are based on the same concept as the method embodiments of the present application, and specific functions and brought technical effects can be referred to the method embodiments part, which will not be repeated here.

[0075] The apparatus provided by the embodiments of the present application can be applied in the foregoing method embodiments, and details can be referred to the description of the method embodiments.

[0076] Please refer to FIG. 4, which is a schematic diagram of the structure of a computer device according to the embodiments of the present application. The computer device 50 comprises:

[0077] a memory 51 storing executable program instructions;

[0078] a processor 52 connected with the memory 51;

[0079] The processor 52 is configured to call the executable program instructions stored in the memory 51 and perform the following steps: obtaining a preoperative CT image and an intraoperative Xray image of a target bone block, inputting the preoperative CT image and the intraoperative Xray image into a deep learning-based regression network, roughly estimating an initial spatial pose of the target bone block through the regression network; adjusting and projecting the preoperative CT image based on the initial spatial pose to generate a DRR image; inputting the DRR image, the intraoperative Xray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating the corresponding relationship between the feature points of the DRR image and the Xray image through the corresponding point relationship estimation network, and optimizing the initial spatial pose of the target bone block according to the corresponding relationship of the feature points to obtain an optimized spatial pose.

[0080] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 can be an integrated circuit chip having a processing capability. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0081] Referring to FIG. 5, it is a structural schematic diagram of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 capable of realizing the following steps: acquiring a preoperative CT image and an intraoperative Xray image of a target bone block, inputting the preoperative CT image and the intraoperative Xray image into a regression network based on deep learning, roughly estimating an initial spatial pose of the target bone block through the regression network; adjusting projection of the preoperative CT image based on the initial spatial pose, generating a DRR image; inputting the DRR image, the intraoperative Xray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating a corresponding relationship between feature points of the DRR image and the Xray image through the corresponding point relationship estimation network, optimizing the initial spatial pose of the target bone block according to the corresponding relationship between the feature points, and obtaining an optimized spatial pose. The program instructions 61 can be stored in the storage medium in the form of a software product, including a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network computer device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program instructions, or a terminal computer device such as a computer, a server, a mobile phone, a tablet computer, etc. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, etc.

[0082] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0083] In addition, each of the function units in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software function unit. The above is only an implementation of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A 2D-3D medical image registration method, characterized in that, The method comprises the following steps: obtaining a preoperative CT image and an intraoperative Xray image of a target bone block, inputting the preoperative CT image and the intraoperative Xray image into a deep learning-based regression network, and roughly estimating an initial spatial pose of the target bone block through the regression network; adjusting the preoperative CT image based on the initial spatial pose to generate a DRR image; inputting the DRR image, the intraoperative Xray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimating the corresponding relationship between the feature points of the DRR image and the Xray image through the corresponding point relationship estimation network, and updating the initial spatial pose of the target bone block according to the corresponding relationship of the feature points to obtain an optimized spatial pose.

2. The 2D-3D medical image registration method of claim 1, wherein, The regression network comprises a CNN module, a transformer module and a feature fusion module; the CNN module comprises a feature extractor, two multilayer perceptrons and an SVD decomposition module, the feature extractor takes the first 6 layers of an efficientnet-B0 network as a baseline method, the two multilayer perceptrons are a rotation regression head and a translation regression head respectively, and the rotation regression head and the translation regression head are both three-layer multilayer perceptrons.

3. The 2D-3D medical image registration method of claim 2, wherein, The regression network roughly estimates the initial spatial pose of the target bone block, specifically: inputting the intraoperative Xray image into the CNN module for first feature extraction; inputting the first feature into the two multilayer perceptrons, and the two multilayer perceptrons output the rotation component and the translation component in the spatial pose of the target bone block through the rotation regression head and the translation regression head respectively; The output of the rotation regression head is transformed into a matrix M of a set size, and the matrix M is input into an SVD decomposition module for SVD decomposition to obtain UΣV T The matrices U and V are mapped to an SO(3) space to obtain the target the rotation component R of the bone block spatial pose: R = UΣ'V T , whereΣ' = diag(1,..., 1, det(UV T )) inputting the intraoperative Xray image into the Tranformer module, performing 3D position encoding and image block encoding on the input image through the Tranformer module, adding the 3D position encoding and the image block encoding result, and obtaining the second feature of the intraoperative Xray image; inputting the first feature and the second feature into the feature fusion module for modulation, and outputting the initial spatial pose of the target bone block.

4. The 2D-3D medical image registration method of claim 3, wherein, The 3D position encoding mode of the Transformer module is: the vertical position, the horizontal position and the edge information of the intraoperative Xray image are respectively encoded by using sinusoidal position encoding, and by adding the edge information, the two-dimensional space coordinates (x, y) of each pixel are extended to three-dimensional coordinates (x, y, e); the encoding formula is as follows: where PE is the three-dimensional encoder information, pos D is the position of the image block in dimension D, and the variable i denotes the i-th position encoding information.

5. The 2D-3D medical image registration method of claim 4, wherein, The first feature and the second feature are inputted into the feature fusion module for modulation, specifically: The modulation formula is as follows: wherein representing the (i+1)th input feature of the CNN module, a feature representing an output of the i-th swim-block, represents the i-th output feature of the CNN module, G(·) represents a convolution operation, and BN(·) represents batch normalization, representing a multiplication operation, representing an addition operation.

6. The 2D-3D medical image registration method of claim 5, wherein, The loss function L of the deep learning-based regression network comprises a rotation component loss L in SO(3) space O and a translation component loss L in Euclidean space t The rotation component loss L O represents a geodesic distance between a predicted value R of a rotation component and a true value Rg t The translation component loss L t represents a translation error between a predicted translation component t and a true value tg t The loss function L is defined as follows:

7. The 2D-3D medical image registration method of any one of claims 1 to 6, characterized in that, The corresponding point relationship estimation network estimates the corresponding relationship between the feature points of the DRR image and the Xray image, and updates the initial spatial pose of the target bone block according to the corresponding relationship of the feature points to obtain an optimized spatial pose, specifically: using a global matching-based optical flow algorithm GMflow to predict the offset dp of the feature point p in the DRR image through the corresponding point relationship estimation network, finding the feature point p' = p + dp matching the feature point p in the intraoperative Xray image through the offset dp, and obtaining the corresponding relationship between the feature points of the DRR image and the intraoperative Xray image. Based on the feature point correspondence, a relative pose relationship matrix dT between the initial spatial pose and the target pose is calculated through a PPC slover algorithm, and the initial spatial pose of the target bone block is updated and optimized according to the relative pose relationship matrix dT, so as to obtain an optimized bone block spatial pose; wherein the PPC slover algorithm establishes a relationship between two-dimensional motion and three-dimensional differential motion by using {p, p'} between the DRR image and the intraoperative Xray image and a three-dimensional point w and its gradient g, and the relative relationship δv=(dw T dt T ) T : where dw T is the rotation vector, dv T is the translation vector, converting the rotation vector dw into an axis-angle representation, where the axis r = dw / ||dw|| and the angle θ = ||dw||, the 3D point w is extracted from preoperative CT images by a 3D sobel edge extractor.

8. A 2D-3D medical image registration apparatus, characterized by, The method comprises the following steps: The rough estimation module is configured to acquire a preoperative CT image and an intraoperative X-ray image of a target bone block, input the preoperative CT image and the intraoperative X-ray image into a deep learning-based regression network, and coarsely estimate an initial spatial pose of the target bone block through the regression network. The DRR projection module is configured to project the preoperative CT image based on the initial spatial pose to generate a DRR image. The fine estimation module is configured to input the DRR image, the intraoperative X-ray image and the preoperative CT image into a pre-trained corresponding point relationship estimation network, estimate a corresponding relationship between feature points of the DRR image and the X-ray image through the corresponding point relationship estimation network, and update the initial spatial pose of the target bone block based on the corresponding relationship to obtain an optimized spatial pose.

9. A computer device, comprising: The computer device comprises a processor and a memory coupled to the processor, wherein The memory stores program instructions for implementing the 2D-3D medical image registration method of any one of claims 1-7. The processor is configured to execute the program instructions stored in the memory to control the 2D-3D medical image registration method.

10. A storage medium, characterized by The memory stores processor-executable program instructions for executing the 2D-3D medical image registration method of any one of claims 1-7.

Images