Image registration method and apparatus, and orthopedic surgery navigation system based on image registration

Abstract

The present application provides an image registration method and apparatus, and an orthopedic surgery navigation system based on image registration. The image registration method comprises: on the basis of a three-dimensional image, an anteroposterior two-dimensional image, and a lateral two-dimensional image of an object to be registered, and internal parameters of an X-ray machine, generating an initial object pose of said object and an initial anteroposterior pose and an initial lateral pose of the X-ray machine; performing registration optimization on the initial object pose to generate a target object pose of said object; performing registration optimization on the initial lateral pose of the X-ray machine to generate a target lateral pose obtained after registration; and integrating the target object pose of said object and the target lateral pose and the initial anteroposterior pose of the X-ray machine into a registration optimization result. The solution provided by the present application not only can optimize the pose of the object, but also can accurately register the lateral pose of the X-ray machine when the position of the X-ray machine is inaccurate, thereby improving the accuracy and usability of the registration method.

Description

Image registration method, device and orthopedic surgery navigation system based on image registration Technical Field

[0001] The present application relates to the field of medical imaging, and in particular to an image registration method and device, and an orthopedic surgery navigation system based on image registration. Background Art

[0002] At present, robot-assisted spinal surgery has been promoted and applied clinically. The navigation system is a key part of robot-assisted spinal surgery. In the existing technology, there are mainly the following methods for robot-assisted spinal navigation: (1) navigation imaging method based on two-dimensional C-arm X-ray machine; (2) intraoperative navigation imaging method based on three-dimensional C-arm image; (3) navigation imaging method based on preoperative three-dimensional CT reconstruction and planning. Among them, the navigation imaging method based on two-dimensional C-arm X-ray machine is the most widely used, but it is not intuitive enough and the surgical efficiency is low; the intraoperative navigation imaging method based on three-dimensional C-arm image has less than ideal imaging quality, is expensive, and has a low equipment penetration rate; the navigation imaging method based on preoperative three-dimensional CT reconstruction and planning cannot accurately perform the surgery without an intraoperative spatial positioning algorithm.

[0003] The navigation imaging method of preoperative 3D CT reconstruction and planning based on 2D / 3D registration can avoid the above problems, but this method requires high 2D / 3D registration accuracy to meet clinical needs.

[0004] Summary of the Invention

[0005] To address the problems existing in the prior art, this application proposes an image registration method and device. This method uses a three-dimensional image, two-dimensional frontal and lateral images, and the internal parameters of an X-ray machine to derive an initial object pose. This initial object pose and the X-ray machine's lateral pose are optimized based on the two-dimensional frontal and lateral images to derive the target object pose and the optimized lateral pose of the object to be registered. The proposed solution not only optimizes the object pose, but also optimizes the lateral pose of the X-ray machine when the X-ray machine is not positioned correctly, thereby improving the accuracy and ease of use of the registration method.

[0006] According to the first aspect of the present application, an image registration method is proposed, which may include the following steps: generating an initial object pose of the object to be registered and an initial anteroposterior pose and an initial lateral pose of the X-ray machine based on a three-dimensional image, an orthogonal two-dimensional image, a lateral two-dimensional image of the object to be registered and an internal parameter of the X-ray machine; performing registration optimization on the initial object pose based on the orthogonal two-dimensional image and the lateral two-dimensional image to generate a target object pose of the object to be registered; performing registration optimization on the initial lateral pose of the X-ray machine based on the target object pose and the lateral two-dimensional image to generate a target lateral pose of the registered X-ray machine; integrating the target object pose of the object to be registered, the target lateral pose of the X-ray machine and the initial orthogonal pose of the X-ray machine into a registration optimization result.

[0007] According to some embodiments, the initial lateral posture of the X-ray machine is aligned and optimized according to the target object posture and the lateral two-dimensional image to generate the target lateral posture of the X-ray machine after alignment, which can include the following steps: generating a first virtual object posture according to the target object posture and the lateral two-dimensional image; generating the target lateral posture of the X-ray machine after alignment according to the relative relationship between the first virtual object posture and the lateral two-dimensional image.

[0008] According to some embodiments, generating a first virtual object pose based on the target object pose and the lateral two-dimensional image may include the following steps: S201, taking the target object pose as the current object pose; S202, generating a first virtual lateral two-dimensional image corresponding to the current object pose; S203, calculating a first similarity between the lateral two-dimensional image and the first virtual lateral two-dimensional image; S204, taking the current object pose as the first virtual object pose if the first similarity meets the first object pose condition; S205, adjusting the current object pose if the first similarity does not meet the first object pose condition, and returning to step S202.

[0009] According to some embodiments, performing registration optimization on the initial object pose based on the anteroposterior two-dimensional image and the lateral two-dimensional image, and generating the target object pose of the object to be registered may include: generating a first temporary object pose based on the initial object pose, the anteroposterior two-dimensional image, and the lateral two-dimensional image; and generating the target object pose based on the first temporary object pose and the anteroposterior two-dimensional image.

[0010] According to some embodiments, generating a first temporary object pose based on an initial object pose, an orthogonal two-dimensional image, and a lateral two-dimensional image may include: S301, taking the initial object pose as the current object pose; S302, generating a second virtual orthogonal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the current object pose; S303, calculating a second similarity between the orthogonal two-dimensional image and the second virtual orthogonal two-dimensional image, and a third similarity between the lateral two-dimensional image and the second virtual lateral two-dimensional image; S304, taking the current object pose as the first temporary object pose if the second similarity and the third similarity meet a second object pose condition; S305, adjusting the current object pose if the second similarity and the third similarity do not meet the second object pose condition, and returning to step S302.

[0011] According to some embodiments, generating a target object pose based on a first temporary object pose and an orthogonal two-dimensional image may include: S401, taking the first temporary object pose as the current object pose; S402, generating a third virtual orthogonal two-dimensional image corresponding to the current object pose; S403, calculating a fourth similarity between the orthogonal two-dimensional image and the third virtual orthogonal two-dimensional image; S404, taking the current object pose as the target object pose if the fourth similarity satisfies a third object pose condition; S405, adjusting the current object pose if the fourth similarity does not satisfy the third object pose condition, and returning to step S402.

[0012] According to some embodiments, adjusting the pose of the object may include: performing a rotation or translation operation on the pose of the object to be registered.

[0013] According to the second aspect of the present application, an image registration device is proposed, which may include an initial pose generation module, a target object pose generation module, a target lateral pose generation module and a registration result integration module, wherein: the initial pose generation module can be used to generate the initial object pose of the object to be registered and the initial anteroposterior pose and initial lateral pose of the X-ray machine based on the three-dimensional image, the frontal two-dimensional image, the lateral two-dimensional image and the internal parameters of the X-ray machine of the object to be registered; the target object pose generation module can be used to perform registration optimization on the initial object pose based on the frontal two-dimensional image and the lateral two-dimensional image, and generate the target object pose of the object to be registered; the target lateral pose generation module can be used to perform registration optimization on the initial lateral pose of the X-ray machine based on the target object pose and the lateral two-dimensional image, and generate the target lateral pose of the registered X-ray machine; the registration result integration module is used to integrate the target object pose of the object to be registered, the target lateral pose of the X-ray machine and the initial frontal pose of the X-ray machine into a registration optimization result.

[0014] According to the third aspect of the present application, an orthopedic surgical navigation system based on image registration is proposed, which may include: a three-dimensional CT device, which can be used to capture a three-dimensional image of the part to be operated; an X-ray machine, which can be used to capture an anteroposterior two-dimensional image and a lateral two-dimensional image of the part to be operated, and provide the internal parameters of the X-ray machine to a processing module; the processing module can be used to determine the surgical operation plan based on the three-dimensional image; the processing module can also be used to generate a registration optimization result based on the three-dimensional image, the anteroposterior two-dimensional image, the lateral two-dimensional image and the internal parameters of the X-ray machine using the method described in the first aspect of the present application, and the registration optimization result is generated. The quantification results include the target object posture of the object to be registered, the target lateral posture of the X-ray machine and the initial orthostatic posture of the X-ray machine; the coordinates of the surgical operation plan are converted from the CT image coordinate system to the patient coordinate system according to the target object posture; the coordinates of the surgical operation plan are converted from the patient coordinate system to the X-ray machine three-dimensional coordinate system according to the target lateral posture and the initial orthostatic posture; the coordinates of the surgical operation plan are converted from the X-ray machine three-dimensional coordinate system to the orthostatic X-ray two-dimensional coordinate system and the lateral X-ray two-dimensional coordinate system according to the internal reference of the X-ray machine; the operating robot can be used to operate the operation part according to the surgical operation plan.

[0015] According to the fourth aspect of the present application, an electronic device is proposed, which may include: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor executes the method described in the first aspect of the present application.

[0016] According to the fifth aspect of the present application, a non-transitory computer-readable storage medium is proposed, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor executes the method described in the first aspect of the present application.

[0017] The solution proposed in this application not only considers optimizing the posture of the object, but also considers optimizing the lateral posture of the X-ray machine. According to the solution proposed in this application, when the actual position of the X-ray machine is different from the calculated theoretical position, accurate image registration can still be achieved by optimizing the posture of the X-ray machine, thereby improving the availability and accuracy of image registration. At the same time, the orthopedic surgery navigation system proposed in this application uses the image registration method proposed in this application for registration, which can improve the registration accuracy, more accurately achieve intraoperative spatial positioning, and then accurately perform surgery to meet clinical needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by this application.

[0019] FIG1 is a flow chart of the image registration method of the present application;

[0020] FIG2 is a flow chart of generating a first virtual object posture according to a target object posture and a lateral two-dimensional image in the image registration method of the present application;

[0021] 3 is a flow chart of generating a first temporary object pose according to an initial object pose, an anteroposterior two-dimensional image, and a lateral two-dimensional image in the image registration method of the present application;

[0022] FIG4 is a flow chart of the image registration method of the present application for generating a target object pose based on a first temporary object pose and an orthogonal two-dimensional image;

[0023] FIG5 is a schematic diagram of an image registration device of the present application;

[0024] FIG6 is a schematic diagram of an orthopedic surgery navigation system based on image registration according to the present application;

[0025] FIG7 is a structural diagram of an electronic device of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0027] Figure 1 is a flow chart of the image registration method of the present application. As shown in Figure 1, the method includes the following steps.

[0028] In some specific embodiments, the X-ray machine is a C-arm X-ray machine. The method proposed in this application is described in detail below using the C-arm X-ray machine as an example.

[0029] Step S101 , generating an initial object pose of the object to be registered and an initial anteroposterior pose and an initial lateral pose of the C-arm X-ray machine based on the 3D image, the anteroposterior 2D image, the lateral 2D image and the internal parameters of the C-arm X-ray machine of the object to be registered.

[0030] In some specific embodiments, the three-dimensional image is a three-dimensional CT image taken before surgery. The three-dimensional image includes a named field for the object to be registered and corresponding three-dimensional image guidance information, wherein the coordinates of the points in the three-dimensional image are in the CT image coordinate system. In some specific embodiments, the two-dimensional image is an X-ray image taken by a C-arm X-ray machine during surgery, and the two-dimensional image includes an anteroposterior two-dimensional image and a lateral two-dimensional image. The anteroposterior two-dimensional image and the lateral two-dimensional image respectively include corresponding named fields and two-dimensional image guidance information for the anteroposterior and lateral objects to be registered, wherein the coordinates of the points in the anteroposterior two-dimensional image and the lateral two-dimensional image are in the X-ray two-dimensional coordinate system. In some specific embodiments, the object pose of the object to be registered is a conversion matrix of the coordinates of the object to be registered from the CT image coordinate system to the patient coordinate system.

[0031] In some specific embodiments, a C-arm X-ray machine is used to capture anterior and lateral two-dimensional images during surgery. The internal parameters of the C-arm X-ray machine include the focal length of the C-arm X-ray machine, the horizontal coordinates and vertical coordinates of the viewpoint center of the C-arm X-ray machine in the captured X-ray image, wherein the horizontal coordinates and the vertical coordinates refer to the coordinates in the X-ray two-dimensional coordinate system. In some specific embodiments, the external parameters of the C-arm X-ray machine are calculated using the PNP (Perspective-n-Point) algorithm based on the three-dimensional image, the anteroposterior two-dimensional image, the lateral two-dimensional image and the internal parameters of the C-arm X-ray machine. The external parameters of the C-arm X-ray machine include the initial anteroposterior posture and the initial lateral posture of the C-arm X-ray machine. In some specific embodiments, the conversion matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine is determined based on the three-dimensional image guidance information and the two-dimensional image guidance information of the object to be registered. In some specific embodiments, the initial object pose of the object to be registered is obtained based on the transformation matrix from the CT image coordinate system to the X-ray machine three-dimensional coordinate system, the external parameters of the C-arm X-ray machine, and the internal parameters of the C-arm X-ray machine.

[0032] Step S102 : performing registration optimization on the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate a target object posture of the object to be registered.

[0033] In some specific embodiments, a first temporary object pose is generated based on the initial object pose, the anteroposterior two-dimensional image and the lateral two-dimensional image, and a DRR (Digitally Reconstructed Radiographs) method is used to generate a virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the initial object pose. Optionally, the DRR method is specifically implemented by using a line integral method. In some specific embodiments, a second similarity between the virtual anteroposterior two-dimensional image and the anteroposterior two-dimensional image generated by the DRR, and a third similarity between the virtual lateral two-dimensional image and the lateral two-dimensional image are calculated. In some specific embodiments, when the second similarity and the third similarity do not meet the second object pose condition, the object pose is adjusted, the virtual anteroposterior two-dimensional image and the virtual lateral two-dimensional image corresponding to the adjusted object pose are regenerated, and the similarity is recalculated; when the second similarity and the third similarity meet the second object pose condition, the object pose of the current object to be registered is used as the first temporary object pose. Optionally, the second object pose condition is that a difference between two adjacent iterations of the second similarity is greater than a preset second similarity threshold, and a difference between two adjacent iterations of the third similarity is greater than a preset third similarity threshold.

[0034] In some specific embodiments, a target object pose is generated based on the first temporary object pose and the orthogonal two-dimensional image. A third virtual orthogonal two-dimensional image corresponding to the first temporary object pose is generated using the DRR method, and a fourth similarity between the third virtual orthogonal two-dimensional image and the orthogonal two-dimensional image is calculated. In some specific embodiments, when the fourth similarity does not meet the third object pose condition, the first temporary object pose is adjusted, and a third virtual orthogonal two-dimensional image corresponding to the adjusted first temporary object pose is regenerated, and a fourth similarity between the adjusted third virtual orthogonal two-dimensional image and the orthogonal two-dimensional image is calculated; when the fourth similarity meets the third object pose condition, the current first temporary object pose is used as the target object pose. Optionally, the third object pose condition is that the difference between two adjacent iterations of the fourth similarity is greater than a preset fourth similarity threshold.

[0035] Step S103 , performing registration optimization on the initial lateral posture of the C-arm X-ray machine according to the target object posture and the lateral two-dimensional image, and generating a target lateral posture of the C-arm X-ray machine after registration.

[0036] In some specific embodiments, a PNP (Perspective-n-Point) algorithm is used to calculate the initial lateral pose of the C-arm X-ray machine based on the lateral two-dimensional image of the C-arm X-ray machine. A first virtual object pose is generated based on the target object pose and the initial lateral pose of the C-arm X-ray machine. A first virtual lateral two-dimensional image corresponding to the first virtual object pose is generated using a DRR method, and a first similarity between the first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. In some specific embodiments, when the first similarity does not meet a first object pose condition, the target object pose is adjusted, a first virtual lateral two-dimensional image corresponding to the adjusted object pose is regenerated, and a first similarity between the adjusted first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. When the first similarity meets the first object pose condition, the current object pose is used as the first virtual object pose. Optionally, the first object pose condition is that the difference between two adjacent iterations of the first similarity is greater than a preset first similarity threshold.

[0037] In some specific embodiments, the registered target lateral posture is generated based on the relative relationship between the first virtual object posture and the lateral two-dimensional image, the first transition posture is generated based on the first virtual object posture, and the first virtual object posture is inverted to obtain the inverse as the first transition posture.

[0038] In some specific embodiments, a PNP algorithm is used to calculate an initial lateral pose of the C-arm X-ray machine, and the first transition pose is multiplied by the current initial lateral pose of the C-arm X-ray machine to obtain a target lateral pose of the C-arm X-ray machine after registration. In some specific embodiments, a relative relationship exists between the first virtual object pose and the initial lateral pose. Based on the relative relationship, the lateral pose of the C-arm X-ray machine is calculated when the object pose remains at the target object pose, and the lateral pose at this time is determined as the target lateral pose of the C-arm X-ray machine.

[0039] In step S104 , the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine, and the initial anteroposterior posture of the C-arm X-ray machine are integrated into a registration optimization result.

[0040] In some specific embodiments, in step S104, the target object pose of the object to be registered is the conversion matrix of coordinates from the CT image coordinate system to the patient coordinate system after registration optimization. The target lateral pose and initial anteroposterior pose of the C-arm X-ray machine are the conversion matrices of coordinates from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine after registration optimization. In some specific embodiments, in step S104, the target object pose of the object to be registered, the target lateral pose of the C-arm X-ray machine, and the initial anteroposterior pose of the C-arm X-ray machine are integrated to generate a registration optimization result, which is provided to a subsequent operator for operation.

[0041] According to the implementation scheme shown in Figure 1, the method proposed in the present application determines the initial object posture of the object to be registered in step S101; in step S102, the initial object posture of the object to be registered is optimized to obtain the target object posture, that is, the conversion matrix of the coordinates from the CT image coordinate system to the patient coordinate system, thereby improving the accuracy of converting the coordinates of the points on the object to be registered from the CT image coordinate system to the patient coordinate system; in step S103, the lateral posture of the C-arm X-ray machine is optimized to improve the accuracy of converting the coordinates of the points on the object to be registered from the patient coordinate system to the C-arm three-dimensional coordinate system; in step S104, the registration optimization results are integrated to facilitate the use of the registration optimization results for subsequent operations.

[0042] The solution proposed in this application not only considers optimizing the object's posture, but also the lateral posture of the C-arm X-ray machine. According to the solution proposed in this application, even when the actual position of the C-arm X-ray machine differs from the calculated theoretical position, accurate image registration can still be achieved by optimizing the C-arm X-ray machine's posture, thereby improving the usability and accuracy of image registration.

[0043] Figure 2 is a flow chart of generating a first virtual object pose based on a target object pose and a lateral two-dimensional image in the image registration method of the present application. As shown in Figure 2, generating a first virtual object pose based on a target object pose and a lateral two-dimensional image includes the following steps.

[0044] S201: Taking the target object posture as the current object posture.

[0045] S202: Generate a first virtual lateral two-dimensional image corresponding to the current object posture.

[0046] S203: Calculate a first similarity between the lateral two-dimensional image and the first virtual lateral two-dimensional image.

[0047] S204 : When the first similarity satisfies a first object pose condition, use the current object pose as a first virtual object pose.

[0048] In some specific embodiments, a PNP (Perspective-n-Point) algorithm is used to calculate an initial lateral pose of the C-arm X-ray machine based on the lateral two-dimensional image of the C-arm X-ray machine. In some specific embodiments, the current object pose is a target object pose, a first virtual object pose is generated based on the target object pose and the initial lateral pose of the C-arm X-ray machine, and a first virtual lateral two-dimensional image corresponding to the first virtual object pose is generated using a DRR method. A first similarity between the first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated, optionally using a normalized cross-correlation coefficient method to calculate the similarity.

[0049] The Normalized Cross-Correlation Coefficient (NCC) method is a commonly used cost function that is robust to scaling and translation of the distribution. This method first requires calculating the mean and standard deviation of the image.

[0050] The mean and variance of image K are defined as follows:

[0051] Among them, μ is the mean, σ is the variance, Ω is the total number of pixels, K(p) is the size of the image pixel, and p represents the position of the pixel in the image, which can be represented by the two-dimensional coordinates (u, v).

[0052] Therefore, the NCC is calculated as follows:

[0053] Among them, I, J represent two images I and J, J' is the image J through the transformation T -1 The transformed image is in the same time domain space as image I.

[0054] To ensure a more stable registration process and eliminate the situation where the X-ray and DRR ranges differ, in some specific embodiments, the Sobel operator is first used to calculate the gradients in the X and Y directions for the two images I and J. The NCCs of the X and Y gradients are then calculated, and the sum of these two NCCs is used as the image similarity. See the following formula, where S_GNCC is the NCC of the gradient image.

[0055] Among them, S_GNCC is the sum of the NCC of the two gradient images of X gradient and Y gradient, C r is the coordinate of the center position in the X direction, C c is the coordinate of the center position in the Y direction, r is the radius, S_GNCC(I1,I2,C r ,C c , r) is C r and C c The similarity of a square area with centered at 2r and a side length of 2r, I1 and I2 represent two images in the same coordinate system. Optionally, I1 is image I and I2 is image J'.

[0056] At the same time, in order to avoid causing the result to deviate too far from the initial value, which may lead to some unexpected extreme situations, in some embodiments, the present application also applies a regularization method to prevent the above situation. The regularization equation adopts the following formula:

[0057] Wherein σ is the variance of translation or rotation, Folded-Norm refers to the normal distribution of translation, and the parameters in the embodiment of the present application are selected as 50 mm for translation and 60 degrees for rotation.

[0058] This formula ensures that the value of the regularization function is large when the independent variable is larger than the variance.

[0059] The final similarity formula is as follows:

[0060] Where SE(3) is the three-dimensional special Euclidean group of rigid body transformation motion (rotation and translation), and θ is the position change in Euclidean space. S is the similarity calculation formula, R is the function of the position change of θ in Euclidean space, λ is an adjustable superderivative, usually with a weight of 0.1, and P(I;θ) is the two-dimensional projection of the three-dimensional image I after the θ transformation.

[0061] In some specific embodiments, the first similarity satisfies a first object pose condition, and the current object pose is used as the first virtual object pose. Optionally, the first object pose condition is that the difference between two adjacent iterations of the first similarity is greater than a preset first similarity threshold.

[0062] S205 , when the first similarity does not satisfy the first object posture condition, adjust the current object posture and return to step S202 .

[0063] In some specific embodiments, if the first similarity does not meet the first object pose condition, the object pose of the object to be registered is rotated or translated. In some specific embodiments, a two-stage method is used to adjust the pose of the object to be registered, that is, first adjust the low-resolution image, then use the high-resolution image, use the covariance matrix adaptive evolutionary strategy optimizer to adjust the low-resolution image, and use a faster gradient-independent optimization algorithm to adjust the high-resolution image. The specific steps of the two-stage method are as follows:

[0064] In the first stage, in some specific embodiments, a fixed-resolution image of 256x256 is selected, and the Covariance Matrix Adaptive Evolutionary Strategy Optimizer (CMA-ES) is used. The advantage of this optimizer is that it does not require the calculation of the gradient equation of the loss function, and the optimization effect is very good. It is the algorithm with the least sensitivity to initial values ​​among modern optimizers. However, this optimizer needs to calculate the loss function multiple times during each iteration, which makes it very slow. The CMA-ES algorithm requires the initial value of the equation to be optimized, the optimization parameter sequence, and the variance of each parameter as input.

[0065] In some specific embodiments, the initial value of the equation to be optimized used by the CMA-ES algorithm is the initial pose matrix from the patient coordinate system to the CT coordinate system solved by the PNP algorithm mentioned above; the optimization parameter list is the translation and rotation 6DOF degrees of freedom (6 parameters) of the object to be aligned; the translation variance and the XYZ input variance are both 5 mm, the three rotation variances are 15 degrees, and CMA-ES samples 100 times each time.

[0066] In the second stage, in some specific embodiments, the image size after downsampling by 1 / 2 is used for registration optimization. If the resolution is less than 256, the original X-ray size is used directly. In this case, slower evolutionary algorithms cannot be used as optimizers. However, for ease of solution, a gradient-independent optimization algorithm is still required. For example, BOBYQA and NEWUOA are optimizers that do not require extensive loss function calculations per iteration and can be used.

[0067] In some specific embodiments, when the first similarity does not meet the first object posture condition, the current object posture is adjusted, and the process returns to step S202 to regenerate the corresponding virtual two-dimensional image and calculate the first similarity.

[0068] FIG3 is a flow chart of the image registration method of the present application for generating a first temporary object pose based on an initial object pose, an anteroposterior 2D image, and a lateral 2D image. As shown in FIG3 , generating the first temporary object pose based on the initial object pose, the anteroposterior 2D image, and the lateral 2D image includes the following steps.

[0069] S301: Taking the initial object pose as the current object pose.

[0070] S302 : Generate a second virtual frontal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the current object posture.

[0071] S303 , calculating a second similarity between the anteroposterior two-dimensional image and the second virtual anteroposterior two-dimensional image, and a third similarity between the lateral two-dimensional image and the second virtual lateral two-dimensional image.

[0072] S304 : When the second similarity and the third similarity satisfy a second object pose condition, use the current object pose as a first temporary object pose.

[0073] In some specific embodiments, the current object pose is the initial object pose, and a second virtual orthogonal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the initial object pose are generated using a DRR method. In some specific embodiments, a second similarity is calculated between the second virtual orthogonal two-dimensional image and the orthogonal two-dimensional image, and a third similarity is calculated between the second virtual lateral two-dimensional image and the lateral two-dimensional image. Optionally, the similarity is calculated using a normalized cross-correlation coefficient method. In some specific embodiments, the second similarity and the third similarity meet a second object pose condition, and the current object pose is used as the first temporary object pose. Optionally, the second object pose condition is that the difference between two adjacent iterations of the second similarity is greater than a preset second similarity threshold, and the difference between two adjacent iterations of the third similarity is greater than a preset third similarity threshold.

[0074] S305 , when the second similarity and the third similarity do not satisfy the second object posture condition, adjust the current object posture and return to step S302 .

[0075] In some specific embodiments, if the second similarity and the third similarity do not meet the second object pose condition, the pose of the object to be registered is rotated or translated. Optionally, a two-stage method is used to adjust the pose of the object to be registered, that is, first adjust the low-resolution image, then use the high-resolution image, use the covariance matrix adaptive evolutionary strategy optimizer to adjust the low-resolution image, and use a faster gradient-independent optimization algorithm to adjust the high-resolution image. In some specific embodiments, if the second similarity and the third similarity do not meet the second object pose condition, the current object pose is adjusted, and the process returns to step S302 to regenerate the corresponding virtual two-dimensional image and calculate the second similarity and the third similarity.

[0076] Figure 4 is a flow chart of generating a target object pose based on a first temporary object pose and an orthogonal two-dimensional image in the image registration method of the present application. As shown in Figure 4, generating a target object pose based on a first temporary object pose and an orthogonal two-dimensional image includes the following steps.

[0077] S401: Using the first temporary object pose as the current object pose.

[0078] S402: Generate a third virtual orthotopic two-dimensional image corresponding to the current object posture.

[0079] S403: Calculate a fourth similarity between the orthogonal two-dimensional image and the third virtual orthogonal two-dimensional image.

[0080] S404 : When the fourth similarity satisfies the third object posture condition, the current object posture is used as the target object posture.

[0081] In some specific embodiments, the current object pose is a first temporary object pose, and a third virtual orthogonal two-dimensional image corresponding to the first temporary object pose is generated using a DRR method. A fourth similarity is calculated between the third virtual orthogonal two-dimensional image and the orthogonal two-dimensional image. Optionally, the similarity is calculated using a normalized cross-correlation coefficient method. In some specific embodiments, if the fourth similarity satisfies a third object pose condition, the current object pose is used as the target object pose. Optionally, the third object pose condition is that the difference between two consecutive iterations of the fourth similarity is greater than a preset fourth similarity threshold.

[0082] S405 : If the fourth similarity does not satisfy the third object posture condition, adjust the current object posture and return to step S402 .

[0083] In some specific embodiments, when the fourth similarity does not meet the third object pose condition, the pose of the object to be registered is adjusted by rotation or translation. Optionally, the pose of the object to be registered is adjusted using a two-stage method, that is, first adjusting the low-resolution image, then using the high-resolution image, using a covariance matrix adaptive evolutionary strategy optimizer to adjust the low-resolution image, and using a faster gradient-independent optimization algorithm to adjust the high-resolution image. In some specific embodiments, when the fourth similarity does not meet the third object pose condition, the current object pose is adjusted, and the process returns to step S402 to regenerate the corresponding virtual two-dimensional image and calculate the fourth similarity.

[0084] Figure 5 is a schematic diagram of the image registration device of the present application. As shown in Figure 5, the device includes an initial pose generation module, a target object pose generation module, a target lateral pose generation module, and a registration result integration module.

[0085] In Figure 5, the initial pose generation module generates the initial object pose of the object to be registered and the initial anteroposterior pose and initial lateral pose of the C-arm X-ray machine based on the three-dimensional image, the two-dimensional image of the frontal view, the two-dimensional image of the lateral view and the internal parameters of the C-arm X-ray machine.

[0086] In some specific embodiments, the three-dimensional image is a three-dimensional CT image taken before surgery. The three-dimensional image includes a named field for the object to be registered and corresponding three-dimensional image guidance information, wherein the coordinates of the points in the three-dimensional image are in the CT image coordinate system. In some specific embodiments, the two-dimensional image is an X-ray image taken by a C-arm X-ray machine during surgery. The anteroposterior two-dimensional image and the lateral two-dimensional image include corresponding named fields for the anteroposterior and lateral objects to be registered and two-dimensional image guidance information, wherein the coordinates of the points in the anteroposterior two-dimensional image and the lateral two-dimensional image are in the X-ray two-dimensional coordinate system. In some specific embodiments, the object pose of the object to be registered is a conversion matrix of the coordinates of the object to be registered from the CT image coordinate system to the patient coordinate system.

[0087] In some embodiments, a C-arm X-ray machine is used to capture two-dimensional anteroposterior and lateral images during surgery. The internal parameters of the C-arm X-ray machine include the focal length of the C-arm and the horizontal and vertical coordinates of the C-arm's viewpoint center in the captured X-ray image, where the horizontal and vertical coordinates refer to coordinates in the two-dimensional X-ray coordinate system. In some embodiments, the external parameters of the C-arm X-ray machine are calculated using a PNP (Perspective-n-Point) algorithm based on the three-dimensional image, the two-dimensional anteroposterior image, the two-dimensional lateral image, and the internal parameters of the C-arm X-ray machine. The external parameters of the C-arm X-ray machine include the initial anteroposterior and initial lateral poses of the C-arm X-ray machine. In some embodiments, a transformation matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine is determined based on the three-dimensional image guidance information and the two-dimensional image guidance information of the object to be registered. In some embodiments, the initial object pose of the object to be registered is obtained based on the transformation matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine, the external parameters of the C-arm X-ray machine, and the internal parameters of the C-arm X-ray machine.

[0088] In FIG5 , the target object pose generation module performs registration optimization on the initial object pose according to the frontal two-dimensional image and the lateral two-dimensional image to generate the target object pose of the object to be registered.

[0089] In some specific embodiments, a first temporary object pose is generated based on an initial object pose, an anteroposterior two-dimensional image, and a lateral two-dimensional image. A virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the initial object pose are generated using a DRR (Digitally Reconstructed Radiographs) method. The DRR method is specifically implemented using a line integral method. In some specific embodiments, a second similarity between the virtual anteroposterior two-dimensional image and the anteroposterior two-dimensional image generated by the DRR, and a third similarity between the virtual lateral two-dimensional image and the lateral two-dimensional image are calculated. In some specific embodiments, when the second similarity and the third similarity do not meet the second object pose condition, the object pose is adjusted, and a virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the adjusted object pose are regenerated, and the similarity is recalculated. In some specific embodiments, when the second similarity and the third similarity meet the second object pose condition, the object pose of the current object to be registered is used as the first temporary object pose. Optionally, the second object pose condition is that a difference between two adjacent iterations of the second similarity is greater than a preset second similarity threshold, and a difference between two adjacent iterations of the third similarity is greater than a preset third similarity threshold.

[0090] In some specific embodiments, a target object pose is generated based on the first temporary object pose and the orthogonal two-dimensional image. A third virtual orthogonal two-dimensional image corresponding to the first temporary object pose is generated using the DRR method, and a fourth similarity between the third virtual orthogonal two-dimensional image and the orthogonal two-dimensional image is calculated. In some specific embodiments, when the fourth similarity does not meet the third object pose condition, the first temporary object pose is adjusted, and a third virtual orthogonal two-dimensional image corresponding to the adjusted first temporary object pose is regenerated, and a fourth similarity between the adjusted third virtual orthogonal two-dimensional image and the orthogonal two-dimensional image is calculated; when the fourth similarity meets the third object pose condition, the current first temporary object pose is used as the target object pose. Optionally, the third object pose condition is that the difference between two adjacent iterations of the fourth similarity is greater than a preset fourth similarity threshold.

[0091] In FIG5 , the target lateral posture generation module performs registration optimization on the initial lateral posture of the C-arm X-ray machine according to the target object posture and the lateral two-dimensional image, and generates the target lateral posture of the C-arm X-ray machine after registration.

[0092] In some specific embodiments, a PNP (Perspective-n-Point) algorithm is used to calculate the initial lateral pose of the C-arm X-ray machine based on the lateral two-dimensional image of the C-arm. A first virtual object pose is generated based on the target object pose and the initial lateral pose of the C-arm X-ray machine. A first virtual lateral two-dimensional image corresponding to the target object pose is generated using a DRR method, and a first similarity between the first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. In some specific embodiments, when the first similarity does not meet a first object pose condition, the target object pose is adjusted, a first virtual lateral two-dimensional image corresponding to the adjusted target object pose is regenerated, and a first similarity between the adjusted first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. In some specific embodiments, when the first similarity meets the first object pose condition, the current object pose is used as the first virtual object pose. Optionally, the first object pose condition is that the difference between two adjacent iterations of the first similarity is greater than a preset first similarity threshold.

[0093] In some specific embodiments, the registered target lateral posture is generated based on the relative positional relationship between the first virtual object posture and the lateral two-dimensional image, the first transition posture is generated based on the first virtual object posture, and the first virtual object posture is inverted to obtain the inverse as the first transition posture.

[0094] In some specific embodiments, a PNP algorithm is used to calculate an initial lateral pose of the C-arm X-ray machine, and the first transition pose is multiplied by the current initial lateral pose of the C-arm X-ray machine to obtain a target lateral pose of the C-arm X-ray machine after registration. In some specific embodiments, a relative relationship exists between the first virtual object pose and the initial lateral pose. Based on the relative relationship, a lateral pose of the C-arm X-ray machine is calculated when the object pose remains at the target object pose, and this lateral pose is determined as the target lateral pose of the C-arm X-ray machine.

[0095] In FIG5 , the registration result integration module integrates the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine, and the initial anteroposterior posture of the C-arm X-ray machine into the registration optimization result.

[0096] In some specific embodiments, the target object posture of the object to be registered is the conversion matrix of coordinates from the CT image coordinate system to the patient coordinate system after registration optimization. The target lateral posture and initial anteroposterior posture of the C-arm X-ray machine are the conversion matrices of coordinates from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine after registration optimization. In some specific embodiments, the registration result integration module integrates the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine, and the initial anteroposterior posture of the C-arm X-ray machine to generate a registration optimization result, which is provided to a subsequent operator for operation.

[0097] Figure 6 is a schematic diagram of the orthopedic surgery navigation system based on image registration of the present application. As shown in Figure 6, the orthopedic surgery navigation system based on image registration includes a 3D CT device, an X-ray machine, a processing module and an operating robot.

[0098] In some specific embodiments, in FIG6 , the X-ray machine is a C-arm X-ray machine. The system shown in FIG6 will be described in detail below using the C-arm X-ray machine as an example.

[0099] As shown in FIG6 , in some specific embodiments, a three-dimensional CT device is used to take a three-dimensional image of the part to be operated on before surgery; a C-arm X-ray machine is used to take an anteroposterior two-dimensional image and a lateral two-dimensional image of the part to be operated on during surgery; a processing module is used to determine a surgical operation plan based on the three-dimensional image; in some specific embodiments, the processing module is further used to generate a registration optimization result based on the three-dimensional image, the anteroposterior two-dimensional image, the lateral two-dimensional image and the internal parameters of the C-arm X-ray machine using the method shown in FIG1 , and the registration optimization result includes the target object posture of the object to be registered, the C-arm X-ray machine The processing module is further used to convert the coordinates of the surgical operation plan from the CT image coordinate system to the patient coordinate system according to the target object posture; convert the coordinates of the surgical operation plan from the patient coordinate system to the X-ray machine three-dimensional coordinate system according to the target lateral posture and the initial frontal posture; convert the coordinates of the surgical operation plan from the C-arm three-dimensional coordinate system to the frontal X-ray two-dimensional coordinate system and the lateral X-ray two-dimensional coordinate system according to the internal reference of the C-arm X-ray machine; the operating robot is used to operate the operation part according to the surgical operation plan.

[0100] In some specific embodiments, the part to be operated on is the spine, and the process of performing orthopedic surgery navigation using the system shown in FIG6 is as follows:

[0101] Before surgery, a 3D CT scanner captures a 3D image of the spinal column to be operated on. A processing module determines a surgical plan based on the 3D image. Optionally, the surgical plan includes the placement of pedicle screws for each individual vertebra in the spinal column to be operated on.

[0102] During the operation, the C-arm X-ray machine takes two-dimensional anteroposterior and lateral images of the spine to be operated. The processing module generates the target object posture of the vertebra to be operated and the target lateral posture of the C-arm according to the method shown in Figure 1. The processing module converts the surgical operation plan from the CT coordinate system to the patient coordinate system through the target object posture. The processing module then uses the internal and external references of the C-arm X-ray machine to project the surgical operation plan in the patient coordinate system onto the X-ray display. The operating robot operates on the spine to be operated according to the surgical operation plan displayed on the X-ray. Optionally, the operating robot fixes the pedicle screws at the position of the surgical operation plan.

[0103] FIG7 is a block diagram of an electronic device provided by the present application, including a processor and a memory. The memory stores computer instructions, which, when executed by the processor, cause the processor to execute the computer instructions to implement the method and detailed solution shown in FIG1.

[0104] It should be understood that the above-described device embodiments are merely illustrative, and the devices disclosed herein may also be implemented in other ways. For example, the division of units / modules described in the above-described embodiments is merely a logical functional division, and actual implementations may employ alternative divisions. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0105] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present invention may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0106] If the integrated unit / module is implemented in hardware, the hardware may be a digital circuit, an analog circuit, or the like. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, and the like. Unless otherwise specified, the processor or chip may be any appropriate hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC. Unless otherwise specified, the on-chip cache, off-chip memory, and storage may be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), and the like.

[0107] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned memory includes: various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0108] An embodiment of the present application further provides a non-transitory computer storage medium storing a computer program. When the computer program is executed by multiple processors, the processors execute the method and detailed solution shown in FIG. 1 .

[0109] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. At the same time, changes or modifications made by those skilled in the art based on the ideas of the present application, the specific implementation methods, and the scope of application of the present application, all fall within the scope of protection of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. An image registration method, characterized in that: include: Generate an initial object pose of the object to be registered and an initial anteroposterior pose and an initial lateral pose of the X-ray machine according to the three-dimensional image, the two-dimensional anteroposterior image, the two-dimensional lateral image of the object to be registered and the internal parameters of the X-ray machine; Performing registration optimization on the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate a target object posture of the object to be registered; Performing registration optimization on the initial lateral posture of the X-ray machine according to the target object posture and the lateral two-dimensional image to generate a target lateral posture of the X-ray machine after registration; The target object posture of the object to be registered, the target lateral posture of the X-ray machine and the initial frontal posture of the X-ray machine are integrated into a registration optimization result.

2. The method according to claim 1, characterized in that The step of performing registration optimization on the initial lateral posture of the X-ray machine according to the target object posture and the lateral two-dimensional image to generate the registered target lateral posture of the X-ray machine includes: Generate a first virtual object posture according to the target object posture and the lateral two-dimensional image; The target lateral posture of the registered X-ray machine is generated according to the relative relationship between the first virtual object posture and the lateral two-dimensional image.

3. The method according to claim 2, characterized in that The step of generating the target lateral position of the registered X-ray machine according to the relative relationship between the first virtual object position and the lateral two-dimensional image comprises: generating a first transition posture according to the first virtual object posture; The target lateral posture of the X-ray machine after the alignment is obtained according to the first transitional posture and the initial lateral posture of the current X-ray machine.

4. The method according to claim 2, characterized in that Generating a first virtual object posture according to the target object posture and the lateral two-dimensional image comprises: S201, taking the target object posture as the current object posture; S202, generating a first virtual lateral two-dimensional image corresponding to the current object posture; S203, calculating a first similarity between the lateral two-dimensional image and the first virtual lateral two-dimensional image; S204, when the first similarity satisfies a first object posture condition, taking the current object posture as the first virtual object posture; S205: When the first similarity does not satisfy the first object posture condition, adjust the current object posture and return to step S202.

5. The method according to claim 1, characterized in that The registering and optimizing the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate the target object posture of the object to be registered comprises: Generate a first temporary object pose according to the initial object pose, the frontal two-dimensional image and the lateral two-dimensional image; The target object pose is generated according to the first temporary object pose and the orthogonal two-dimensional image.

6. The method according to claim 5, characterized in that Generating a first temporary object posture according to the initial object posture, the frontal two-dimensional image and the lateral two-dimensional image comprises: S301, taking the initial object posture as the current object posture; S302, generating a second virtual frontal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the current object posture; S303, calculating a second similarity between the anteroposterior two-dimensional image and the second virtual anteroposterior two-dimensional image, and a third similarity between the lateral two-dimensional image and the second virtual lateral two-dimensional image; S304: When the second similarity and the third similarity satisfy a second object posture condition, use the current object posture as the first temporary object posture; S305: When the second similarity and the third similarity do not satisfy the second object posture condition, adjust the current object posture and return to step S302.

7. The method according to claim 5, characterized in that Generating the target object posture according to the first temporary object posture and the orthogonal two-dimensional image comprises: S401, using the first temporary object posture as the current object posture; S402, generating a third virtual orthotopic two-dimensional image corresponding to the current object posture; S403, calculating a fourth similarity between the orthostatic two-dimensional image and the third virtual orthostatic two-dimensional image; S404, when the fourth similarity satisfies the third object posture condition, taking the current object posture as the target object posture; S405, when the fourth similarity does not satisfy the third object posture condition, adjust the current object posture and return to step S402.

8. The method according to any one of claims 4, 6 and 7, characterized in that: The adjusting the object posture comprises: The position and posture of the object to be registered is rotated or translated.

9. An image registration device, characterized in that: It includes an initial pose generation module, a target object pose generation module, a target lateral pose generation module and a registration result integration module, among which: The initial posture generation module is used to generate the initial object posture of the object to be registered and the initial anteroposterior posture and initial lateral posture of the X-ray machine according to the three-dimensional image, the two-dimensional image of the anteroposterior position, the two-dimensional image of the lateral position and the internal parameters of the X-ray machine of the object to be registered; The target object posture generation module is used to perform registration optimization on the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate the target object posture of the object to be registered; The target lateral posture generation module is used to perform registration optimization on the initial lateral posture of the X-ray machine according to the target object posture and the lateral two-dimensional image, and generate the target lateral posture of the X-ray machine after registration; The registration result integration module is used to integrate the target object posture of the object to be registered, the target lateral posture of the X-ray machine and the initial frontal posture of the X-ray machine into a registration optimization result.

10. An orthopedic surgery navigation system based on image registration, characterized in that: include: 3D CT equipment, used to take 3D images of the part to be operated on; An X-ray machine, used to take two-dimensional anteroposterior and lateral images of the part to be operated on, and provide the internal reference of the X-ray machine to the processing module; A processing module, used for determining a surgical operation plan according to the three-dimensional image; Using the method according to any one of claims 1 to 8, a registration optimization result is generated according to the three-dimensional image, the frontal two-dimensional image, the lateral two-dimensional image and the internal parameters of the X-ray machine, wherein the registration optimization result includes the target object posture of the object to be registered, the X-ray machine a target lateral position and an initial anteroposterior position of the X-ray machine; The coordinates of the surgical operation plan are converted from the CT image coordinate system to the patient coordinate system according to the target object posture; the coordinates of the surgical operation plan are converted from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine according to the target lateral posture and the initial anteroposterior posture; the coordinates of the surgical operation plan are converted from the three-dimensional coordinate system of the X-ray machine to the two-dimensional coordinate system of the anteroposterior X-ray and the lateral X-ray according to the internal reference of the X-ray machine; An operating robot is used to operate the part to be operated according to the surgical operation plan.

11. An electronic device, comprising: processor; A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

Images