A two-dimensional three-dimensional image registration method, device, medium and program product

Abstract

The present invention provides a two-dimensional and three-dimensional image registration method, system, device, medium and program product, which relate to the field of intelligent medical care. The method includes: obtaining intraoperative anteroposterior and lateral two-dimensional images of the subject with a registration ball; selecting feature points in the two-dimensional image and obtaining the two-dimensional coordinates of the feature points in the two-dimensional image; obtaining the three-dimensional coordinates of the feature points in the world coordinate system based on the two-dimensional coordinates; obtaining the three-dimensional coordinates of the feature points in the preoperative three-dimensional model; pairing the feature points in the three-dimensional model with the feature points in the world coordinate system to obtain the rough position relationship of the three-dimensional model under the C-arm machine; based on the rough position relationship, automatically extracting the feature points in the two-dimensional image and the preoperative three-dimensional model for iterative optimization, calculating the precise position of the preoperative three-dimensional model under the C-arm machine, and completing the registration. The present application simplifies the operation and reduces the computing resources by first roughly registering to give an initial posture with a small error, and then performing precise registration based on the initial posture as the initial iteration value.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical treatment, and more specifically, to a two-dimensional and three-dimensional image registration method, device, medium and program product. Background Art

[0002] The registration of two-dimensional (2D) images and three-dimensional (3D) models refers to the process of aligning and matching a two-dimensional image containing a target object with the three-dimensional model of the target object. This process is widely used in fields such as medical imaging, computer vision, and computer graphics.

[0003] In traditional spinal surgery, a three-dimensional model that can reflect the actual condition of the patient's affected area is constructed based on the image of the patient's affected area before the operation. The three-dimensional model is used for subsequent surgical planning and intraoperative guidance. However, the image of the affected area presented during the operation is still a two-dimensional image, which is difficult to intuitively reflect the three-dimensional structure of the patient's affected area. Therefore, it is necessary to align the intraoperative two-dimensional image with the preoperative three-dimensional model to facilitate intraoperative guidance.

[0004] When performing the registration operation of two-dimensional images and three-dimensional models, it is necessary to manually mark the characteristic lines of the target object in the two-dimensional image and the three-dimensional model, and then achieve registration by matching the two-dimensional characteristic lines and the three-dimensional characteristic lines. However, some characteristic lines in the three-dimensional model are difficult to draw manually, which makes it impossible for the two-dimensional image and the three-dimensional model to achieve registration. Based on this, the patent application 202311190386.2 of the registration method and device of two-dimensional image and three-dimensional model proposes a method of automatically extracting the three-dimensional characteristic lines on the three-dimensional model of the target object and using it for subsequent registration without manually drawing characteristic lines. Compared with the manual drawing method, it can significantly improve the accuracy of three-dimensional feature line extraction, thereby achieving accurate registration of two-dimensional images and three-dimensional models. However, this method still needs to match through characteristic lines. For irregular bones, especially for various types of bones that have already undergone lesions, repeated projection and calculation of similarity measures will lead to excessive computing resources and make it inconvenient for the network model to perform iterative optimization of parameters; in addition, the iterative optimization method adopted by the current registration algorithm is prone to fall into the local optimal solution, resulting in errors in the results. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a two-dimensional three-dimensional image registration method, device, medium and program product; the method of the present invention simplifies the operation and reduces the computing resources by first coarsely registering to give an initial posture with a small error, and then performing precise registration based on the initial posture as an initial iteration value.

[0006] The first aspect of the present application discloses a two-dimensional three-dimensional image registration method, the method comprising:

[0007] S1: Obtain the intraoperative anteroposterior and lateral two-dimensional images of the subject with the registration ball. The two-dimensional images are taken with a C-arm machine;

[0008] S2: Selecting feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image, obtaining the two-dimensional coordinates of the feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image; and obtaining the three-dimensional coordinates of the feature points in the world coordinate system based on the two-dimensional coordinates;

[0009] S3: obtaining the three-dimensional coordinates of the feature points in the preoperative three-dimensional model;

[0010] S4: Pair the coordinates of the feature points in the preoperative three-dimensional model with the feature points in the world coordinate system to obtain the rough position relationship M of the preoperative three-dimensional model under the C-arm machine;

[0011] S5: Based on the rough position relationship M, N feature points in the two-dimensional image and the preoperative three-dimensional model are automatically extracted for iterative optimization, the precise position of the preoperative three-dimensional model under the C-arm is calculated, and the two-dimensional and three-dimensional image registration is completed.

[0012] In some embodiments, the method for obtaining the three-dimensional coordinates of the feature point in the world coordinate system based on the two-dimensional coordinates in S2 includes: based on the two-dimensional coordinates of any of the feature points in the anteroposterior and lateral two-dimensional images, obtaining the anteroposterior ray corresponding to any feature point in the anteroposterior two-dimensional image in space and the lateral ray in the lateral two-dimensional image corresponding to the corresponding feature point in the anteroposterior two-dimensional image, and the intersection point of the anteroposterior ray and the lateral ray in space through the C-arm projection parameters corresponding to the anteroposterior and lateral two-dimensional images, and the coordinates of the intersection point are the coordinates of any feature point in the world coordinate system, denoted as p i (x i ,y i ,z i );

[0013] Optionally, for any feature point, it identifies the same position of the subject in the frontal two-dimensional image and the lateral two-dimensional image;

[0014] Optionally, the feature points are feature points that can be used to identify key parts of the subject.

[0015] In some embodiments, the method for pairing the coordinates of feature points in the preoperative three-dimensional model with those in the world coordinate system in S4 includes: receiving operation instructions for the world coordinate system or the preoperative three-dimensional model, and pairing the three-dimensional coordinates of feature points in the preoperative three-dimensional model with those in the world coordinate system using the least squares method to obtain the rough position relationship M.

[0016] In some embodiments, the number of the feature points is a natural number greater than or equal to 3; preferably greater than or equal to 4; when the number of the feature points is 4, the 4 feature points are arranged asymmetrically with respect to each other.

[0017] In some embodiments, the method of automatically extracting N feature points from the two-dimensional image and the preoperative three-dimensional model for iterative optimization in S5 includes:

[0018] S501: Calculate a projection operator based on the rough position relationship and the preoperative three-dimensional model data, and generate a DRR image; the DRR is created by the projection operator (camera parameters and projection integral) P;

[0019] S502: Calculating the similarity between the DRR image and the original X-ray image used to construct the preoperative three-dimensional model;

[0020] S503: Determine whether the similarity meets a preset threshold; when the similarity meets the preset standard, output the precise position of the preoperative three-dimensional model under the C-arm machine;

[0021] Optionally, the similarity calculation formula includes: argmin S(I x ,P(I CT ;θ));

[0022] Among them, I CT represents the preoperative 3D model data constructed based on the preoperative CT image of the subject; θ is the pose information of the preoperative 3D model, and its initial value is the rough position relationship M(R, t matrix) of the preoperative 3D model under the C-arm machine; P is the projection operator, which is calculated based on the CT image and the position information under the C-arm machine when the CT image is taken, and is obtained through I CT and I CT The world pose information (θ) generates a virtual X-ray, namely, a DRR image; Ix represents the intraoperative X-ray image; S represents the similarity between Ix and the DRR image obtained after calculation by the projection operator.

[0023] Optionally, after S502 and before S503, the method further includes: determining whether the number of times the similarity is calculated and output exceeds the number of cycles, and if it exceeds the number of cycles, adjusting the rough position relationship and continuing to repeat steps S501-S503, outputting the similarity, and finally obtaining the spatial position of the preoperative three-dimensional model on the C-arm machine;

[0024] Optionally, the rough position relationship is adjusted by CMA-ES.

[0025] In some embodiments, after S1 and before S2, the method further includes: calculating perspective projection parameters of the C-arm machine based on the projection coordinates of the registration ball on the two-dimensional image and a calibration algorithm;

[0026] Optionally, the method for calculating the perspective projection parameters of the C-arm machine includes:

[0027] Calculate the C-arm internal parameters based on the three-dimensional coordinates of the registration ball in the VSPOTS reference system and its corresponding two-dimensional coordinates in the two-dimensional image;

[0028] Calculate C-arm machine external parameters based on the C-arm machine internal parameters, where the C-arm machine external parameters include a rotation matrix R and a translation vector T;

[0029] Optionally, the calculation method of the C-arm machine internal parameter includes: through the three-dimensional coordinates of the registration ball in the VSPOTS reference system and the two-dimensional coordinates corresponding to the two-dimensional image; the registration ball is A balls, and when every two or three balls are connected, they form B lines, and the extension lines of the B lines intersect at a point on different planes, and every two extension lines form a plane, based on the plane and the points on the plane, the internal parameter is obtained according to the Zhang Zhengyou calibration method; wherein A and B are natural numbers greater than 4, and A is greater than B; A is preferably 12;

[0030] Optionally, the calculation method of the C-arm external parameters includes: obtaining the relative posture of the C-arm and the VSPOTS reference system based on the C-arm internal parameters, the three-dimensional coordinates of the registration ball in the VSPOTS reference system, and the two-dimensional coordinates of the registration ball in the two-dimensional image; obtaining the relative posture of VSPOTS and the reference frame based on the relative posture of VSPOTS and the camera, and the relative posture of the reference frame fixed on the subject and the camera; obtaining the relative posture of the C-arm and the reference frame based on the relative posture of VSPOTS and the reference frame, and the relative posture of the C-arm and the VSPOTS reference system, that is, the C-arm external parameters;

[0031] Optionally, calculating the internal parameters of the C-arm, the three-dimensional coordinates of the registration ball in the VSPOTS reference system, and the two-dimensional coordinates of the registration ball in the two-dimensional image;

[0032] Optionally, the relative position of the VSPOTS under the camera and the relative position of the reference frame on the subject and the camera are recorded based on the optical tracking ball of the VSPOTS;

[0033] Optionally, the camera is a binocular tracker;

[0034] Optionally, the intraoperative anteroposterior and lateral two-dimensional images are images of a VSPOTS tool with a registration ball.

[0035] In some embodiments, the preoperative three-dimensional model is constructed based on preoperative CT images taken of the subject; optionally, the method for constructing the preoperative three-dimensional model includes:

[0036] S301, obtaining a three-dimensional CT image of the subject's spine before surgery, and segmenting the three-dimensional CT image to obtain a W value mask image containing a ROI spine region;

[0037] S302, cropping the single vertebra in the W value mask image containing the ROI spine region to obtain a W value mask image in which adjacent single vertebra are not connected;

[0038] S303, locating the centroid of a single vertebra in the W-value mask image that is not connected between adjacent single vertebrae, and obtaining an image of the located centroid;

[0039] S304, taking the single centroid as the center, segmenting from the image of the located centroid to obtain a single vertebra image corresponding to the single centroid;

[0040] Optionally, W is a natural number greater than or equal to 2;

[0041] Optionally, the ROI spinal region includes any one or more of the following regions: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx.

[0042] The second aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store a computer program; and the processor executes the computer program to implement the steps of the above method.

[0043] A third aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0044] A fourth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0045] This application has the following beneficial effects:

[0046] 1. This application innovatively discloses a two-dimensional and three-dimensional image registration method, which adopts a two-dimensional to three-dimensional registration method to accurately calculate the spatial position of the model in the C-arm machine, and can accurately obtain the precise position of the preoperative three-dimensional model under the C-arm machine, thereby guiding the subsequent intraoperative navigation operation. Specifically, the technical solution of this application simplifies the operation (no need to extract three-dimensional feature lines, and no need to manually draw feature lines) by first coarsely registering to give an initial position Rt (rough position relationship M) with a small error, and then based on the initial position as the initial iteration value, it performs precise registration, reduces computing resources (coarse registration is simple to operate, simple to calculate, and has poor accuracy, while precise registration is complex to calculate, has high accuracy, and depends on the initial position), and has a wide range of application scenarios (not only suitable for complex spines, but also for relatively simple structures such as long bones; the adapted procedures include but are not limited to the following procedures: vertebral fusion, vertebroplasty (PKP), spinal decompression (UBE), and long bone fracture surgery).

[0047] The method of obtaining the initial pose by rough registration first and then using the initial pose for precise registration can effectively overcome the problem that the current iterative optimization method is easy to fall into the local optimal solution and thus cause erroneous results.

[0048] 2. This application innovatively discloses the similarity calculation formula applicable to the two-step registration method of coarse registration and precise registration, optimizes the iterative optimization method, and outputs more accurate results.

[0049] 3. In the registration process of the two-dimensional images and the three-dimensional images in this application, the frontal and lateral two-dimensional images and the three-dimensional model are interconnected, and the registration is more accurate.

[0050] 4. The present application innovatively discloses a method for constructing a preoperative three-dimensional model, which achieves a good segmentation of spinal CT images; it first preliminarily locates the spinal area in the three-dimensional image, reduces the area, and reduces the influence of areas outside the spinal area on subsequent segmentation, and corrodes a single vertebra in the N-value mask image of the entire spine so that there is no connection between adjacent vertebrae, and then uses a connected domain algorithm to determine the different areas where the single vertebrae are located after corrosion, locates the center of mass of the single vertebrae in different areas as the center of mass of the entire vertebra, and uses a Gaussian distribution centered on the center of mass to restrict the network to only segment the vertebrae at the center of mass, thereby achieving fine segmentation of the spine, and further obtaining the entire spine; the above method performs bone segmentation through a three-step process of coarse positioning, center of mass positioning, and fine segmentation, while ensuring segmentation accuracy and reducing segmentation difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a schematic diagram of a method flow chart provided by the first aspect of an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a two-dimensional and three-dimensional image registration system provided by a second aspect of an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of a storage medium provided by an embodiment of the present invention;

[0057] Figure 6 is a schematic diagram of a projection matrix P used in the process of inversely deriving coordinates from image plane coordinates to three-dimensional space coordinates provided by an embodiment of the present invention;

[0058] Figure 7 The system of equations for solving the projection matrix P by using the image coordinates of the four points and the corresponding world coordinates when four-point registration is used in the coarse registration provided by the embodiment of the present invention;

[0059] Figure 8 It is the calculation formula of ray projection and attenuation in the precise registration provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0061] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0063] Figure 1 1 is a flow chart of a two-dimensional and three-dimensional image registration method provided by an embodiment of the present invention. Specifically, the method comprises the following steps:

[0064] S1: Obtain the intraoperative anteroposterior and lateral two-dimensional images of the subject with the registration ball. The two-dimensional images are taken with a C-arm machine;

[0065] In some embodiments, the term "subject" or "test subject" or "test sample" used herein refers to any animal (e.g., mammal), including but not limited to humans, non-human primates, rodents, etc., which will be the recipient of a specific treatment. Generally, the terms "subject" and "patient" are used interchangeably herein when referring to human subjects.

[0066] Preferably, the subject is a human. The human spine has a complex structure, which includes vertebrae and muscles. The spine is composed of arranged vertebrae, and each vertebra includes a vertebral body, pedicle, lamina, vertebral foramen, transverse process, spinous process, accessory process, superior articular process, and mastoid process.

[0067] In some embodiments, the two-dimensional image is a DRR image (digital X-ray imaging), which refers to a technology that uses a one-dimensional or two-dimensional C-line detector to convert X-ray image information into a digital signal under computer control.

[0068] In some embodiments, after S1 and before S2, the method further includes: calculating the perspective projection parameters of the C-arm based on the projection coordinates of the registration ball on the two-dimensional image and a calibration algorithm; the calibration algorithm is the Zhang Zhengyou calibration method, and the projection parameters of the C-arm can be obtained through this calculation method (calibration algorithm); the perspective projection parameters of the C-arm are the internal and external parameters of the C-arm, and the projection form of the C-arm can be known through the internal and external parameters; the purpose of calculation: the mathematical representation of the projection of the C-arm when shooting the two-dimensional image, and respectively obtaining the mathematical representation of the projection of the C-arm when the frontal and lateral DR images are shot. For any point P in the space, if the mathematical representation is known, the position of the point in the image can be obtained.

[0069] In some embodiments, the method for calculating the perspective projection parameters of the C-arm machine includes:

[0070] Based on the three-dimensional coordinates of the registration ball in the VSPOTS reference system and its corresponding two-dimensional coordinates in the two-dimensional image, the C-arm machine internal parameters (K, internal parameter matrix, including the focal length and principal point coordinates of the C-arm machine) are calculated;

[0071] The C-arm machine external parameters are calculated based on the C-arm machine internal parameters, and the C-arm machine external parameters include the rotation matrix R and the translation vector T (rotation matrix R (describing the rotation from the world coordinate system to the camera coordinate system), the translation vector T (describing the translation from the world coordinate system to the camera coordinate system), and the world position of the perspective machine); wherein, the internal parameters

[0072] (Intrinsic Parameters): Intrinsic parameters are parameters that describe the internal properties of the camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc. Intrinsic parameters are usually determined during camera calibration, because they are usually fixed for a specific camera model and do not change over time. Once the camera intrinsic parameters are determined, they usually remain unchanged during the use of the camera. Extrinsic Parameters: Extrinsic parameters are parameters that describe the position and posture of the camera in the world coordinate system, usually including rotation matrices and translation vectors. Extrinsic parameters may change at different camera positions or shooting times. For example, in stereo vision, if you have two cameras, their relative positions and orientations will change every time you move the camera, resulting in changes in extrinsic parameters.

[0073] Optionally, the calculation method of the internal parameters of the C-arm machine includes: using the three-dimensional coordinates of the alignment balls in the VSPOTS reference system and the two-dimensional coordinates corresponding to the two-dimensional image; the alignment balls are A balls, and when every two or three balls are connected, they form B lines, and the extension lines of the B lines intersect at a point on different planes, and every two extension lines form a plane, and based on the plane and the points on the plane, the internal parameters are obtained according to the Zhang Zhengyou calibration method; wherein A and B are natural numbers greater than 4, and A is greater than B; A is preferably 12; due to the special design of the 12 alignment balls, 4 straight lines formed by 3 points can be formed, and then according to these 4 straight lines, every two form a plane, and a total of 6 planes can be obtained, each with 6 points, and the internal parameters are obtained according to the Zhang Zhengyou calibration method.

[0074] Optionally, the calculation method of the C-arm external parameters includes: obtaining the relative posture of the C-arm and the VSPOTS reference system based on the C-arm internal parameters, the three-dimensional coordinates of the registration ball in the VSPOTS reference system, and the two-dimensional coordinates of the registration ball in the two-dimensional image; obtaining the relative posture of VSPOTS and the reference frame based on the relative posture of VSPOTS and the camera, and the relative posture of the reference frame fixed on the subject and the camera; obtaining the relative posture of the C-arm and the reference frame based on the relative posture of VSPOTS and the reference frame, and the relative posture of the C-arm and the VSPOTS reference system, that is, the C-arm external parameters;

[0075] Optionally, the C-arm internal parameters, the three-dimensional coordinates of the registration ball in the VSPOTS reference system, and the two-dimensional coordinates of the registration ball in the two-dimensional image are calculated by a pnp algorithm but not limited to the pnp algorithm; in addition to the pnp algorithm, any of the following algorithms is also included: a deep learning method;

[0076] Optionally, the relative position of the VSPOTS under the camera and the relative position of the reference frame on the subject and the camera are recorded based on the optical tracking ball of the VSPOTS;

[0077] Optionally, the camera is a binocular tracker;

[0078] Optionally, the intraoperative anteroposterior and lateral two-dimensional images are images of a VSPOTS (image registration system) tool with a registration ball.

[0079] In some embodiments, the principle of setting up VSPOTS with registration balls is as follows: the patient's fracture site, the implanted instrument, and the registration device are connected with a reference price, and an optical tracking ball is installed on the reference frame. The optical tracking ball is captured by the infrared light positioning system, and then the spatial position of the patient's fracture site, the implanted instrument, and the mounting device in the three-dimensional spatial positioning coordinate system is obtained. The registration device is fixed on an image shooting device for shooting the anteroposterior and lateral images of the patient's fracture site. The registration device has a registration ball. When shooting the anteroposterior and lateral images of the patient's fracture site, a two-dimensional image containing the registration ball is obtained; in one embodiment, the registration device has a registration ball and an optical tracking ball, the optical tracking ball has a spatial coordinate in the spatial positioning coordinate system, the registration ball has an image coordinate in the image coordinate system, the optical tracking ball and the registration ball have a fixed spatial position relationship, and a mapping relationship between the spatial positioning coordinate system and the image coordinate system is constructed based on the spatial coordinates, the image coordinates, and the spatial position relationship, and the real-time spatial coordinates of the implanted instrument are mapped to the image coordinate system through the mapping relationship. The registration device is named v-spots. V-spots is equipped with 4 optical tracking balls and 12 registration balls. The 12 registration balls have corresponding two-dimensional image coordinates in the anteroposterior and lateral images taken by the C-arm. There are n registration balls. When two or three balls are connected, m lines are formed. The extension lines of the M lines meet at one point on different planes. n and m are natural numbers greater than 4, and n is greater than m.

[0080] S2: Selecting feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image, obtaining the two-dimensional coordinates of the feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image; and obtaining the three-dimensional coordinates of the feature points in the world coordinate system based on the two-dimensional coordinates;

[0081] In some embodiments, the method for obtaining the three-dimensional coordinates of the feature point in the world coordinate system based on the two-dimensional coordinates in S2 includes: based on the two-dimensional coordinates of any of the feature points in the anteroposterior and lateral two-dimensional images, obtaining the anteroposterior ray corresponding to any feature point in the anteroposterior two-dimensional image in space and the lateral ray in the lateral two-dimensional image corresponding to the corresponding feature point in the anteroposterior two-dimensional image, and the intersection point of the anteroposterior ray and the lateral ray in space through the C-arm projection parameters corresponding to the anteroposterior and lateral two-dimensional images, and the coordinates of the intersection point are the coordinates of any feature point in the world coordinate system, denoted as p i (x i ,y i ,z i) ; space is the space where the world coordinate system is located; the C-arm projection parameters include projection internal parameters and projection external parameters. The projection internal parameters represent the physical structure of the C-arm (the distance between the receiving plate and the X-ray source, as well as the intersection of the optical axis and the receiving plate, etc.), and the projection external parameters represent the position of the C-arm in the world coordinate system; that is, in the obtained projection parameters, the internal parameters are the same but the external parameters are different.

[0082] Optionally, for any feature point, it identifies the same position of the subject in the frontal two-dimensional image and the lateral two-dimensional image;

[0083] Optionally, the feature points are feature points that can be used to identify key parts of the subject. In some embodiments, in long bone reduction surgery, the four feature points include: the center point of the femoral head, the greater trochanter point, the medial point of the femoral condyle, and the lateral point. In lumbar spine surgery, the four feature points include: the center of the vertebral body, the left transverse process, the right transverse process, and the spinous process.

[0084] In some embodiments, the number of feature points is a natural number greater than or equal to 3; preferably greater than or equal to 4; when the number of feature points is 4, the 4 feature points are not arranged symmetrically; when the number of feature points is 3, positive and negative direction problems are prone to occur; when the number of feature points is 4 (the minimum cost that can achieve the desired effect), the 4 feature points are not arranged symmetrically, and 5 is better; take 1 feature point as an example: the 2D coordinates of 1 feature point in the frontal 2D image are ai-1 (xi-1, yi-1), and the 2D coordinates in the lateral 2D image are ai-2 (xi-2, yi-2); the 3D coordinates ai-3 (xi-3, yi-3, zi-3) of the feature point in the world coordinate system are calculated based on ai-1 (xi-1, yi-1) and ai-2 (xi-2, yi-2); the 3D coordinates Ai (Xi, Yi, Zi) of the corresponding feature point in the preoperative 3D model are paired with ai-3 (xi-3, yi-3, zi-3) to obtain the rough position relationship M.

[0085] S3: obtaining the three-dimensional coordinates of the feature points in the preoperative three-dimensional model;

[0086] In some embodiments, the three-dimensional coordinates of the feature points in the three-dimensional model are marked as P i (X i ,Y i ,Z i ).

[0087] In some embodiments, the method for constructing the preoperative three-dimensional model includes:

[0088] S301, obtaining a three-dimensional CT image of the subject's spine before surgery, and segmenting the three-dimensional CT image to obtain a W value mask image containing a ROI spine region;

[0089] S302, cropping the single vertebra in the W value mask image containing the ROI spine region to obtain a W value mask image in which adjacent single vertebra are not connected;

[0090] S303, locating the centroid of a single vertebra in the W-value mask image that is not connected between adjacent single vertebrae, and obtaining an image of the located centroid;

[0091] S304, taking the single centroid as the center, segmenting from the image of the located centroid to obtain a single vertebra image corresponding to the single centroid, which is the preoperative three-dimensional model;

[0092] Optionally, the ROI spinal region includes any one or more of the following regions: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx.

[0093] Optionally, W is a natural number greater than or equal to 2; when it is a binary mask image, the ROI area is any one of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx; when it is a ternary mask image, the ROI area is any two of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx; when it is a four-value mask image, the ROI area is any three of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx; when it is a five-value mask image, the ROI area is any four of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx; when it is a six-value mask image, the ROI area includes the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx.

[0094] In some embodiments, the method of cropping a single vertebra in the W value mask image containing the ROI spine region in S302 includes: cropping gradually from the edges of the single vertebra to the center; preferably, eroding from the edges of the single vertebra to the center in equal proportion; optionally, the volume of the single vertebra after cropping is smaller than the volume of the single vertebra before cropping; preferably, the ratio of the volume of the single vertebra after cropping to the volume of the single vertebra before cropping is: 1 / 10-9 / 10, preferably 1 / 5-4 / 5, preferably 1 / 2;

[0095] In some embodiments, the method of locating the centroid of a single vertebra in the W-value mask image that is not connected between adjacent single vertebra in S303 includes: using a connected domain algorithm to identify at least one separate region in the W-value mask image that is not connected between adjacent single vertebra; locating the centroid of the separate region as the centroid of the single vertebra; the separate region is the region where the single vertebra is located;

[0096] Optionally, when locating the centroid of a single vertebra in S303, the vertebrae are sorted according to the z-axis coordinate of the centroid.

[0097] In some embodiments, the method for acquiring the single vertebral image in S304 includes:

[0098] Obtain a 3D Gaussian distribution with a1×b1×c1 fixed around the single centroid; the value of a1 / b1 / c1 needs to meet the following conditions: since a1 / b1 / c1 is a hyperparameter, it needs to be a multiple of 16; and it needs to be used to crop a single spine, so the cropping area must include the complete spine. The larger the area, the greater the interference. Therefore, it should be as small as possible while ensuring that the entire vertebra is covered, preferably 224×224×128, but its specific value is variable;

[0099] Get a 3D image with a2×b2×c2 fixed at the center of mass. The values ​​of a2 / b2 / c2 need to meet the following conditions: Since a2 / b2 / c2 is a hyperparameter, it needs to be a multiple of 16. If the cropping is too large, the upper and lower vertebrae may be placed in this area. The preferred value is 224×224×128, but its specific value is variable.

[0100] According to the 3D Gaussian distribution centered at the centroid, a single vertebral image corresponding to the centroid around the Gaussian distribution is segmented from the 3D image.

[0101] In some embodiments, after S301 and before S302, the method further includes: cropping the W value mask image of the spine to obtain a W value mask image containing the ROI spine region;

[0102] Optionally, the method for cropping the W value mask image of the spine includes: when the W value mask image of the spine is consistent with the direction of the spine when the human body is standing, locating any one or more of the highest point, the lowest point, the leftmost point, and the rightmost point of the ROI spine area; making one or two straight lines parallel to the horizontal line through the highest point and / or the lowest point, and making one or two straight lines perpendicular to the horizontal line through the leftmost point and / or the rightmost point; and cropping the W value mask image of the spine along at least one straight line to obtain a W value mask image showing the ROI spine area.

[0103] In some embodiments, after S304, the method further comprises S305, identifying the category of a single vertebra in the vertebral image based on the position of the centroid;

[0104] Optionally, when the category of a single vertebra cannot be determined according to the method in S305, the method further comprises: inputting the image of the single vertebra into a classifier to identify the category of the single vertebra;

[0105] Optionally, the training method of the classifier includes: inputting a single vertebral image of a training set sample into the classifier, obtaining a predicted classification result, comparing it with the classification label, and optimizing it according to the comparison result to obtain a constructed classifier;

[0106] Optionally, the classifier includes any one or more of the following: decision tree, logistic regression, naive Bayes, neural network; preferably DenseNet in the neural network.

[0107] In some embodiments, after S305, the method further comprises S306, synthesizing at least two single vertebrae based on the categories of the single vertebrae to obtain a synthesized image containing the ROI spinal region, which is a preoperative three-dimensional model after synthesizing the single vertebrae.

[0108] S4: Pair the coordinates of the feature points in the preoperative three-dimensional model with the feature points in the world coordinate system to obtain a rough position relationship M of the preoperative three-dimensional model under the C-arm machine.

[0109] In some embodiments, the method of pairing the coordinates of the feature points in the preoperative three-dimensional model with the feature points in the world coordinate system in S4 includes: receiving an operation instruction for the world coordinate system or the preoperative three-dimensional model, pairing the three-dimensional coordinates of the feature points in the preoperative three-dimensional model with the feature points in the world coordinate system using the least squares method, and obtaining the rough position relationship M; M represents a 4x 4 matrix used to represent the position relationship; m represents the elements in the matrix, and solving M is to solve 16 m. The position relationship M can also be generally represented by Rt, where R represents rotation and t represents translation. Specifically, if there are n intersections, in the case of known coordinates in space and coordinates in CT, there are 3n linear equations about the elements of the M matrix, and the unknown number m is solved by the least squares method to the rough position relationship M of the C-arm machine when the CT and two DR images are taken. The rough position relationship M of the C-arm machine when the CT and two DR images are taken is quickly initialized to meet the initial position requirements of the precise registration. During specific operations, different feature points can be mapped, or the 3D model can be directly dragged to match the 3D model with the world coordinate system. However, the basis for dragging is also based on the matching of feature points that serve as identifiers.

[0110] S5: Based on the rough position relationship M, N feature points in the two-dimensional image and the preoperative three-dimensional model are automatically extracted for iterative optimization, the precise position of the preoperative three-dimensional model under the C-arm is calculated, and the two-dimensional and three-dimensional image registration is completed.

[0111] In some embodiments, the method of automatically extracting N feature points from the two-dimensional image and the preoperative three-dimensional model for iterative optimization in S5 includes:

[0112] S501: Calculate a projection operator based on the rough position relationship and the preoperative three-dimensional model data, and generate a DRR image; the DRR is created by the projection operator (camera parameters and projection integral) P;

[0113] S502: Calculating the similarity between the DRR image and the original X-ray image used to construct the preoperative three-dimensional model;

[0114] S503: Determine whether the similarity meets a preset threshold; when the similarity meets the preset standard, output the precise position of the preoperative three-dimensional model under the C-arm machine;

[0115] Optionally, the similarity calculation formula includes: argmin S(I x ,P(I CT ;θ));

[0116] Among them, I CT Represents the preoperative 3D model data constructed based on the preoperative CT image of the subject (I CT The image provides density information of different tissues in the patient's body); θ is the posture information of the preoperative 3D model, and its initial value is the rough position relationship M (R, t matrix) of the preoperative 3D model under the C-arm machine, and its initial value is the rough position relationship M; P is a projection operator, which is calculated based on the CT image and the position information of the C-arm machine when the CT image is taken, and the CT data (I CT ) and CT data (I CT ) generates a virtual X-ray, i.e., a DRR image, from the world pose information (θ) where the CT image is located. The CT image is only used to generate the DRR image; Ix represents the intraoperative X-ray image; S represents the similarity between Ix and the DRR image obtained after the projection operator is calculated. Specifically, the process of calculating the similarity is: respectively calculate the gradient maps of the two images, then subtract them, obtain the gradient difference, and finally calculate the corresponding root mean square as the image similarity. argmin refers to the index of the minimum value in the array;

[0117] Optionally, after S502 and before S503, the method further includes: determining whether the number of times the similarity is calculated and output exceeds the number of cycles, and if it exceeds the number of cycles, adjusting the rough position relationship and continuing to repeat steps S501-S503, outputting the similarity, and finally obtaining the spatial position of the preoperative three-dimensional model on the C-arm machine;

[0118] Optionally, the rough position relationship is adjusted through CMA-ES; CMA-ES is denoted as the adaptive covariance matrix evolution strategy algorithm; the reasons for adjustment through CMA-ES include: if the standard is a certain value, and the value cannot be reached and the cycle continues, it is necessary to consider whether there is a problem with the process or the value, so the number of cycles must be set, and whether the rough position relationship needs to be adjusted is determined based on the number of cycles.

[0119] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 3As shown, the device may include: one or more processors, and one or more memories; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0120] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0121] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0122] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0123] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5As shown, it is a schematic diagram of a storage medium provided by an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0124] The embodiments of the present disclosure further provide a computer program product or system, including a computer program, which implements the steps of the above method when executed by a processor.

[0125] In some embodiments, this embodiment also discloses a two-dimensional three-dimensional image registration system, such as Figure 2 As shown, the system comprises:

[0126] A two-dimensional image acquisition module 201 is used to acquire intraoperative anteroposterior and lateral two-dimensional images of the subject with registration balls, and the two-dimensional images are taken by a C-arm machine;

[0127] The module 202 for acquiring feature points in the two-dimensional image is used to select feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image, and acquire the two-dimensional coordinates of the feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image; and acquire the three-dimensional coordinates of the feature points in the world coordinate system based on the two-dimensional coordinates;

[0128] A feature point acquisition module 203 in the three-dimensional model is used to acquire the three-dimensional coordinates of the feature points in the preoperative three-dimensional model;

[0129] A 2D and 3D rough registration module 204 is used to match the coordinates of feature points in the preoperative 3D model with those in the world coordinate system to obtain a rough position relationship M of the preoperative 3D model under the C-arm machine;

[0130] The 2D / 3D precise registration module 205 is used to automatically extract N feature points from the 2D image and the preoperative 3D model based on the rough position relationship M for iterative optimization, calculate the precise position of the preoperative 3D model under the C-arm machine, and complete the 2D / 3D image registration.

[0131] Example

[0132] 1 Obtain the patient's anteroposterior and lateral images during surgery

[0133] 1) Purpose: To obtain the projection mathematical representation of the C-arm machine when shooting the anteroposterior and lateral dr images respectively (for any point P in space, if the mathematical representation is known, the position of the point in the image can be obtained)

[0134] 2) Image shooting requirements: VSPOTS (image register) tool with 12 steel balls is required

[0135] 3) Calculate the internal parameters of the C-arm (K) (mathematical model of fluoroscopic projection)

[0136] The specific calculation method of the internal parameters includes: through the known three-dimensional coordinates of the 12 steel balls in the VSPOTS reference system and the corresponding two-dimensional pixel coordinates on the image, due to the special design of the 12 steel balls, 4 straight lines formed by 3 points can be formed, and then based on these 4 straight lines, every two of them form a plane, a total of 6 planes can be obtained, each plane has 6 points, and according to Zhang Zhengyou's calibration method, the internal parameters can be obtained.

[0137] 4) Calculate the external parameters of the C-arm (R, T) (world position of the X-ray machine)

[0138] The specific calculation method of the external parameters includes: according to the internal parameters obtained above, the 12 three-dimensional coordinates in the VSPOTS reference system and the corresponding two-dimensional pixel coordinates are then used through the PNP algorithm to obtain the relative position and posture of the C-arm in the VSPOTS reference system. At the same time, by using the infrared optical ball of VSPOT, the relative position and posture of VSPOTS under the binocular tracker and the relative position and posture of the patient reference frame under the binocular tracker are recorded, and the relative position and posture of VSPOTS under the patient reference frame can be obtained. Then, based on the relative position and posture of the C-arm in the VSPOTS reference system, the relative position and posture of the C-arm in the patient reference frame can be obtained, which is the external parameter of the C-arm.

[0139] The relationship between the points in the calculation space and the corresponding points on the image can be used to calculate the perspective projection parameters of the C-arm machine through the positions of the 12 steel balls on the image and the calculation formula.

[0140] 2. Rough registration (four-point registration technique)

[0141] When the initial position deviates too much, the registration will be inaccurate, so start with rough and then accurate.

[0142] 1) Purpose: To quickly initialize the rough position relationship M between the CT and CB machine during the two DR image captures, so as to achieve the initial position requirement for precise registration.

[0143] 2) Requirements: Get the four points in CT and find the corresponding points in the anteroposterior and lateral dr images respectively. Through the CB projection parameters corresponding to the anteroposterior and lateral positions, we can get the corresponding anteroposterior and lateral rays in space and the intersection of the two rays in space; if there are n intersection points, and we know the coordinates in space and in CT, then we have 3n linear equations about the elements of the M matrix. Through the least squares method, we can solve the unknown number m and get the rough position relationship M of the CT and the CB machine when the two dr images were taken.

[0144] Specifically, the principle of coarse registration: In computer vision and image processing, when using camera projections to perform four-point registration to solve for position, it usually involves mapping from two-dimensional image coordinates to three-dimensional world coordinates. This type of problem is usually handled by solving the inverse problem of perspective transformation, that is, from image plane coordinates to three-dimensional space coordinates. The perspective transformation can be represented by a 3x4 projection matrix P, which maps three-dimensional world coordinates (X, Y, Z) to two-dimensional image coordinates (x, y), as shown in the following figure. Figure 6 As shown in the figure, the projection matrix P is a 3x4 matrix, which can be decomposed into the product of the intrinsic matrix K and the extrinsic matrix [R|t], P = K[R|t]; K is the intrinsic matrix, which contains the focal length and principal point coordinates of the C-arm. R is the rotation matrix, which describes the rotation from the world coordinate system to the camera coordinate system. t is the translation vector, which describes the translation from the world coordinate system to the camera coordinate system.

[0145] When the two-dimensional coordinates of four points on the image and their corresponding positions in the world coordinate system (the real coordinates of the object in the three-dimensional world) are given, the projection matrix P can be solved by solving a system of linear equations. Once P is obtained and the internal parameter K has been obtained by 12 steel ball calibration, the R t pose information can be obtained by SVD decomposition.

[0146] For four-point registration, at least four point coordinate pairs (xi, yi, Xi, Yi, Zi) are required to solve P. This is usually solved by setting up a linear equation system and using the least squares method. Assuming that there are four points in the image coordinates and the corresponding world coordinates, you can set them as shown in the following figure. Figure 7By solving the equations shown in the figure, all elements of the projection matrix P can be obtained. Then, by left-multiplying the inverse matrix of the intrinsic parameter K matrix, the pose information of R and t can be obtained. This is used as the initial value of the precise registration.

[0147] 3. Accurate registration

[0148] 1) Calculate the position of the preoperative CT scan under the C-arm.

[0149] The precise registration uses an optimized iterative algorithm to calculate the pose of the model R and t. The optimization objective function expression is as follows: argmin S(I x ,P(I CT ;θ));

[0150] Among them, (1) Ict represents CT model data, which is the CT scan images obtained from the patient. These images provide density information of different tissues in the patient's body.

[0151] (2)θ is the pose information of the ct model (R, t matrix)

[0152] (3) P-projection operator: Generates virtual X-ray (DDR) through ct data (Ict) and the world pose (θ) where the ct data is located. The calculation steps are as follows:

[0153] 1) Set projection parameters: determine the position and direction of the virtual X-ray source, as well as the position and size of the imaging plane (determined by θ and the internal parameter K and the resolution of DDR)

[0154] 2) Ray projection: The rays emitted from the X-ray source pass through the CT data to simulate the propagation path of the X-rays. For each ray, the attenuation of the ray passing through tissues of different densities is calculated.

[0155] 3) Integrated attenuation value: For each ray, the attenuation values ​​of all voxels (CT pixels) on its path are integrated to simulate the absorption and scattering of X-rays in the body.

[0156] 4) Generate image: Map the integrated attenuation values ​​of all rays to the imaging plane to generate the final DRR image.

[0157] The calculation of ray projection and attenuation can be done as shown in the following example. Figure 8 The formula shown in the figure is: Figure 8 In the formula shown, I(x,y) is the brightness of the point (x,y) on the imaging plane. I0 is the initial ray intensity. μ(s) is the attenuation coefficient of a point on the ray path s, which depends on the density of the tissue passed through. ∫μ(s)ds represents the integral of the attenuation coefficient along the ray path, i.e. the total attenuation. This process needs to be repeated for each ray until the entire imaging plane is covered, thus generating a virtual X-ray (DDR).

[0158] (4) Ix represents the original X-ray image.

[0159] (5) S represents the difference between the original image (Ix) and the image obtained after the projection operator (DDR). The gradient maps of the two images are calculated separately, then subtracted to obtain the gradient difference, and finally the corresponding root mean square is calculated as the image similarity.

[0160] 2) S represents the similarity of two images: the intraoperative perspective image (DRR) Ix (original image) and the virtual perspective image (DRR) (obtained by calculation). DRR is created by the projection operator (camera parameters and projection integral) P, and θ is the position of the CT under the C-arm before surgery.

[0161] Step 1: Set the preoperative CT position θ

[0162] Step 2: Create a virtual image DRR

[0163] Step 3: Calculate the similarity between the two images (calculate the mean square error)

[0164] Step 4: Determine whether the similarity s meets the threshold or exceeds the number of cycles

[0165] Step 5: Change the position θ of the preoperative ct by CMA-ES (Adaptive Covariance Matrix Evolution Strategy Algorithm), and repeat steps 1-4 until the position θ meets the conditions.

[0166] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A two-dimensional three-dimensional image registration method, characterized in that: The method comprises: S1: Obtain an intraoperative anteroposterior two-dimensional image and a lateral two-dimensional image of the subject with a registration ball; calculate C-arm projection parameters based on the projection coordinates of the registration ball in the two-dimensional image and a calibration algorithm; the calculation method of the C-arm projection parameters includes: calculating the C-arm internal parameters based on the three-dimensional coordinates of the registration ball in the VSPOTS reference system and its corresponding two-dimensional coordinates in the two-dimensional image; calculating the C-arm external parameters including the rotation matrix R and the translation vector T based on the C-arm internal parameters; S2: Selecting feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image, obtaining the two-dimensional coordinates of the feature points in the two-dimensional anteroposterior image and the two-dimensional lateral image; and obtaining the three-dimensional coordinates of the feature points in the world coordinate system based on the two-dimensional coordinates; S3: obtaining the three-dimensional coordinates of the feature points in the preoperative three-dimensional model; S4: Pairing the coordinates of the feature points in the preoperative three-dimensional model with the feature points in the world coordinate system to obtain a rough position relationship M of the preoperative three-dimensional model under the C-arm machine; adjusting the rough position relationship by CMA-ES; S5: Based on the rough position relationship M, automatically extract N feature points from the two-dimensional image and the preoperative three-dimensional model for iterative optimization, calculate the precise position of the preoperative three-dimensional model under the C-arm machine, and complete the alignment; the method for automatically extracting N feature points from the two-dimensional image and the preoperative three-dimensional model for iterative optimization includes: S501: calculating a projection operator based on the rough position relationship and the preoperative three-dimensional model, and generating a DRR image; DRR is created through the projection operator P; S502: calculating the similarity between the DRR image and the original X-ray image used to construct the preoperative three-dimensional model; S503: judging whether the similarity meets the preset threshold; when the similarity meets the preset threshold, outputting the precise position of the preoperative three-dimensional model under the C-arm machine.

2. The two-dimensional three-dimensional image registration method according to claim 1, characterized in that: The method for obtaining the three-dimensional coordinates of the feature point in the world coordinate system based on the two-dimensional coordinates in S2 includes: based on the two-dimensional coordinates of any feature point in the anteroposterior two-dimensional image and the lateral two-dimensional image, obtaining the anteroposterior ray corresponding to any feature point in the anteroposterior two-dimensional image and the lateral ray in the lateral two-dimensional image corresponding to the corresponding feature point in the anteroposterior two-dimensional image in space through the C-arm projection parameters corresponding to the anteroposterior two-dimensional image and the lateral ray in the lateral two-dimensional image, and the intersection point of the anteroposterior ray and the lateral ray in space, the coordinates of the intersection point are the coordinates of any feature point in the world coordinate system, recorded as .

3. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: For any feature point, it identifies the same position of the subject in the frontal two-dimensional image and the lateral two-dimensional image.

4. The two-dimensional three-dimensional image registration method according to claim 1, characterized in that: The feature points are feature points that can be used to identify key parts of the subject.

5. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The method for pairing the coordinates of feature points in the preoperative three-dimensional model with those in the world coordinate system in S4 includes: receiving an operation instruction for the world coordinate system or the preoperative three-dimensional model, and pairing the three-dimensional coordinates of the feature points in the preoperative three-dimensional model with those in the world coordinate system using the least squares method to obtain the rough position relationship M; specifically, if there are n intersections, in the case of known coordinates in the space and coordinates in the CT, there are 3n linear equations about the elements of the M matrix, and the unknown number m is solved by the least squares method to obtain the rough position relationship M of the C-arm machine when the CT and two dr images are taken.

6. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The number of the feature points is a natural number greater than or equal to 3.

7. The two-dimensional and three-dimensional image registration method according to claim 6, characterized in that: The number of the feature points is greater than or equal to 4; when the number of the feature points is 4, the 4 feature points are arranged asymmetrically with respect to each other.

8. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The calculation formula of the similarity includes: ; Among them, I CT represents the preoperative three-dimensional model constructed based on the preoperative CT image of the subject; is the pose information of the preoperative 3D model, and its initial value is the rough position relationship M of the preoperative 3D model under the C-arm machine; P is the projection operator, which is calculated based on the CT image and the position information under the C-arm machine when the CT image is taken. CT and I CT Located Generate virtual X-ray, i.e. DRR image; I x represents intraoperative X-ray images; S represents I x The similarity between the DRR image calculated by the projection operator.

9. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: After S502 and before S503, the method further includes: determining whether the number of times the similarity is calculated and output exceeds the number of cycles; if so, adjusting the rough position relationship and continuing to repeat steps S501-S503, outputting the similarity, and finally obtaining the spatial posture of the preoperative three-dimensional model in the C-arm machine, that is, the precise position.

10. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The calculation method of the C-arm internal parameters includes: using the three-dimensional coordinates of the registration ball in the VSPOTS reference system and the two-dimensional coordinates corresponding to the two-dimensional image; the registration balls are A balls, and when every two or three balls are connected, they form B lines, and the extension lines of the B lines intersect at a point on different planes, and every two extension lines form a plane, and based on the plane and the points on the plane, the internal parameters are obtained according to the Zhang Zhengyou calibration method; wherein A and B are natural numbers greater than 4, and A is greater than B.

11. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The calculation method of the C-arm external parameters includes: obtaining the relative posture of the C-arm and the VSPOTS reference system based on the C-arm internal parameters, the three-dimensional coordinates of the registration ball in the VSPOTS reference system, and the two-dimensional coordinates of the registration ball in the two-dimensional image; obtaining the relative posture of VSPOTS and the reference frame based on the relative posture of VSPOTS and the camera, and the relative posture of the reference frame fixed on the subject and the camera; obtaining the relative posture of the C-arm and the reference frame based on the relative posture of VSPOTS and the reference frame, and the relative posture of the C-arm and the VSPOTS reference system, that is, the C-arm external parameters.

12. The two-dimensional and three-dimensional image registration method according to claim 11, characterized in that: The relative position of the VSPOTS and the camera and the relative position of the reference frame on the subject and the camera are recorded based on the optical tracking ball of the VSPOTS.

13. The two-dimensional and three-dimensional image registration method according to claim 11, characterized in that: The camera is a binocular tracker.

14. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The intraoperative anteroposterior two-dimensional images and lateral two-dimensional images are images of the VSPOTS tool with a registration ball.

15. The two-dimensional and three-dimensional image registration method according to claim 1, characterized in that: The preoperative three-dimensional model is constructed based on the preoperative CT images taken by the subject; the method for constructing the preoperative three-dimensional model includes: S301, obtaining a three-dimensional CT image of the subject's spine before surgery, and segmenting the three-dimensional CT image to obtain a W value mask image containing a ROI spine region; S302, cropping the single vertebra in the W value mask image containing the ROI spine region to obtain a W value mask image in which adjacent single vertebra are not connected; S303, locating the centroid of a single vertebra in the W-value mask image that is not connected between adjacent single vertebrae, and obtaining an image of the located centroid; S304, taking the single centroid as the center, segmenting from the image of the located centroid to obtain a single vertebra image corresponding to the single centroid.

16. The two-dimensional and three-dimensional image registration method according to claim 15, characterized in that: The W is a natural number greater than or equal to 2.

17. The two-dimensional and three-dimensional image registration method according to claim 15, characterized in that: The ROI spine region includes any one or more of the following regions: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacrum, and coccyx.

18. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

20. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 17 are implemented.

Images