Bone posture prediction method and system, computer equipment and storage medium

Abstract

The invention relates to a skeleton posture prediction method and system, computer equipment and a storage medium. The method comprises the steps that preoperative three-dimensional image data and intra-operative projection data of a target skeleton are acquired; obtaining an initial attitude matrix of the target skeleton according to the three-dimensional image data; obtaining preoperative projection data of the target skeleton according to the initial attitude matrix and the preoperative three-dimensional image data; performing style migration on the preoperative projection data to obtain intra-operative simulation projection data; and obtaining a target attitude matrix of the target skeleton according to the intraoperative simulation projection data and the intraoperative projection data. According to the method, the image style migration is performed on the preoperative projection data to obtain the intraoperative simulation projection data, so that the intraoperative simulation projection data is closer to the real intraoperative projection image to improve the registration precision, prediction is performed in combination with the actually acquired intraoperative projection data, the precision and robustness of bone pose prediction are improved, the iteration process is simplified, and the accuracy and robustness of bone pose prediction are improved. And the bone posture prediction efficiency is greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a method, system, computer device, and storage medium for predicting bone postures. Background Art

[0002] Image-guided surgery relies on the registration accuracy of preoperative and intraoperative image data and the accuracy of the image registration algorithm between 2D and 3D images.

[0003] However, when registering 2D and 3D images of bones, due to factors such as surgical postures, patient positions, patient individual differences, and image quality, there are significant differences in the bone postures of different patients, making it difficult to guarantee the accuracy of registration. Summary of the Invention

[0004] To solve the above technical problems, this application provides a method, system, computer device, and storage medium for predicting bone postures.

[0005] In a first aspect, this application provides a method for predicting bone postures, including:

[0006] S101. Obtain preoperative three-dimensional image data and intraoperative projection data of the target bone;

[0007] S102. Obtain an initial pose matrix of the target bone according to the three-dimensional image data;

[0008] S103. Obtain preoperative projection data of the target bone according to the initial pose matrix and the preoperative three-dimensional image data;

[0009] S104. Perform style transfer on the preoperative projection data to obtain intraoperative simulation projection data;

[0010] S105. Obtain a target pose matrix of the target bone according to the intraoperative simulation projection data and the intraoperative projection data.

[0011] In one embodiment, the intraoperative projection data includes intraoperative projection images of the target bone in projection spaces corresponding to at least two viewpoints, and the intraoperative simulation projection data includes intraoperative simulation projection images of the target bone in each of the projection spaces;

[0012] Step S105 includes:

[0013] Input the intraoperative simulation projection data and the intraoperative projection data into a pre-trained target bone pose prediction model to obtain a predicted pose matrix of the target bone in each of the projection spaces;

[0014] Determine the target pose matrix according to at least one of the predicted pose matrices.

[0015] In one embodiment, the target bone pose prediction model includes a geometric feature extraction network and a pose prediction network;

[0016] Input the intraoperative simulation projection data and the intraoperative projection data into the geometric feature extraction network respectively to obtain the intraoperative real geometric features and the intraoperative simulation geometric features;

[0017] Input the intraoperative real geometric features and the intraoperative simulation geometric features into the pose prediction network to obtain the predicted transformation information of the target bone;

[0018] Based on the predicted transformation information, determine the predicted pose matrices of the target bone in each of the projection spaces.

[0019] In one embodiment, the geometric feature extraction network includes a plurality of extraction networks with shared weights, and different extraction networks are used to extract geometric features from images corresponding to different projection spaces. The geometric features include density correspondence features, surface area attention features, and contour mask features.

[0020] In one embodiment, the predicted transformation information includes the initial predicted translation information and the predicted rotation information of the target bone in each of the projection spaces;

[0021] The determining the predicted pose matrices of the target bone in each of the projection spaces based on the predicted transformation information includes:

[0022] Determine the intersection points of a plurality of space lines according to the space lines formed by the initial predicted translation parameters and the ray light sources corresponding to the projection spaces;

[0023] Convert the intersection points into the ray light source coordinate systems corresponding to the projection spaces respectively to obtain the corresponding target predicted translation parameters;

[0024] Based on the predicted rotation information and the target predicted translation parameters, determine the predicted pose matrices of the target bone in each of the projection spaces respectively.

[0025] In one embodiment, the determining the target pose matrix according to at least one of the predicted pose matrices includes:

[0026] Determine a new initial pose matrix from the plurality of predicted pose matrices, and repeat steps S103 to S105 to iterate the initial pose matrix until the iteration stop condition is satisfied;

[0027] Determine the target pose matrix according to multiple said predicted pose matrices.

[0028] In one embodiment, the new initial pose matrix M N is expressed as:

[0029]

[0030] where N is the number of iterations of the initial pose matrix, is the frontal predicted pose matrix in the projection space corresponding to the frontal view of the target bone, is the lateral predicted pose matrix in the projection space corresponding to the lateral view of the target bone.

[0031] In one embodiment, the determining the target pose matrix according to multiple said predicted pose matrices includes:

[0032] Determine a target correction matrix according to the frontal predicted pose matrix and the lateral predicted pose matrix;

[0033] Use the target correction matrix to correct and transform the frontal predicted pose matrix or the lateral predicted pose matrix to the target coordinate system to obtain the target pose matrix;

[0034] where the target coordinate system is determined by the straight line formed by the ray light source of each of the frontal view and the lateral view to the corresponding ray detector.

[0035] In one embodiment, the preoperative projection data includes preoperative projection images of the target bone in each of the projection spaces;

[0036] Step S104 includes:

[0037] Input the preoperative projection images into a pre-trained image style transfer model respectively to obtain corresponding intraoperative simulation projection images, so as to form the intraoperative simulation projection data.

[0038] In one embodiment, the image style transfer model includes a CycleGan network, and the total loss of the CycleGan network is expressed as:

[0039] loss total = loss adv (G, D Y , X, Y) + loss adv (F, D X , X, Y) + λ loss cyc (G, F);

[0040]

[0041] Among them, loss total represents the total loss, and loss adv (G, D Y , X, Y) represents the adversarial loss between the generator G and the discriminator D Y ; loss adv (F, D X , X, Y) represents the adversarial loss between the generator F and the discriminator D X ; loss cyc (G, F) represents the cycle consistency loss between the generator G and the generator F. X represents the preoperative projection data, x represents the preoperative projection image in X, Y represents the intraoperative projection data, y represents the intraoperative projection image in Y. The generator G is used to convert the input image into an image with the same style as Y, the generator F is used to convert the input image into an image with the same style as X, and the discriminator D X is used to determine whether the style of the input image belongs to X, and the discriminator D Y is used to determine whether the style of the input image belongs to Y.

[0042] In one embodiment, obtaining the initial pose matrix of the target bone according to the three-dimensional image data includes:

[0043] Obtaining the first point cloud data of the target bone according to the preoperative three-dimensional image data;

[0044] Based on the first point cloud data and the second point cloud data of the preset bone, obtaining a registration matrix, where the registration matrix is used to register the target bone with the preset bone;

[0045] For each of the perspectives, obtaining the initial pose matrix according to the registration matrix and the standard pose matrix of the preset bone;

[0046] Or,

[0047] The intraoperative projection data includes intraoperative projection images of the target bone in the projection spaces corresponding to at least two perspectives;

[0048] Obtaining the initial pose matrix of the target bone according to the three-dimensional image data includes:

[0049] Obtaining the first point cloud data of the target bone according to the preoperative three-dimensional image data;

[0050] Inputting the first point cloud data and the intraoperative projection data into a pre-trained initial bone pose prediction model to obtain the first pose matrix of the target bone in each of the projection spaces;

[0051] Select one from the multiple first attitude matrices and transform it into the target coordinate system as the initial attitude matrix, where the target coordinate system is determined by the straight lines formed by the ray light sources of at least two of the perspectives to the corresponding ray detectors.

[0052] In a second aspect, the present application provides a bone attitude prediction system, including:

[0053] A data acquisition module for acquiring preoperative three-dimensional image data and intraoperative projection data of a target bone;

[0054] An initial attitude matrix acquisition module for obtaining the initial attitude matrix of the target bone according to the three-dimensional image data;

[0055] A preoperative data projection module for obtaining preoperative projection data of the target bone according to the initial attitude matrix and the preoperative three-dimensional image data;

[0056] An intraoperative data simulation module for performing style transfer on the preoperative projection data to obtain intraoperative simulated projection data;

[0057] A bone attitude prediction module for obtaining the target attitude matrix of the target bone according to the intraoperative simulated projection data and the intraoperative projection data.

[0058] In a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the bone attitude prediction method described in the first aspect is implemented.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the bone attitude prediction method described in the first aspect is implemented.

[0060] On the basis of conforming to the common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred examples of the present application.

[0061] The beneficial effects that the above bone attitude prediction method, system, computer device, and storage medium can achieve include: obtaining preoperative projection data of preoperative three-dimensional image data in the projection spaces corresponding to multiple perspectives based on the initial attitude matrix, and obtaining intraoperative simulated projection data through image style transfer to make it closer to the real intraoperative projection image, so as to improve the registration accuracy. Combining the actually collected intraoperative projection data for prediction solves the deficiency of single-perspective attitude estimation in depth information prediction, improves the accuracy and robustness of bone pose prediction, and simplifies the iterative process compared with the traditional iterative method, greatly improving the efficiency of bone attitude prediction. Description of the Drawings

[0062] Figure 1 It is a schematic flow chart of a bone pose prediction method in an embodiment;

[0063] Figure 2 It is a schematic diagram of a target coordinate system in an embodiment;

[0064] Figure 3 It is a schematic diagram of the deviation between the frontal prediction pose matrix and the lateral prediction pose matrix in an embodiment;

[0065] Figure 4 It is a schematic flow chart of bone pose prediction based on intraoperative projection images from dual perspectives in an embodiment;

[0066] Figure 5 A schematic flow chart of the training process of a target bone pose prediction model in an embodiment;

[0067] Figure 6 It is a schematic diagram of the modules of a bone pose prediction system in an embodiment;

[0068] Figure 7 It is the first internal structure diagram of a computer device in an embodiment;

[0069] Figure 8 It is the second internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0070] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] It should be noted that the illustrations provided in this embodiment only schematically illustrate the basic concept of the present application. The figures only show the components related to the present application, rather than being drawn according to the number, shape, and size of the components in actual implementation. The form, quantity, and proportion of each component in its actual implementation can be arbitrarily changed, and the layout form of its components may also be more complex. The structures, proportions, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the limiting conditions under which the present application can be implemented. Therefore, they do not have technical substantive significance. Any modification of the structure, change in the proportional relationship, or adjustment of the size, without affecting the effects that the present application can produce and the purposes that can be achieved, should still fall within the scope that can be covered by the technical content disclosed in the present application. At the same time, the terms such as "upper", "lower", "left", "right", "middle", and "one" cited in this specification are only for the convenience of clear narration, rather than used to limit the scope under which the present application can be implemented. The change or adjustment of their relative relationship, without substantial change in the technical content, should also be regarded as the scope under which the present application can be implemented.

[0072] References to "embodiments" in this document mean that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the text and does not necessarily refer to the same embodiment each time, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0073] As used herein, unless the context clearly indicates otherwise, words such as "a", "an", "one", and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0074] The definitions included herein, such as the terms "having", "may have", "including", or "may include" as used herein, indicate the existence of the corresponding functions, operations, elements, etc. of the present application, and do not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms "including" or "having" as used herein indicate the existence of the features, numbers, steps, operations, elements, components, or combinations thereof described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0075] In the embodiments of the present application, prefix words such as "first" and "second" are only used to distinguish different described objects, and have no restrictive effect on the position, order, priority, quantity, content, etc. of the described objects. In the embodiments of the present application, the use of prefix words such as ordinal numbers to distinguish described objects does not constitute a restriction on the described objects. For the statements of the described objects, refer to the descriptions in the claims or the context of the embodiments. It should not constitute an unnecessary restriction due to the use of such prefix words. In addition, in the description of this embodiment, unless otherwise specified, the meaning of "a plurality" is two or more.

[0076] Image-guided surgery is a multidisciplinary research direction that involves disciplines such as computer vision, biomedicine, and imaging. In modern surgical navigation systems, due to the popularity of tomographic images such as CT (Computer Tomography) and MRI (Magnetic Resonance Imaging), it is usually necessary to achieve real-time positioning of the target during the operation. The success of image-guided surgery depends to a large extent on the registration accuracy of preoperative image data and intraoperative image data, as well as the accuracy of the registration algorithm between 2D and 3D.

[0077] In a medical surgical navigation system, the navigation system based on 2D images has the problem of lack of depth information. Although it can be compensated by 2D images from multiple angles, it is not intuitive enough, prone to visual blind spots, and not friendly to novice doctors. The navigation system based on intraoperative 3D images has poor image quality due to dose limitations, which is likely to affect the doctor's judgment and use.

[0078] To ensure the quality of 3D images and real-time 2D image information during the operation, preoperative 3D images can be used to register intraoperative 2D images. However, intraoperative 2D images are affected by factors such as surgical posture, patient position, and image quality, and it is difficult to guarantee their accuracy and timeliness, and additional time and labor costs are required.

[0079] The registration technology of preoperative 3D images and intraoperative 2D images of medical images can be naturally expressed as a pose regression problem at different times and in different modalities. For example, given an image or a sequence of images, directly estimate the pose of the object therein, including information such as position, rotation, and scaling.

[0080] Traditional image registration algorithms are an iterative optimization process. In this process, one image is considered fixed and the other is moving. The best alignment is achieved by iteratively moving the moving image on the fixed image. This iterative method is very slow and requires extracting appropriate features, selecting similarity metrics and transformation models, and finally searching the search space through a mechanism to find the optimal solution. The process is cumbersome and time-consuming. However, the actual use in clinical operations is real-time. Such a long-time image registration is difficult to meet the real-time requirements in clinical operations.

[0081] In response to this, this application proposes a scheme for predicting iliac posture. Through the multi-angle intraoperative projection images and the multi-angle two-dimensional projection images of the preoperative three-dimensional image data, the preoperative projection data of the preoperative three-dimensional image data in the projection spaces corresponding to multiple perspectives is obtained based on the initial pose matrix. The intraoperative simulation projection data is obtained through image style transfer to make it closer to the real intraoperative projection image, so as to improve the registration accuracy. Combining with the actually collected intraoperative projection data for prediction, it solves the deficiency of single-view pose estimation in depth information prediction, improves the accuracy and robustness of bone pose prediction, and simplifies the iterative process compared with the traditional iterative method, greatly improving the efficiency of bone pose prediction.

[0082] In one embodiment, as Figure 1 shown, it includes steps S101 to S105:

[0083] S101. Obtain the preoperative three-dimensional image data and intraoperative projection data of the target bone.

[0084] Among them, the intraoperative projection data includes intraoperative projection images of the target bone in the projection spaces corresponding to at least two perspectives respectively. The target bone includes, but is not limited to, vertebral bodies such as lumbar vertebrae and cervical vertebrae, and bone blocks such as femurs, tibias and patellas.

[0085] Exemplarily, before the operation, the CT image of the target bone is obtained through tomography, that is, the preoperative three-dimensional image data of the target bone is obtained. Among them, the preoperative three-dimensional image data usually includes voxel information, spatial resolution and the position information of the image in the physical space, etc.

[0086] During the operation, the X-ray images of the target bone are obtained from the anterior-posterior view and the lateral view respectively through a G-type X-ray machine, that is, the intraoperative projection images in the anterior-posterior projection space and the lateral projection space are obtained, and they are used as the intraoperative projection data.

[0087] When taking an X-ray image of a target bone, the target bone needs to be placed between the X-ray light source and the X-ray detector. When the X-ray passes through the target bone, tissues with different densities and thicknesses will absorb X-rays to different degrees. The remaining X-rays are absorbed by the X-ray detector, and imaging can be performed based on the remaining X-rays absorbed by the X-ray detector to obtain an X-ray image. Tissues with higher density (such as bones) absorb more X-rays, so they appear as bright areas in the image; while tissues with lower density (such as muscles and fats) absorb less X-rays, so they appear as dark areas in the image.

[0088] S102. Obtain the initial pose matrix of the target bone according to the three-dimensional image data.

[0089] Among them, the initial pose matrix is used to express the pose information of the target bone.

[0090] In one embodiment, step S102 includes:

[0091] S201. Obtain the first point cloud data of the target bone according to the preoperative three-dimensional image data;

[0092] S202. Based on the first point cloud data and the second point cloud data of the preset bone, obtain a registration matrix, and the registration matrix is used to register the target bone with the preset bone;

[0093] S203. Obtain the initial pose matrix according to the registration matrix and the standard pose matrix of the preset bone.

[0094] Among them, the standard pose matrix is used to express the pose information of the preset bone. The standard pose matrix can be the gold standard obtained through the mechanism interactive registration method. For example, the mechanism interactive registration of the vertebral body usually involves accurately registering the vertebral bodies in medical images of different modalities. In this process, key feature points or regions of the vertebral body are selected manually or semi-automatically, and finally the standard pose matrix of the vertebral body is obtained. This is common knowledge in this field and will not be elaborated here.

[0095] Exemplarily, first annotate the contour of the target bone in the preoperative CT image to obtain annotation data. Then, segment the target bone from the preoperative CT image according to the annotation data to obtain segmentation data. Finally, obtain the point cloud data of the target bone according to the segmentation data.

[0096] The edge voxel set of the point cloud data can be expressed as:

[0097] P = {p1, p2, …, p n}, n ∈ N *

[0098] Among them, p i = [x i , yi , z i , 1], i = {1, 2, …, n}, p i represents the spatial position of the i-th edge voxel, x i , y i , z i respectively represent the x-axis, y-axis, and z-axis coordinates of the i-th edge voxel, and the last element of p i being 1 indicates that the type of the edge voxel is a point, and if it is 0, it indicates that the type of the edge voxel is a vector.

[0099] Since the sizes and shapes of different target bones are different, the number of edge voxels contained in the point cloud data obtained from the contour data is usually also different. The farthest point sampling method can be used to ensure that the selected neighbor nodes are distributed globally, guarantee the effective capture of the global features of the point cloud data, and convert the point cloud data of the target bones in different sample data into structured data with the same number of edge voxels.

[0100] The set of edge voxels after the point cloud data is sampled by the farthest point sampling function can be expressed as:

[0101]

[0102] where F fps is the farthest point sampling function, and n1 is the expected number of edge voxels, that is, the number of edge voxels after being sampled by the farthest point sampling function.

[0103] Obtain the standard pose matrix and point cloud data of the preset bone corresponding to the target bone, and calculate the registration matrix for registering the two based on the point cloud data of the target bone and the preset bone. According to the registration matrix and the standard pose matrix, the initial pose matrix of the target bone can be obtained.

[0104] Based on this, for bones with different shapes, the corresponding initial pose matrix can be obtained quickly and efficiently, thereby improving the efficiency of bone pose prediction to meet the real-time requirements in clinical operations.

[0105] Considering the flexibility of the scheme and the accuracy of the preoperative projection image obtained based on the initial pose matrix, a pre-trained bone pose prediction model can also be used to obtain the initial pose matrix based on the point cloud data of the target bone and the intraoperative projection data.

[0106] That is, in another embodiment, step S102 includes:

[0107] S301. Obtain the first point cloud data of the target bone according to the preoperative three-dimensional image data;

[0108] S302. Input the first point cloud data and the intraoperative projection data into the initially trained initial bone pose prediction model to obtain the first pose matrix of the target bone in each projection space;

[0109] S303. Select one from multiple first pose matrices and transform it to the target coordinate system as the initial pose matrix.

[0110] Among them, the implementation process of step S301 can refer to that of step S201.

[0111] The initial bone pose prediction model is used to obtain the pose features of the target bone according to the first point cloud data, obtain the geometric features of the target bone corresponding to each perspective according to the intraoperative projection data, fuse the pose features and geometric features, and output the first pose matrix of the target bone in each projection space according to the fused features.

[0112] In one embodiment, step S303 includes:

[0113] S3031. Construct the target coordinate system.

[0114] Among them, the target coordinate system is determined by the straight line formed by the ray source corresponding to at least two perspectives to the corresponding ray detector.

[0115] Exemplarily, referring to Figure 2 , the ray source of the anterior-posterior perspective is the anterior-posterior ray source The ray detector corresponding to the anterior-posterior ray source is the anterior-posterior ray detector. Similarly, the ray source of the lateral perspective is the lateral ray source The ray detector corresponding to the lateral ray source is the lateral ray detector. Taking The straight line formed towards the anterior-posterior ray detector as the X-axis, and taking The straight line formed towards the lateral ray detector as the Y-axis, and taking the intersection point O between the two straight lines int as the origin to construct the target coordinate system.

[0116] The axis information of the target coordinate system can be expressed as:

[0117]

[0118] S3032. Obtain the transformation matrix between the target coordinate system and the ray source coordinate system corresponding to each perspective respectively.

[0119] The coordinate system with the anterior-posterior ray source as the origin is the anterior-posterior ray source coordinate system. According to the target coordinate system and the anterior-posterior ray source coordinate system, the first transformation matrix can be obtained for transforming the spatial position in the target coordinate system to the anterior-posterior ray source coordinate system.

[0120] Similarly, the coordinate system with the lateral ray light source as the origin is the lateral ray light source coordinate system. According to the target coordinate system and the lateral ray light source coordinate system, a second transformation matrix can be obtained. It is used to transform the spatial position in the target coordinate system into the lateral ray light source coordinate system.

[0121] S3033. Use the transformation matrix to transform the selected first attitude matrix into the target coordinate system to obtain the initial attitude matrix.

[0122] According to the projection space where the selected first attitude matrix is located, use the first transformation matrix or the second transformation matrix to transform it into the target coordinate system.

[0123] S103. According to the initial attitude matrix and the preoperative three-dimensional image data, obtain the preoperative projection data of the target bone.

[0124] That is, use the initial attitude matrix to project the target bone into the target projection space according to the preoperative three-dimensional image data to obtain the corresponding preoperative projection images. Among them, the target projection space is determined according to the projection space where the intraoperative projection image included in the intraoperative projection data is located.

[0125] Based on this, the obtained preoperative projection data includes the preoperative projection images of the target bone in each projection space.

[0126] S104. Perform style transfer on the preoperative projection data to obtain intraoperative simulation projection data.

[0127] Among them, the intraoperative simulation projection data includes the intraoperative simulation projection images corresponding to each intraoperative projection image. In one embodiment, step S104 includes:

[0128] Input the preoperative projection images into the pre-trained image style transfer model respectively to obtain the corresponding intraoperative simulation projection images to form the intraoperative simulation projection data. That is, the intraoperative simulation projection data includes the intraoperative simulation projection images of the target bone in each projection space.

[0129] In one embodiment, an existing image style transfer model can be used for training and learning, such as NeuralStyle Transfer (neural style transfer), CycleGAN, StyleGAN, DualStyleGAN, SigStyle, etc.

[0130] In another embodiment, a CycleGan network can be built based on Pytorch, so that the CycleGan network includes a generator G, a generator F, and a discriminator D X and discriminator DY Among them, X represents the preoperative projection data, Y represents the intraoperative projection data, the generator G is used to convert the input image into an image with the same style as Y, and the generator F is used to convert the input image into an image with the same style as X. The discriminator D X is used to determine whether the style of the input image belongs to X. If it belongs, it outputs 1; if it does not belong, it outputs 0; the discriminator D Y is used to determine whether the style of the input image belongs to Y. If it belongs, it outputs 1; if it does not belong, it outputs 0. That is, the image style transfer model includes the above CycleGan network.

[0131] Exemplarily, the training process of the CycleGan network includes:

[0132] Use the Adam optimizer to train the generator G and the generator F, and use Ranger to train the discriminator D X and the discriminator D Y for training.

[0133] In one embodiment, the loss function of the CycleGan network is constructed based on the adversarial loss and the cycle consistency loss.

[0134] The adversarial loss can be expressed as:

[0135]

[0136] Among them, x represents the preoperative projection image in X, y represents the intraoperative projection image in Y, loss adv (G,D Y ,X,Y) represents the adversarial loss between the generator G and the discriminator, loss adv (F,D X ,X,Y) represents the adversarial loss between the generator F and the discriminator, D Y (y) represents the discrimination result of the discriminator on y, D X (x) represents the discrimination result of the discriminator on x.

[0137] Expectation represents the average value of all logarithmic probabilities logD Y (y), G(x) represents the synthetic sample generated by the generator G using x, D Y (G(x)) represents the discrimination result of the discriminator D Y on the synthetic sample G(x), expectation represents the average value of all logarithmic probabilities log(1 - D Y (G(x))).

[0138] Expectation represents the average value of all logarithmic probabilities logDX The average value of (x), F(y) represents the synthetic sample generated by the generator F according to y, D X (F(x)) represents the discriminator D X The discrimination result of the synthetic sample F(x), the expectation Is expressed as the average value of all logarithmic probabilities log(1 - D X (F(y))).

[0139] The cycle consistency loss can be expressed as:

[0140]

[0141] Among them, loss cyc (G, F) represents the cycle consistency loss of the generators G and F, F(y) represents the synthetic sample generated by the generator F according to y, G(F(y)) represents the reconstructed y by the generator G according to the synthetic sample F(y), theoretically G(F(y)) = y, Represents the expectation of the L1 norm of (G(F(y)) - y), F(G(x)) represents the sample x reconstructed according to the generators G and F, Represents the expectation of the L1 norm of (F(G(x)) - x).

[0142] Based on this, the total loss of the CycleGan network is a linear combination of the adversarial loss and the cycle consistency loss, and can be expressed as:

[0143] loss total = loss adv (G, D Y , X, Y) + loss adv (F, D Y , X, Y) + λloss cyc (G, F);

[0144] Among them, loss total Represents the total loss of the CycleGan network, and λ is a hyperparameter used to balance the adversarial loss and the cycle consistency loss.

[0145] The image style transfer model includes the CycleGan network pre-trained according to the above training process to perform style transfer on the preoperative projection image to make it closer to the real intraoperative projection image. When using the CycleGan network to perform transfer on the preoperative projection image, only the generator G is used because the quality of the preoperative projection image is better, and the generator G can generate a higher-quality and real intraoperative projection-style image according to the preoperative projection image.

[0146] S105. Obtain the target pose matrix of the target bone according to the intraoperative simulation projection data and the intraoperative projection data.

[0147] In one embodiment, step S105 includes:

[0148] S1051. Input the intraoperative simulation projection data and the intraoperative projection data into a pre-trained target bone pose prediction model to obtain the predicted pose matrices of the target bone in each projection space;

[0149] S1052. Determine the target pose matrix according to at least one predicted pose matrix.

[0150] In one embodiment, the target bone pose prediction model includes a geometric feature extraction network and a pose prediction network.

[0151] The geometric feature extraction network is used to obtain the intraoperative real geometric features and the intraoperative simulation geometric features according to the intraoperative simulation projection data and the intraoperative projection data.

[0152] The pose prediction network is used to determine the target fusion features according to the intraoperative real geometric features and the intraoperative simulation geometric features, and output the predicted transformation information of the target bone according to the target fusion features. Among them, the predicted transformation information includes the predicted rotation matrices of the target bone in each projection space and the initial predicted translation parameters.

[0153] That is, step S1051 includes:

[0154] S401. Input the intraoperative simulation projection data and the intraoperative projection data into the geometric feature extraction network respectively to obtain the intraoperative real geometric features and the intraoperative simulation geometric features;

[0155] S402. Input the intraoperative real geometric features and the intraoperative simulation geometric features into the pose prediction network to obtain the predicted transformation information of the target bone;

[0156] S403. Based on the predicted transformation information, determine the predicted pose matrices of the target bone in each projection space.

[0157] Among them, the geometric features include density correspondence features, surface area attention features, and contour mask features.

[0158] Considering that the intraoperative projection images and the intraoperative simulation projection images obtained from the same perspective have similar structural features, for the intraoperative projection images and the intraoperative simulation projection images in the same projection space, using an extraction network with shared weights for feature extraction can further improve the accuracy of pose prediction.

[0159] Shared weights mean using the same weight parameters in different parts of the neural network. This method greatly reduces the number of model parameters, reduces the computational amount, and at the same time enables the model to learn the general features in the data and enhances the generalization ability of the model.

[0160] That is, in one embodiment, the geometric feature extraction network includes multiple extraction networks with shared weights. Generally speaking, different extraction networks are used to extract geometric features from images corresponding to different projection spaces. That is, the number of extraction networks is usually greater than or equal to the number of projection spaces involved in the intraoperative projection data.

[0161] Taking the intraoperative projection data including the anteroposterior intraoperative projection image and the lateral intraoperative projection image, and the geometric feature extraction network including the first ResNet network and the second ResNet network as an example:

[0162] The first ResNet network is used to extract the geometric features of the image in the projection space corresponding to the anteroposterior view, and the second ResNet is used to extract the geometric features of the image in the projection space corresponding to the lateral view.

[0163] ResNet itself solves the gradient problem in the training of deep networks through residual blocks. Combining the mechanism of weight sharing can further reduce the number of parameters of the ResNet network, thereby reducing the complexity of the parameters, making the network more efficient in the training and inference processes, and more stable during the training process.

[0164] When dealing with large-scale data sets, this stability is particularly important, which can accelerate the convergence speed of the model and improve the training efficiency.

[0165] At the same time, the combination of the residual structure of ResNet and the weight sharing mechanism can effectively extract multi-level features in the data, from simple local features to complex global features, improving the quality and efficiency of feature extraction.

[0166] Based on this, the ResNet network can learn the general patterns in the data without relying on the specific positions where the features appear. This enables the ResNet network with shared weights to better generalize when facing different input data, improving the performance of the model on different data sets and tasks. For example, in an image classification task, regardless of how the position, size, and pose of the object in the image change, the network can accurately extract its features and classify them.

[0167] Take the region corresponding to the target bone in the anteroposterior intraoperative projection image as the region of interest (ROI), and scale the region of interest to the first target region image with a size of 256×256. Input the first target region image into the first ResNet network, and through the first ResNet network, the anteroposterior geometric features can be extracted, including the anteroposterior contour mask feature, the anteroposterior density correspondence feature, and the anteroposterior surface area attention feature.

[0168] Among them, the frontal density corresponding features include the frontal 2D projection density feature and the frontal 3D spatial density, which are used to characterize the density correspondence of the target bone projected from 2D to 3D space from the frontal view. The frontal surface area attention feature, as an additional fuzzy perception supervision signal, implicitly characterizes the frontal symmetry of the target bone, and can improve the prediction accuracy of the projection space corresponding to the target bone from the frontal view. That is, the frontal geometric features include the frontal density features and the frontal surface area attention features.

[0169] Similarly, the area corresponding to the target bone in the lateral intraoperative projection image is used as the region of interest (ROI), and the region of interest is scaled to a second target region image with a size of 256×256. The second target region image is input into the second ResNet network, and the lateral geometric features can be extracted through the second ResNet network, including the lateral contour mask feature, the lateral density corresponding feature, and the lateral surface area attention feature.

[0170] Among them, the lateral density corresponding features include the lateral 2D projection density feature and the lateral 3D spatial density, which are used to characterize the density correspondence of the target bone projected from 2D to 3D space from the lateral view. The lateral surface area attention feature, as an additional fuzzy perception supervision signal, implicitly characterizes the lateral symmetry of the target bone, and can improve the prediction accuracy of the projection space corresponding to the target bone from the lateral view. That is, the lateral geometric features include the lateral density features and the lateral surface area attention features.

[0171] The frontal geometric features and the lateral geometric features obtained according to the intraoperative projection data are combined to obtain the intraoperative true geometric features.

[0172] The intraoperative simulation projection data includes the frontal intraoperative simulation projection image and the lateral intraoperative simulation projection image. Based on the above first ResNet network, the frontal intraoperative simulation geometric features can be obtained, and based on the above second ResNet network, the lateral intraoperative simulation geometric features can be obtained, and they are combined to obtain the intraoperative simulation geometric features.

[0173] In one embodiment, the pose prediction network includes a 2D convolution module and a fully connected module.

[0174] Among them, the 2D convolution module is used to output the target fusion feature according to the geometric features in the intraoperative true geometric features.

[0175] Exemplarily, first, the geometric features in the intraoperative true geometric features are fused to obtain a two-dimensional fusion feature, and convolution operation and flattening operation are performed on the two-dimensional fusion feature, and then it is fused with the intraoperative simulation geometric features to obtain the target fusion feature.

[0176] The fully connected module is used to output the predicted transformation information according to the target fusion feature.

[0177] In one embodiment, step S403 includes:

[0178] S4021. Determine the intersection points of multiple space lines according to the space line composed of the initial predicted translation parameters and the ray light sources in the corresponding projection space;

[0179] S4022. Convert the intersection points into the corresponding ray light source coordinate systems in the projection space respectively to obtain the corresponding target predicted translation parameters;

[0180] S4023. Based on the predicted rotation information and the target predicted translation parameters, determine the predicted pose matrices of the target bone in each projection space respectively.

[0181] Taking the intraoperative real geometric features including the anterior-posterior geometric features and the lateral geometric features, and the pose prediction network being the Patch-PnP network as an example:

[0182] The Patch-PnP network includes a 2D convolution module and a fully connected module. The 2D convolution module is used to fuse the anterior-posterior geometric features and the lateral geometric features to obtain two-dimensional fused features. The two-dimensional fused features can be expressed as:

[0183]

[0184] where F cat is the feature fusion operation, is the anterior-posterior density corresponding feature, is the anterior-posterior surface area attention feature, is the anterior-posterior contour mask feature, is the lateral density corresponding feature, is the lateral surface area attention feature, is the lateral contour mask feature.

[0185] Perform a convolution operation and a flattening operation on the two-dimensional fused features in sequence, and then fuse them with the intraoperative simulation geometric features to obtain the target fused features.

[0186] The target fused features can be expressed as:

[0187]

[0188] where F flat is the flattening operation, F conv is the convolution operation, is the intraoperative simulation geometric feature.

[0189] Input the target fusion feature into the fully-connected module of the Patch-PnP network. The fully-connected module is used to obtain the predicted transformation information of the target skeleton based on the target fusion feature. The predicted transformation information includes the initial predicted translation information and the predicted rotation information of the target skeleton in each projection space respectively.

[0190] For example, the predicted transformation information obtained based on a multi-layer perceptron (MLP) can be expressed as:

[0191]

[0192] Where, is the predicted transformation information, F MLP is the multi-layer perceptron, R f is the anterior-posterior predicted rotation matrix, R l is the lateral predicted rotation matrix, T f is the first predicted translation parameter of the target skeleton in the anterior-posterior projection space, T l is the second predicted translation parameter in the lateral projection space.

[0193] Since the predicted transformation information includes the initial predicted translation information and the predicted rotation information of the target skeleton in the anterior-posterior projection space and the lateral projection space respectively, it is necessary to convert the predicted translation information into the same coordinate system to determine the final pose matrix of the target skeleton.

[0194] The specific process is as follows:

[0195] First, construct the target coordinate system according to the description in the previous text, and obtain the transformation matrix between the target coordinate system and the ray light source coordinate system corresponding to each view respectively. Combining the distance between the ray light source and the corresponding detector, the spatial position of each predicted translation parameter and the corresponding ray light source in the corresponding projection space coordinate system can be obtained, and then the spatial position of each predicted translation parameter and the corresponding ray light source in the target coordinate system can be determined.

[0196] Then, determine multiple space lines and the intersection points of the multiple space lines according to the spatial position of each predicted translation parameter and the corresponding ray light source in the target coordinate system.

[0197] Finally, convert the intersection points from the target coordinate system into the anterior-posterior projection space and the lateral projection space respectively to obtain the corresponding target predicted translation parameters. According to the predicted rotation matrix and the target predicted translation parameters, determine the predicted pose matrices of the target skeleton in each projection space respectively.

[0198] Exemplarily, given the first transformation matrix and the second transformation matrix According to the distance dis between the anterior-posterior ray light source and the anterior-posterior ray detector f, and the distance dis between the lateral ray light source and the lateral ray detector l , it can be determined that:

[0199]

[0200] Among them, is the spatial position of the first predicted translation parameter in the frontal projection space coordinate system, is the spatial position of the frontal ray light source in the frontal projection space, is the spatial position of the second predicted translation parameter in the lateral projection space coordinate system, is the spatial position of the lateral ray light source in the lateral projection space coordinate system.

[0201] Based on the first transformation matrix, the spatial positions of the first predicted translation parameter and the frontal ray light source in the target coordinate system can be expressed as:

[0202]

[0203] Among them, is the spatial position of the first predicted translation parameter in the target coordinate system, is the spatial position of the frontal ray light source in the target coordinate system.

[0204] Similarly, based on the second transformation matrix, the spatial positions of the second predicted translation parameter and the lateral ray light source in the target coordinate system can be expressed as:

[0205]

[0206] Among them, is the spatial position of the second predicted translation parameter in the target coordinate system, is the spatial position of the lateral ray light source in the lateral ray light source coordinate system.

[0207] Based on the above and In the target coordinate system, two spatial lines L1 and L2 can be obtained, which are respectively expressed as:

[0208]

[0209] By solving the homogeneous linear equations, the (pseudo) intersection point of the two spatial lines can be obtained

[0210] Finally, through the above first transformation matrix and second transformation matrix, the intersection point can be transformed from the target coordinate space system to the frontal projection space coordinate system, and the target predicted translation parameter of the target bone in the frontal projection space, that is, the frontal predicted translation parameter, can be obtained and converting the intersection point from the target coordinate system to the lateral projection space coordinate system, the target predicted translation parameter of the target bone in the lateral projection space can be obtained, that is, the lateral predicted translation parameter

[0211] The anterior predicted translation parameter and the lateral predicted translation parameter can be expressed as:

[0212]

[0213] According to the above anterior predicted translation parameter and anterior predicted rotation matrix, an anterior predicted pose matrix can be constructed:

[0214]

[0215] wherein, the anterior predicted pose matrix is used to express the pose information of the target bone based on the anterior projection space coordinate system.

[0216] Similarly, according to the above lateral predicted translation parameter and lateral predicted rotation matrix, a lateral predicted pose matrix can be constructed:

[0217]

[0218] wherein, the lateral predicted pose matrix is used to express the pose information of the target bone based on the lateral projection space coordinate system.

[0219] It can be understood that the anterior predicted pose matrix and the lateral predicted pose matrix actually express the same pose of the target bone, but are expressed using different projection space coordinate systems. If both are converted back to the target coordinate system, they are theoretically the same. One can be selected from the predicted pose matrices of the target bone in each projection space and converted to the target coordinate system to be used as the target pose matrix.

[0220] However, due to the existence of prediction errors and system errors, see Figure 3 , after converting the predicted pose matrices expressed using different projection space coordinate systems back to the target space coordinate system, there are often certain pose deviations.

[0221] In this regard, the present application also proposes an iterative method to optimize the 6D pose information of the bone in the projection space, further reducing the pose deviation caused by converting the predicted pose matrices obtained from different projection space coordinate systems to the target space coordinate system.

[0222] In one embodiment, step S1052 includes:

[0223] S501. Determine a new initial pose matrix from multiple predicted pose matrices, and repeatedly execute steps S103 to S105 to iterate the initial pose matrix until the iteration stop condition is met;

[0224] S502. Determine the target pose matrix according to multiple predicted pose matrices.

[0225] See Figure 4 , obtain the preoperative CT image data of the target bone and the intraoperative projection images. The intraoperative projection images include the intraoperative anteroposterior X-ray image and the intraoperative lateral X-ray image.

[0226] Based on the preoperative CT image data, obtain the preoperative anteroposterior projection image and the preoperative lateral projection image respectively, and combine them to obtain the preoperative projection data.

[0227] Perform image style transfer processing on the preoperative projection data to obtain the intraoperative anteroposterior simulated X-ray image and the intraoperative lateral simulated X-ray image, and combine them to obtain the intraoperative simulated projection data.

[0228] Use the first ResNet network to extract the geometric features of the intraoperative anteroposterior X-ray image and the intraoperative anteroposterior simulated X-ray image, and use the first ResNet network to extract the geometric features of the intraoperative lateral X-ray image and the intraoperative lateral simulated X-ray image.

[0229] Fuse the geometric features to obtain the target fusion feature. Obtain the anteroposterior predicted pose matrix and the lateral predicted pose matrix of the target bone according to the target fusion feature.

[0230] Alternately select the anteroposterior predicted pose matrix and the lateral predicted pose matrix, and transform them into the target coordinate system to obtain a new initial pose matrix M N .

[0231]

[0232] Where N is the iteration number of the initial pose matrix, is the anteroposterior predicted pose matrix, is the lateral predicted pose matrix.

[0233] It can be understood that N is 1 when step S105 is executed for the first time, N is 2 when step S105 is executed for the second time, and so on. Each time step S105 is executed, N is incremented by 1.

[0234] Then, steps S103 to S105 are repeatedly executed. Based on the new initial pose matrix, re-projection is performed on the preoperative CT image data to obtain new preoperative projection data of the target bone. The preoperative projection data is subjected to style transfer to obtain new intraoperative simulation projection data. Then, according to the intraoperative simulation projection data and the intraoperative projection data, a new predicted pose matrix is obtained to achieve iteration of the pose matrix of the target bone. Stop after meeting the iteration stop condition, for example, when the iteration reaches a preset number of times, and the difference between the anteroposterior predicted pose matrix and the lateral predicted pose matrix is less than a preset value.

[0235] After the iteration stops, any one of the predicted pose matrices can be selected and transformed to the target coordinate system to obtain the target pose matrix of the target bone. However, to further improve the accuracy and robustness of bone pose prediction, the present application also provides a correction scheme for the dual-view fusion deviation.

[0236] That is, in one embodiment, step S502 includes:

[0237] S5021. Determine the target correction matrix according to the anteroposterior predicted pose matrix and the lateral predicted pose matrix;

[0238] S5022. Use the target correction matrix to correct and transform the anteroposterior predicted pose matrix or the lateral predicted pose matrix to the target coordinate system to obtain the target pose matrix.

[0239] Among them, the target coordinate system is determined by the straight line formed by the ray source of each of the anteroposterior view and the lateral view to the corresponding ray detector.

[0240] Exemplarily, one can choose or for correction. According to and an intermediate rotation matrix representing the deviation between the anteroposterior and lateral prediction results and an intermediate offset matrix

[0241] The intermediate rotation matrix can be expressed as:

[0242]

[0243] The intermediate offset matrix can be expressed as:

[0244]

[0245] Among them, represents the transformation matrix from the anteroposterior light source coordinate system to the lateral light source coordinate system,

[0246] Let Then

[0247] Convert to the quaternion Q, and Q can be expressed as:

[0248]

[0249] where θ is the angle of rotation, and μ is the rotation axis and rotation direction.

[0250] The sign of q0 can be arbitrarily selected, and the signs of q1, q2, and q3 can be determined according to the following formula:

[0251]

[0252] Let With the direction of the rotation axis μ unchanged, a new quaternion can be obtained:

[0253]

[0254] Convert P to the rotation correction matrix, and the obtained rotation correction matrix can be expressed as:

[0255]

[0256] Obtain the translation correction matrix according to the intermediate offset matrix Expressed as:

[0257]

[0258] Then the target correction matrix can be expressed as:

[0259]

[0260] Correct the matrix to be corrected according to the correction matrix, and the corrected transformation matrix can be expressed as:

[0261]

[0262] Use the first transformation matrix or the second transformation matrix to transform it to the target coordinate system to obtain the target attitude matrix. The target attitude matrix can be expressed as:

[0263]

[0264] In the above embodiments, the present application realizes the registration of preoperative 3D images to intraoperative 2D images through multi-angle intraoperative projection images and multi-angle 2D projection images of preoperative 3D image data, so as to reduce the influence of different patients and different body positions on the prediction accuracy of bone postures.

[0265] The present application also uses a pre-trained CycleGan network to perform style transfer on the multi-angle 2D projection images of preoperative 3D projection data to obtain corresponding intraoperative simulation projection images, making them close to real intraoperative projection images to improve the registration accuracy. Then, a hybrid depth network is used for iterative regression of double-angle projection images to further improve the registration accuracy and efficiency. Finally, combined with a deviation correction scheme for dual-view fusion, a fast and accurate pose estimation technology is ultimately achieved.

[0266] Exemplarily, for the training process of the above target bone pose prediction model, refer to Figure 5 , including:

[0267] First, obtain several groups of sample data. Each group of sample data includes a preoperative 3D image data sample of the target bone and an intraoperative projection data sample, where the intraoperative projection data sample includes an anteroposterior intraoperative projection image sample and a lateral intraoperative projection image sample. According to the preoperative 3D image data sample, an anteroposterior preoperative projection image and a lateral preoperative projection image of the target bone can be obtained, and through style transfer, an anteroposterior intraoperative simulation projection image sample and a lateral intraoperative simulation projection image sample can be obtained respectively as intraoperative simulation projection data samples.

[0268] Take the intraoperative projection data sample and the corresponding intraoperative simulation projection data sample as a group to obtain multiple training data. Then, divide several groups of training data into a training set and a test set according to a ratio, and the specific ratio can be set and adjusted according to actual needs. That is, the training set includes several groups of sample data, and each group of sample data includes an intraoperative simulation projection data sample and an intraoperative projection data sample of the target bone.

[0269] For each group of training data, set a corresponding standard pose matrix. That is, the label corresponding to each group of sample data includes the standard pose matrix of the target bone. The method for obtaining the standard pose matrix refers to the previous text and will not be elaborated here.

[0270] The target bone pose prediction model uses the first ResNet network to extract the geometric features of the anteroposterior intraoperative projection image and the anteroposterior intraoperative simulation projection image respectively according to the input training data, uses the second ResNet network to extract the geometric features of the lateral intraoperative projection image and the lateral intraoperative simulation projection image respectively, and then inputs the geometric features into the pose prediction network to obtain the predicted pose matrix of the target bone in the projection space corresponding to each view. Among them, the azimuth and number of views are determined by the specific views involved in the input sample data.

[0271] The predicted pose matrices in the projection spaces corresponding to different perspectives actually represent the same pose of the target skeleton, and the predicted pose matrices of different perspectives are expressed based on the coordinate systems corresponding to the perspectives. If the predicted pose matrices of the target skeleton corresponding to each perspective are converted to the same coordinate system for expression, they are theoretically the same.

[0272] Therefore, the pose prediction loss can be calculated according to the error between at least one of the predicted pose matrices and the standard pose matrix of the target skeleton, and then the pose prediction network can be updated according to the pose prediction loss until the prediction loss reaches the expected range and the training stops.

[0273] Among them, the pose prediction loss includes the multi-perspective space conversion loss and the 6D pose prediction loss.

[0274] (1) Multi-perspective space conversion loss

[0275] In order to regress the initial predicted translation parameter in the predicted transformation information, the target skeleton is regarded as a particle with the initial position at the origin, and the three-dimensional translation amount of the target skeleton is the initial predicted translation parameter of the matrix.

[0276] For example, based on the intraoperative simulation projection data samples and the intraoperative projection data samples, the predicted translation parameters T f 、T l are predicted. The regressed T f 、T l can be approximated as the first projection point pointA of the particle in the frontal ray light source coordinate system, and T l is approximated as the second projection point pointB of the particle in the lateral ray light source coordinate system. However, the regressed pointA and pointB are not strictly projected from one point. Only when the distance between pointA and pointB is 0 does it meet the condition of being projected from one particle. At this time, the target predicted translation parameter of the matrix can be calculated according to T f 、T l , that is, there is a dual-perspective space conversion loss loss t , loss t can be expressed as:

[0277] loss t = ||pointA - pointB||

[0278] The multi-perspective space conversion loss of three or more perspectives includes multiple dual-perspective space conversion losses. That is, according to the dual-perspective space conversion loss loss tThe calculation method is used to calculate the dual-view spatial transformation loss between each pair of perspectives respectively, and adding up multiple dual-view spatial transformation losses can obtain the multi-view spatial transformation loss.

[0279] (2) 6D pose prediction loss

[0280] In addition to the parameterization of rotation and translation, the choice of loss function is also crucial for 6D pose optimization. Most works do not directly utilize the rotation matrix and translation parameters, but instead adopt a variant of the point matching loss based on the ADD (Average Distance of Keypoints) (-S) metric to couple the rotation and translation estimations.

[0281] Among them, the ADD metric is an index used to evaluate the accuracy of three-dimensional object pose estimation. It is widely applied in fields such as robotics and computer vision, especially when evaluating the effectiveness of object pose estimation algorithms. ADD-S (Symmetric ADD) is an improved version of ADD, suitable for evaluating the pose estimation of symmetric objects.

[0282] In this embodiment, a new disentangled 6D pose loss variant can be adopted by separately supervising the rotation matrix R, the scale-invariant 2D object center (δx, δy), and the translation parameter δz.

[0283] For example, based on the intraoperative simulation projection data samples and intraoperative projection data samples, the anteroposterior prediction pose matrix and the lateral prediction pose matrix are predicted. The 6D pose prediction loss can be expressed as:

[0284]

[0285] Among them, is the anteroposterior pose prediction loss, is the lateral pose prediction loss, is the anteroposterior-lateral combined pose prediction loss, P is the set of edge voxels of the target vertebra point cloud data, x is the spatial position of a single voxel, is the anteroposterior standard pose matrix, is the lateral standard pose matrix, and E is the identity matrix.

[0286] For the 6D pose prediction loss of three or more perspectives, it is necessary to first calculate the pose prediction loss of each perspective separately, then calculate the combined pose prediction loss between each pair of perspectives respectively, and adding up the pose prediction losses and combined pose prediction losses of all perspectives can obtain the multi-view 6D pose prediction loss.

[0287] According to the above-mentioned pose prediction loss, by continuously adjusting the parameters in the target bone pose prediction model, such as network weights, biases, hyperparameters, learning rates, batch sizes, etc., the target bone pose prediction model can learn the patterns and features in the sample data to reduce the prediction loss of the model.

[0288] Finally, the performance of the trained target bone pose prediction model is evaluated using the sample data in the test set, and various performance metrics can be calculated as the performance evaluation results, such as prediction loss, prediction accuracy, recall rate, etc. Based on the performance evaluation results of the target bone pose prediction model, decide whether to continue training or put it into actual use.

[0289] After training the target bone pose prediction model according to the above training process, enter the testing process.

[0290] Exemplarily, input the N groups of sample data in the test set into the target bone pose prediction model to obtain the output anterior-posterior prediction pose matrix and the lateral prediction pose matrix

[0291] According to the first transformation matrix the anterior-posterior prediction pose matrix can be transformed into the target coordinate system to obtain According to the second transformation matrix the lateral prediction pose matrix can be transformed into the target coordinate system to obtain

[0292]

[0293] Theoretically, and the gap between them is very small. You can arbitrarily choose one of them, or use both at the same time to evaluate the performance of the trained bone pose prediction model.

[0294] Among them, and both include the predicted rotation information and the target predicted translation parameter which can be expressed as:

[0295]

[0296] The mean absolute error between the predicted rotation information and the standard rotation parameter can be calculated through Euler angles:

[0297]

[0298] In addition, the error between the target predicted translation parameter and the standard translation parameter can be calculated through the Euclidean distance:

[0299]

[0300] Among them, Euler represents the function that converts the rotation matrix into Euler angles. The standard rotation parameter corresponding to the i-th sample data is the standard translation parameter corresponding to the i-th sample data.

[0301] The standard rotation parameter and the standard translation parameter can be obtained from the standard pose matrix. The standard pose matrix is referred to in the previous text and will not be elaborated here.

[0302] It should be understood that although Figure 1 and Figure 4 each step in the flowchart of Figure 1 and Figure 4 is shown in sequence according to the indication of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover,

[0303] Corresponding to the foregoing bone pose prediction method, the present application also provides a bone pose prediction system.

[0304] In one embodiment, as Figure 6 shown, the bone pose prediction system includes:

[0305] A data acquisition module 601, configured to acquire preoperative three-dimensional image data and intraoperative projection data of the target bone;

[0306] An initial pose acquisition module 602, configured to obtain the initial pose matrix of the target bone according to the three-dimensional image data;

[0307] A preoperative data projection module 603, configured to obtain preoperative projection data of the target bone according to the initial pose matrix and the preoperative three-dimensional image data;

[0308] An intraoperative data simulation module 604, configured to perform style transfer on the preoperative projection data to obtain intraoperative simulated projection data;

[0309] A bone pose prediction module 605, configured to obtain the target pose matrix of the target bone according to the intraoperative simulated projection data and the intraoperative projection data.

[0310] For the specific limitations of the skeletal pose prediction system, reference can be made to the limitations of the skeletal pose prediction method in the foregoing text, which will not be elaborated herein. Each module in the above skeletal pose prediction system can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or independent thereof, or stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0311] This application also provides a computer device. In one embodiment, the computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the skeletal pose prediction method in the above embodiment are implemented.

[0312] In one embodiment, the computer device may be a server, and its internal structure diagram may be as Figure 7 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data for skeletal pose prediction. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a skeletal pose prediction method is implemented.

[0313] In one embodiment, the computer device may be a terminal, and its internal structure diagram may be as Figure 8 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a skeletal pose prediction method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0314] Those skilled in the art can understand, Figure 7 andFigure 8 The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0315] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the bone pose prediction method in the above embodiments are implemented.

[0316] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0317] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0318] The above embodiments only express several implementation manners of this application, and their descriptions are relatively specific and detailed, but they should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application patent should be subject to the appended claims.

Claims

1. A skeleton posture prediction method, characterized in that: The skeleton posture prediction method comprises the following steps: S101, obtaining preoperative three-dimensional image data and intraoperative projection data of the target bone; S102, obtaining an initial posture matrix of the target skeleton according to the three-dimensional image data; S103, obtaining preoperative projection data of the target bone according to the initial posture matrix and the preoperative three-dimensional image data; S104, performing style migration on the pre-operative projection data to obtain intra-operative simulation projection data; S105. Obtain a target posture matrix of the target bone according to the intraoperative simulation projection data and the intraoperative projection data.

2. The skeleton posture prediction method according to claim 1, characterized in that: The intraoperative projection data includes intraoperative projection images of the target bone in projection spaces corresponding to at least two viewing angles, and the intraoperative simulation projection data includes intraoperative simulation projection images of the target bone in each of the projection spaces; Step S105 includes: Inputting the intraoperative simulation projection data and the intraoperative projection data into a pre-trained target bone posture prediction model to obtain a prediction posture matrix of the target bone in each of the projection spaces; The target pose matrix is ​​determined based on at least one of the predicted pose matrices.

3. The skeleton posture prediction method as claimed in claim 2, characterized in that: The target skeleton posture prediction model includes a geometric feature extraction network and a posture prediction network; Inputting the intraoperative simulated projection data and the intraoperative projection data into the geometric feature extraction network respectively to obtain the intraoperative real geometric features and the intraoperative simulated geometric features; Inputting the intraoperative real geometric features and the intraoperative simulated geometric features into the posture prediction network to obtain the predicted transformation information of the target bone; Based on the predicted transformation information, a predicted posture matrix of the target bone in each of the projection spaces is determined.

4. The skeleton posture prediction method according to claim 3, characterized in that: The geometric feature extraction network includes multiple extraction networks that share weights, and different extraction networks are used to extract geometric features from images corresponding to different projection spaces. The geometric features include density correspondence features, surface area attention features, and contour mask features.

5. The skeleton posture prediction method according to claim 4, characterized in that: The predicted transformation information includes the initial predicted translation information and the predicted rotation information of the target bone in each of the projection spaces; The step of determining the predicted posture matrix of the target bone in each of the projection spaces based on the predicted transformation information comprises: Determine the intersection points of a plurality of spatial straight lines according to the spatial straight lines formed by the initial predicted translation parameters and the ray light sources corresponding to the projection space; The intersection points are respectively converted into the ray light source coordinate system corresponding to the projection space to obtain corresponding target prediction translation parameters; Based on the predicted rotation information and the target predicted translation parameters, the predicted posture matrix of the target bone in each of the projection spaces is determined respectively.

6. The skeleton posture prediction method according to claim 2, characterized in that: Determining the target posture matrix according to at least one of the predicted posture matrices comprises: Determine a new initial posture matrix from the multiple predicted posture matrices, and repeat steps S103 to S105 to iterate the initial posture matrix until an iteration stop condition is met; The target posture matrix is ​​determined according to the plurality of predicted posture matrices.

7. The skeleton posture prediction method according to claim 6, characterized in that: The new initial posture matrix M N It is expressed as: Wherein, N is the number of iterations of the initial posture matrix, is the orthogonal predicted posture matrix of the target bone in the projection space corresponding to the orthogonal viewing angle, The lateral predicted posture matrix of the target bone in the projection space corresponding to the lateral viewing angle.

8. The skeleton posture prediction method according to claim 7, characterized in that: Determining the target posture matrix according to the plurality of predicted posture matrices comprises: Determining a target correction matrix according to the frontal predicted posture matrix and the lateral predicted posture matrix; The target correction matrix is ​​used to correct the frontal prediction posture matrix or the lateral prediction posture matrix and transform it into a target coordinate system to obtain the target posture matrix; The target coordinate system is determined by the straight lines formed from the respective ray light sources of the orthogonal viewing angle and the lateral viewing angle to the corresponding ray detectors.

9. The skeleton posture prediction method according to claim 1, characterized in that: The preoperative projection data includes preoperative projection images of the target bone in each of the projection spaces; Step S104 includes: The pre-operative projection images are respectively input into a pre-trained image style transfer model to obtain corresponding intra-operative simulated projection images to form the intra-operative simulated projection data.

10. The skeleton posture prediction method according to claim 9, characterized in that: The image style transfer model includes a CycleGan network, and the total loss of the CycleGan network is expressed as: loss total =loss adv (G,D Y ,X,Y)+loss adv (F,D X ,X,Y)+λloss cyc (G,F); Among them, loss total Represents the total loss, loss adv (G,D Y ,X,Y) represents the generator G and the discriminator D Y The adversarial loss between adv (F,D X ,X,Y) represents the generator F and the discriminator D X The adversarial loss between cyc (G, F) represents the cycle consistency loss of the generator G and the generator F, X represents the preoperative projection data, x represents the preoperative projection image in X, Y represents the intraoperative projection data, y represents the intraoperative projection image in Y, the generator G is used to convert the input image into an image consistent with the style of Y, the generator F is used to convert the input image into an image consistent with the style of X, and the discriminator D X Used to determine whether the style of the input image belongs to X, the discriminator D Y Used to determine whether the style of the input image belongs to Y.

11. The skeleton posture prediction method according to claim 1, characterized in that: The step of obtaining the initial posture matrix of the target skeleton according to the three-dimensional image data comprises: Obtaining first point cloud data of the target bone according to the preoperative three-dimensional image data; Based on the first point cloud data and the second point cloud data of the preset bone, a registration matrix is ​​obtained, wherein the registration matrix is ​​used to register the target bone with the preset bone; For each of the viewing angles, obtaining the initial posture matrix according to the registration matrix and the standard posture matrix of the preset skeleton; or, The intraoperative projection data includes intraoperative projection images of the target bone in projection spaces corresponding to at least two viewing angles; The step of obtaining the initial posture matrix of the target skeleton according to the three-dimensional image data comprises: Obtaining first point cloud data of the target bone according to the preoperative three-dimensional image data; Inputting the first point cloud data and the intraoperative projection data into a pre-trained initial bone posture prediction model to obtain a first posture matrix of the target bone in each of the projection spaces; One is selected from the plurality of first posture matrices and converted to a target coordinate system as the initial posture matrix, wherein the target coordinate system is determined by straight lines formed from the respective ray light sources of at least two of the viewing angles to the corresponding ray detectors.

12. A skeleton posture prediction system, characterized in that: The skeleton posture prediction system comprises: A data acquisition module, used to acquire preoperative three-dimensional image data and intraoperative projection data of the target bone; An initial posture matrix acquisition module, used for obtaining an initial posture matrix of the target bone according to the three-dimensional image data; A preoperative data projection module, used for obtaining preoperative projection data of the target bone according to the initial posture matrix and the preoperative three-dimensional image data; An intraoperative data simulation module, used for performing style migration on the preoperative projection data to obtain intraoperative simulated projection data; The bone posture prediction module is used to obtain the target posture matrix of the target bone according to the intraoperative simulation projection data and the intraoperative projection data.

13. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the skeletal posture prediction method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the skeletal posture prediction method according to any one of claims 1 to 11 is implemented.

Images