Skeleton pose prediction method and device, computer device and storage medium

Abstract

The application relates to a bone posture prediction method and device, computer equipment and a storage medium, comprising: acquiring perspective images of a target bone in at least two visual angles in surgery and three-dimensional image data of the target bone before surgery; obtaining point cloud data of the target bone according to the three-dimensional image data; inputting the perspective images and the point cloud data into a pre-trained bone posture prediction model, fusing features extracted from the perspective images and the point cloud data respectively, and outputting a prediction posture matrix of the target bone corresponding to each of the visual angles according to the fused features. The posture features extracted from the three-dimensional image data of the bone before surgery are used as geometric guidance, the posture of the bone can be predicted based on the perspective images of the bone in multiple visual angles in surgery, the problem of insufficient depth information in single-angle posture prediction is solved, and the accuracy of posture prediction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bone posture prediction, in particular to a bone posture prediction method and device, computer equipment and a storage medium. BACKGROUND

[0002] When registering 2D images and 3D images of bones, the accuracy of registration is difficult to guarantee due to the influence of factors such as surgical posture, patient position, patient individual differences and image quality, and the bone postures of different patients have great differences.

[0003] The traditional bone posture prediction method is implemented based on bone images under a single perspective, which is inaccurate for bone posture prediction, mainly because of the following problems:

[0004] Different bones have differences in 3D structure, and are therefore easily affected by occlusion and perspective changes, and have poor robustness.

[0005] The clarity of bone images under a single perspective is insufficient, and the 3D texture of bones is missing, which cannot provide accurate depth information. SUMMARY

[0006] To solve the above technical problems, the present application provides a bone posture prediction method, device, computer equipment and storage medium.

[0007] In a first aspect, the present application provides a bone posture prediction method, which comprises:

[0008] Obtaining perspective images of a target bone under at least two perspectives in surgery, and three-dimensional image data of the target bone before surgery;

[0009] Obtaining point cloud data of the target bone according to the three-dimensional image data;

[0010] Inputting the perspective images and the point cloud data into a pre-trained bone posture prediction model, the bone posture prediction model fusing features extracted from the perspective images and the point cloud data respectively, and outputting a predicted posture matrix of the target bone corresponding to each perspective according to the fused features.

[0011] In one embodiment, the bone posture prediction model comprises a posture prediction network.

[0012] Before the step of inputting the perspective images and the point cloud data into the pre-trained bone posture prediction model, further comprising:

[0013] obtaining sample data, the sample data comprising perspective image samples of the target bone in at least two views during surgery and three-dimensional image data samples of the target bone before surgery;

[0014] obtaining point cloud data samples of the target bone according to the three-dimensional image data samples;

[0015] extracting geometric features of the target bone corresponding to each of the views respectively according to the perspective image samples;

[0016] extracting pose features of the target bone according to the point cloud data samples;

[0017] inputting the geometric features and the pose features into the pose prediction network, and outputting predicted pose matrices of the target bone corresponding to each of the views respectively through the pose prediction network;

[0018] calculating a pose prediction loss according to errors between at least one of the predicted pose matrices and a standard pose matrix of the target bone;

[0019] updating the pose prediction network according to the pose prediction loss.

[0020] In an embodiment, the bone pose prediction model further comprises a pose feature extraction network;

[0021] The step of extracting the pose features of the target bone according to the point cloud data samples comprises:

[0022] constructing a plurality of face structure features according to the point cloud data samples, the face structure feature comprising three spatial points connected to each other, a center point of the three spatial points, and a normal vector of a plane formed by the three spatial points;

[0023] inputting the plurality of face structure features into the pose feature extraction network, and obtaining the pose features through the pose feature extraction network.

[0024] In an embodiment, the bone pose prediction model further comprises a geometric feature extraction network;

[0025] The step of extracting the geometric features of the target bone corresponding to each of the views respectively comprises:

[0026] extracting a target region image comprising the target bone from each of the perspective image samples respectively;

[0027] inputting the target region image into the geometric feature extraction network, and obtaining predicted density features and predicted surface region attention features corresponding to each of the views through the geometric feature extraction network.

[0028] In an embodiment, the geometry feature extraction network is further configured to extract a predicted contour mask feature of each of the views from the target region image respectively;

[0029] After the step of extracting the geometry feature of each of the views from the target skeleton respectively, the method further comprises:

[0030] According to the predicted contour mask feature, the predicted density feature and the predicted surface region attention feature, a geometry feature loss of each of the views is calculated respectively;

[0031] According to the geometry feature loss of each of the views, a multi-view geometry feature loss is calculated;

[0032] The geometry feature extraction network is updated according to the multi-view geometry feature loss.

[0033] In an embodiment, the step of outputting a predicted pose matrix of each of the views from the target skeleton by the pose prediction network comprises:

[0034] The predicted density feature and the predicted surface region attention feature of each of the views are fused to obtain a two-dimensional fusion feature;

[0035] The two-dimensional fusion feature and the pose feature are fused to obtain a target fusion feature;

[0036] According to the target fusion feature, a predicted pose matrix of each of the views from the target skeleton is determined.

[0037] In an embodiment, the step of determining a predicted pose matrix of each of the views from the target skeleton comprises:

[0038] The target fusion feature is input into a full connection module in the pose prediction network to obtain a predicted transformation information of each of the views from the target skeleton, the predicted transformation information comprising a predicted rotation parameter and a predicted translation parameter of the target skeleton in an imaging coordinate system corresponding to the view;

[0039] According to a space straight line composed of each of the predicted translation parameters and a corresponding ray light source, an intersection of a plurality of the space straight lines is determined;

[0040] The intersection is respectively converted into a ray light source coordinate system corresponding to each of the views to obtain a corresponding target predicted translation parameter;

[0041] Based on the predicted rotation parameter and the target predicted translation parameter, the predicted pose matrix of each of the views is determined respectively.

[0042] In one embodiment, the step of determining the intersection of the plurality of straight lines in space comprises:

[0043] constructing a target coordinate system according to the ray source of at least two of the perspectives and the straight line formed by the ray source to the corresponding ray detector;

[0044] obtaining a conversion matrix between the target coordinate system and the ray source coordinate system corresponding to each of the perspectives, respectively;

[0045] obtaining the predicted translation parameter of each of the perspectives and the spatial position of the corresponding ray source in the target coordinate system based on the conversion matrix, respectively;

[0046] determining a plurality of straight lines in space according to the predicted translation parameter of each of the perspectives and the spatial position of the corresponding ray source in the target coordinate system;

[0047] determining the intersection of the plurality of straight lines in space.

[0048] In one embodiment, the sample data comprises perspective image samples of the target bone in at least two of the perspectives during surgery, and three-dimensional image data samples of the target bone before surgery;

[0049] The step of outputting the predicted pose matrix of the target bone corresponding to each of the perspectives by the pose prediction network comprises:

[0050] inputting the frontal geometric feature, the lateral geometric feature and the pose feature into the pose prediction network to obtain the frontal predicted pose matrix and the lateral predicted pose matrix output by the pose prediction network;

[0051] The pose prediction loss comprises a multi-perspective spatial conversion loss and a 6D pose prediction loss;

[0052] The multi-perspective spatial conversion loss is represented as:

[0053] loss t =||pointA-pointB||

[0054] Wherein, loss t is the multi-perspective spatial conversion loss, pointA is the first projection point of the target bone in the frontal ray source coordinate system, and pointB is the second projection point of the target bone in the lateral ray source coordinate system;

[0055] The 6D pose prediction loss is represented as:

[0056]

[0057] wherein, is a positive pose prediction loss, is a lateral pose prediction loss, is a positive-lateral combined pose prediction loss, M is a point cloud data sample of a target skeleton, p is a spatial position of an edge voxel in the point cloud data sample, is a conversion matrix for converting a spatial position in the target coordinate system to a positive position ray source coordinate system, is a conversion matrix for converting a spatial position in the target coordinate system to a lateral position ray source coordinate system, is a positive standard pose matrix, is the positive predicted pose matrix, is a lateral standard pose matrix, is the lateral predicted pose matrix, E is an identity matrix.

[0058] In a second aspect, the present application provides a skeleton pose prediction device, the device comprising:

[0059] a data acquisition module, configured to acquire perspective images of a target skeleton in at least two views during surgery, and three-dimensional image data of the target skeleton before surgery;

[0060] a data preprocessing module, configured to obtain point cloud data of the target skeleton according to the three-dimensional image data;

[0061] a skeleton pose prediction module, configured to input the perspective images and the point cloud data into a pre-trained skeleton pose prediction model, the skeleton pose prediction model fusing features extracted from the perspective images and the point cloud data respectively, and outputting a predicted pose matrix of the target skeleton corresponding to each of the views according to the fused features.

[0062] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the skeleton pose prediction method of the first aspect.

[0063] In a fourth aspect, the present application provides a computer readable storage medium, having a computer program stored thereon, wherein the computer program is executable by a processor to implement the skeleton pose prediction method of the first aspect.

[0064] On the basis of common knowledge in the art, the above-mentioned preferred conditions can be combined arbitrarily, i.e. to obtain each preferred example of the present application.

[0065] The aforementioned skeletal pose prediction method, apparatus, computer equipment, and storage medium can achieve the following beneficial effects: extracting geometric features from perspective images obtained from at least two viewpoints, extracting pose features from preoperative three-dimensional image data of the skeleton, fusing the pose features as geometric guides with geometric features from multiple viewpoints, and predicting the pose of the skeleton based on the fused features. This can solve the problem of insufficient depth information when performing pose prediction based on a single viewpoint skeleton image, and improve the accuracy of pose prediction. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the training process of a skeletal pose prediction model in one embodiment.

[0067] Figure 2 This is a schematic diagram of the target coordinate system in one embodiment;

[0068] Figure 3 This is a flowchart illustrating a skeletal pose prediction method in one embodiment;

[0069] Figure 4 This is a schematic diagram of a skeletal pose prediction device in another embodiment;

[0070] Figure 5 This is a first internal structure diagram of a computer device in one embodiment;

[0071] Figure 6 This is a second internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] It is to be noted that the drawings provided in the embodiments are only schematic and are intended to provide the basic understanding of the application. In the drawings, the shape, the number and the size of the components are not drawn according to the actual implementation, and the shape, the number and the size of the components in the actual implementation can be changed arbitrarily, and the component layout form can be more complex. The structure, the ratio, the size, etc. shown in the drawings of the present specification are only used to cooperate with the content disclosed in the specification, so as to be understood and read by those skilled in the art, and do not limit the defined conditions that can be implemented by the present application, so it does not have the technical significance. Any modification of the structure, change of the ratio relationship or adjustment of the size, without affecting the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" in the specification are only for the convenience of clear understanding of the description, and are not used to limit the scope of the application that can be implemented, and the change or adjustment of the relative relationship is also regarded as the scope of the application that can be implemented without changing the technical content.

[0074] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly contemplated that embodiments described herein can be combined with other embodiments.

[0075] As used herein, unless the context clearly indicates otherwise, the words "comprise", "comprising", "consisting of and "consisting essentially of" do not preclude the addition of one or more additional elements, features, steps, or components.

[0076] As used herein, the term "comprise", "comprising", "comprises" or "comprised of" indicates the presence of the described feature, operation, element, etc. in the corresponding function, operation, element, etc. described herein, and does not preclude the presence or addition of one or more other features, operations, elements, etc. In addition, it should be understood that the term "comprise" or "comprising", as used herein, indicates the presence of the features, numbers, steps, operations, elements, components or combinations thereof described in the specification, without excluding the presence or addition of one or more other features, numbers, steps, operations, elements, components or combinations thereof.

[0077] The prefix words such as "first", "second" in the embodiments of the present application are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of ordinal words such as "first" in the embodiments of the present application does not limit the described objects, and the description of the described objects should be referred to the description in the context of claims or embodiments, and should not be construed as redundant limitation because of the use of such prefix words. In addition, in the description of the embodiments, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0078] The present application provides a method for predicting the posture of a bone, comprising:

[0079] Obtaining perspective images of the target bone from at least two perspectives during surgery, and three-dimensional image data of the target bone before surgery;

[0080] Obtaining point cloud data of the target bone from the three-dimensional image data;

[0081] Inputting the perspective images and the point cloud data into a pre-trained bone posture prediction model, the bone posture prediction model fusing the features extracted from the perspective images and the point cloud data respectively, and outputting a predicted posture matrix of the target bone corresponding to each perspective based on the fused features.

[0082] Wherein, the target bone includes but is not limited to vertebral bodies such as lumbar vertebrae and cervical vertebrae, and bone blocks such as femur, tibia and patella.

[0083] The present application uses the geometric features of the target bone extracted from at least two perspectives during surgery to fuse with the posture features obtained based on the three-dimensional image data of the target bone before surgery, and predicts the posture information of the target bone in different perspective spaces. Based on the predicted posture information, the registration from the preoperative three-dimensional image data of the target bone to the intraoperative two-dimensional image can be realized, so as to reduce the influence of different patients and different body positions on the registration accuracy, and further improve the registration accuracy.

[0084] The predicted posture matrix of the target bone corresponding to each perspective actually represents the same posture of the target bone, and the predicted posture matrix of different perspectives is expressed based on the corresponding coordinate system of the perspective. If the predicted posture matrix of the target bone corresponding to each perspective is converted to the same coordinate system for expression, they are theoretically the same.

[0085] Wherein, the bone posture prediction model includes a posture prediction network. Before using the bone posture prediction model, a plurality of sample data can be used to iteratively train the posture prediction network, and the prediction loss is calculated according to the predicted posture matrix output by the bone posture prediction model and the standard posture matrix, until the prediction loss reaches the expected range to stop training.

[0086] Firstly, a plurality of sets of sample data are acquired, and the plurality of sets of sample data are divided into a training set and a test set according to a proportion, and the specific proportion can be set and adjusted according to actual needs.

[0087] Each set of sample data includes perspective image samples of the target bone at at least two viewing angles in surgery and three-dimensional image data samples of the target bone before surgery.

[0088] Then, the initial bone posture prediction model is trained by using the sample data in the training set. The specific training process includes steps S101-S107.

[0089] S101, acquiring sample data.

[0090] S102, obtaining point cloud data samples of the target bone according to the three-dimensional image data samples.

[0091] In an embodiment, step S102 includes:

[0092] S1021, obtaining segmentation data from the three-dimensional image data according to the contour of the target bone;

[0093] S1022, obtaining point cloud data samples of the target bone according to the segmentation data.

[0094] The principle of obtaining the point cloud data and the point cloud data samples of the target bone is the same. First, the contour of the target bone in the three-dimensional image data is labeled to obtain labeled data. Then, the target bone is segmented from the three-dimensional image data samples according to the labeled data to obtain segmentation data. Finally, the point cloud data samples of the target bone are obtained according to the segmentation data.

[0095] The edge voxel set of the point cloud data can be expressed as:

[0096] P={p1,p2,…,p n},n∈N *

[0097] Wherein, p i =[x i ,y i ,z i ,1],i={1,2,…,n},p i represents the spatial position of the i-th edge voxel, x i , y i , z i represent the x-axis, y-axis and z-axis coordinates of the i-th edge voxel, respectively, and the last element of p i is 1, indicating that the type of the edge voxel is a point, and if it is 0, it indicates that the type of the edge voxel is a vector.

[0098] Due to the different sizes and shapes of the target bone in different sample data, the number of edge voxels contained in the point cloud data obtained according to the contour data is usually different. The farthest point sampling method can be used to ensure that the selected neighbor nodes are distributed in a global range, effectively capture the global features of the point cloud data, and convert the point cloud data of the target bone in different sample data into structured data with the same number of edge voxels.

[0099] The edge voxel set after the point cloud data is sampled by the farthest point sampling function can be expressed as:

[0100]

[0101] where F fps is the farthest point sampling function, and n1 is the expected number of edge voxels, that is, the number of edge voxels after the farthest point sampling function is sampled.

[0102] S103、According to the perspective image sample, the geometric features corresponding to each view angle of the target bone are extracted respectively.

[0103] In one embodiment, the bone pose prediction model further includes a geometric feature extraction network. Accordingly, step S103 includes:

[0104] S1031, the target region image including the target bone is extracted from each perspective image sample respectively;

[0105] S1032, the target region image is input into the geometric feature extraction network, and the predicted density feature and the predicted surface region attention feature corresponding to each view angle are obtained by the geometric feature extraction network. The predicted density feature includes a predicted 2D projection density feature and a predicted 3D space density feature.

[0106] That is, the geometric features include the predicted density feature and the predicted surface region attention feature.

[0107] In another embodiment, the geometric feature extraction network is further used to extract the predicted contour mask feature of each view angle according to the target region image. That is, the geometric features further include the predicted contour mask feature.

[0108] Accordingly, after step S1032, it further includes:

[0109] S1033, according to the predicted contour mask feature, the predicted density feature and the predicted surface region attention feature, the geometric feature loss of each view angle is calculated respectively;

[0110] S1034, according to the geometric feature loss of multiple view angles, the multi-view geometric feature loss is calculated;

[0111] S1035、updating the geometric feature extraction network according to the multi-view geometric feature loss.

[0112] wherein the multi-view geometric feature loss can be a sum of geometric feature losses of multiple views. The multi-view geometric feature loss can be represented as:

[0113]

[0114] The geometric feature loss of the s-th view can be represented as:

[0115]

[0116] wherein, denotes the geometric feature loss of the s-th view, denotes the standard contour mask feature of the target bone in the s-th view, denotes the predicted contour mask feature of the target bone in the s-th view, denotes the predicted density feature of the target bone in the s-th view, denotes the standard density feature of the target bone in the s-th view, and CE denotes a cross-entropy loss function, is the predicted surface region attention feature of the s-th view, is the predicted surface region attention standard feature of the s-th view, and Θ denotes feature multiplication.

[0117] For example, referring to Figure 1 , the sample data includes an anteroposterior perspective image sample and a lateral perspective image sample of the target bone in surgery, and a three-dimensional image data sample of the target bone before surgery.

[0118] Taking the extraction of geometric features from the anteroposterior perspective image sample and the lateral perspective image sample as an example:

[0119] The region corresponding to the target bone in the anteroposterior perspective image is taken as a region of interest (ROI), which is scaled into a first target region image with a size of 256x256. The first target region image is input into a ResNet network, and the anteroposterior predicted contour mask feature, the anteroposterior predicted 2D projection density feature, the anteroposterior predicted 3D space density feature, and the anteroposterior predicted surface region attention feature can be extracted through the ResNet network.

[0120] The ortho-predicted 2D projection density feature and the ortho-predicted 3D space density are combined into an ortho-predicted density feature, which is used to represent the density correspondence of the target bone in the 2D projection to the 3D space in the ortho-perspective. The ortho-predicted surface region attention feature is an additional blur perception supervision signal, which implicitly represents the ortho-symmetry of the target bone and can improve the prediction accuracy of the target bone in the ortho-perspective space. That is, the ortho-geometric feature includes the ortho-predicted density feature and the ortho-predicted surface region attention feature.

[0121] Similarly, the region corresponding to the target bone in the lateral perspective image is taken as a region of interest (ROI), which is scaled into a second target region image with a size of 256x256. The second target region image is input into the ResNet network, and the lateral-predicted contour mask feature, the lateral-predicted 2D projection density feature, the lateral-predicted 3D space density feature, and the lateral-predicted surface region attention feature are extracted through the ResNet network.

[0122] The lateral-predicted 2D projection density feature and the lateral-predicted 3D space density are combined into a lateral-predicted density feature, which is used to represent the density correspondence of the target bone in the 2D projection to the 3D space in the lateral perspective. That is, the lateral-geometric feature includes the lateral-predicted density feature and the lateral-predicted surface region attention feature.

[0123] Correspondingly, the geometric feature regression loss includes an ortho-geometric feature loss and a lateral-geometric feature loss. The geometric feature regression loss can be represented as:

[0124]

[0125] wherein, the ortho-geometric feature loss is the lateral-geometric feature loss is. the ortho-standard contour mask feature is the ortho-predicted density feature is the ortho-standard density feature is the ortho-predicted contour mask feature is the ortho-predicted surface region attention feature is the ortho-standard surface region attention feature is. the lateral-standard contour mask feature is the lateral-predicted density feature is the lateral-standard density feature is the lateral-predicted contour mask feature is the lateral-predicted surface region attention feature is the lateral-standard surface region attention feature is.

[0126] S104, extracting a pose feature of the target skeleton according to the point cloud data sample.

[0127] In one embodiment, the skeleton pose prediction model further comprises a pose feature extraction network. Accordingly, step S104 comprises:

[0128] S1041, constructing a plurality of face structure features according to the point cloud data sample, the face structure feature comprising three spatial points connected to each other, a center point of the three spatial points, and a normal vector of a plane formed by the three spatial points;

[0129] S1042, inputting the plurality of face structure features into the pose feature extraction network to obtain the pose feature through the pose feature extraction network.

[0130] Wherein, the point cloud data sample comprises a plurality of spatial points, each spatial point having coordinate information in a three-dimensional space.

[0131] Each spatial point can be mapped as a node of a graph structure. However, the node in the graph structure can only represent the spatial structure feature of the target skeleton, and cannot effectively represent the spatial pose thereof. Therefore, in the present embodiment, a triangular face structure is used instead of a spatial point as a node to input into the pose feature extraction network, so as to obtain the pose feature of the target skeleton.

[0132] For example, sampling a plurality of spatial points in the point cloud data sample through a face structure feature sampling function can obtain a plurality of face structure features.

[0133] The plurality of face structure features can be represented as:

[0134]

[0135] Wherein, F mesh is a face structure feature sampling function, m is the number of spatial points in the point cloud data sample, is the mth face structure feature, is the three spatial points connected to each other in the mth face structure feature, is the center point of the three spatial points in the mth face structure feature, is the unit normal vector in the mth face structure feature.

[0136] On the basis of the GCN network, by using the triangular face structure instead of the spatial point as the face node input into the pose feature extraction network, not only the number of nodes can be reduced, but also the topological structure of the point cloud data can be better maintained through the connection relationship of the triangular face structure.

[0137] For a face structure based graph neural network Face-GCN, its adjacency matrix A represents the connection relationship between face nodes. If there is an edge between face node i and face node j, A ij = 1, otherwise A ij = 0.

[0138] Suppose that the face node features are updated using the adjacency matrix A, and the specific update rule is as follows:

[0139]

[0140] Where H l is the face node feature of the l-th layer, σ is an activation function, I is an identity matrix, is the adjacency matrix plus the self-loop adjacency matrix, is the degree matrix of , and W l is the weight matrix of the l-th layer.

[0141] Through multi-layer graph convolution operation, the representation of the face node can be updated step by step, so as to obtain multi-scale feature information. The face node feature representation of each layer is obtained by the output of the last layer through graph convolution operation. Therefore, after the above multi-layer graph convolution operation, the deep spatial feature of the target skeleton, i.e. the pose feature

[0142] S105, input the geometric feature and the pose feature into the pose prediction network, and output the predicted pose matrix of the target skeleton corresponding to each view angle through the pose prediction network.

[0143] In one embodiment, step S105 includes:

[0144] S1051, fuse the predicted density features and the predicted surface region attention features of multiple views to obtain two-dimensional fusion features;

[0145] S1052, fuse the two-dimensional fusion features and the pose features to obtain target fusion features;

[0146] S1053, determine the predicted pose matrix of the target skeleton corresponding to each view angle according to the target fusion features.

[0147] Wherein, step S1053 specifically includes:

[0148] S201, input the target fusion features into the full connection module in the pose prediction network to obtain the predicted transformation information of the target skeleton corresponding to each view angle, and the predicted transformation information includes the predicted rotation parameter and the predicted translation parameter of the target skeleton in the imaging coordinate system corresponding to the view angle;

[0149] S202, determining an intersection of the plurality of space straight lines according to each predicted translation parameter and a corresponding space straight line composed of a ray light source;

[0150] S203, converting the intersection into a ray light source coordinate system corresponding to each view angle respectively to obtain a corresponding target predicted translation parameter;

[0151] S204, determining a predicted pose matrix of each view angle based on the predicted rotation parameter and the target predicted translation parameter.

[0152] Exemplarily, the above orthosteric predicted density feature, orthosteric predicted surface area attention feature, lateral predicted density feature and lateral surface area attention feature are input into a pose prediction network, for example, a Patch-PnP network, to obtain an orthosteric predicted pose matrix and a lateral predicted pose matrix. The Patch-PnP network includes a 2D convolution module and a full connection module.

[0153] The 2D convolution module is used to fuse the orthosteric predicted density feature, the orthosteric predicted surface area attention feature, the lateral predicted density feature and the lateral surface area attention feature to obtain a two-dimensional fusion feature. The two-dimensional fusion feature can be represented as:

[0154]

[0155] wherein F cat is a feature fusion operation, is the orthosteric predicted density feature, is the orthosteric predicted surface area attention feature, is the lateral predicted density feature, is the lateral predicted surface area attention feature.

[0156] The two-dimensional fusion feature is sequentially subjected to a convolution operation and a flattening operation, and then fused with the pose feature of the target skeleton to obtain a target fusion feature.

[0157] The target fusion feature can be represented as:

[0158]

[0159] wherein F flat is a flattening operation, and F conv is a convolution operation.

[0160] The target fusion feature is input into the full connection module of the Patch-PnP network, for example, a multi-layer perception (MLP), to obtain predicted transformation information of the target skeleton corresponding to the orthosteric view angle and the lateral view angle respectively.

[0161] The predicted transformation information can be represented as:

[0162]

[0163] wherein, F is a prediction transformation information, MLP R is a multi-layer perception, f R is a frontal prediction rotation matrix, l T is a lateral prediction rotation matrix, f T is a first prediction translation parameter of the target bone in the imaging coordinate system corresponding to the frontal view, l T is a second prediction translation parameter in the imaging coordinate system corresponding to the lateral view.

[0164] Taking an X-ray image of a target bone as an example, the target bone needs to be placed between an X-ray light source and an X-ray detector. When X-rays pass through the target bone, different densities and thicknesses of tissues will absorb different degrees of X-rays. The remaining X-rays are absorbed by the X-ray detector, and according to the remaining X-rays absorbed by the X-ray detector, imaging can be performed to obtain an X-ray image. Higher density tissues (such as bones) absorb more X-rays, and thus appear as bright areas on the image; while lower density tissues (such as muscles, fat) absorb less X-rays, and thus appear as dark areas on the image.

[0165] Since the full connection module of the Patch-PnP network directly outputs the first prediction translation parameter and the second prediction translation parameter in the imaging coordinate system. In order to determine the spatial straight line between the prediction translation parameter and the corresponding ray light source, it is necessary to convert the first prediction translation parameter and the second prediction translation parameter in the prediction transformation information to a unified coordinate system.

[0166] In one embodiment, step S202 specifically includes S2021-S2024:

[0167] S2021, constructing a target coordinate system;

[0168] S2022, respectively acquiring a conversion matrix between the target coordinate system and the ray light source coordinate system corresponding to each view;

[0169] S2022, based on the conversion matrix, respectively obtaining the spatial position of each prediction translation parameter and the corresponding ray light source in the target coordinate system;

[0170] S2023, determining a plurality of spatial straight lines according to the spatial position of each prediction translation parameter and the corresponding ray light source in the target coordinate system;

[0171] S2024, determining the intersection of the plurality of spatial straight lines.

[0172] In step S2021, the X-axis and Y-axis of the target coordinate system can be determined by the straight lines formed by the ray light sources of any two views to the corresponding ray detectors.

[0173] For example, referring to Figure 2 , the ray source at the frontal view angle is a frontal ray source The ray detector corresponding to the frontal ray source is a frontal ray detector. Similarly, the ray source at the lateral view angle is a lateral ray source The ray detector corresponding to the lateral ray source is a lateral ray detector. For example The straight line formed towards the frontal ray detector is the X-axis, and The straight line formed towards the lateral ray detector is the Y-axis, and the intersection O int between the two straight lines is the origin to construct the target coordinate system. The coordinate axis information of the target coordinate system can be represented as:

[0174]

[0175] The coordinate system with the frontal ray source as the origin is the frontal ray source coordinate system. According to the target coordinate system and the frontal ray source coordinate system, a first conversion matrix is obtained for converting the spatial position in the target coordinate system to the frontal ray source coordinate system.

[0176] Similarly, the coordinate system with the lateral ray source as the origin is the lateral ray source coordinate system. According to the target coordinate system and the lateral ray source coordinate system, a second conversion matrix is obtained for converting the spatial position in the target coordinate system to the lateral ray source coordinate system.

[0177] In step S2023, the spatial positions of the predicted translation parameters in the corresponding ray source coordinate systems are first determined, and then the spatial positions of the predicted translation parameters and the corresponding ray sources in the target coordinate system are determined.

[0178] According to the distance dis f between the frontal ray source and the frontal ray detector, and the distance dis l between the lateral ray source and the lateral ray detector, it can be determined that:

[0179]

[0180] wherein, is the spatial position of the first predicted translation parameter in the frontal ray source coordinate system, is the spatial position of the second predicted translation parameter in the lateral ray source coordinate system.

[0181] According to the spatial positions of the predicted translation parameters in the corresponding ray source coordinate systems, the spatial positions of the predicted translation parameters and the corresponding ray sources in the target coordinate system are determined.

[0182] For example, the first predicted translation parameter and the spatial position of the frontal ray light source in the target coordinate system can be represented as:

[0183]

[0184] wherein, is the spatial position of the first predicted translation parameter in the target coordinate system, is the spatial position of the frontal ray light source in the frontal ray light source coordinate system, is the spatial position of the frontal ray light source in the target coordinate system.

[0185] Similarly, the second predicted translation parameter and the spatial position of the lateral ray light source in the target coordinate system can be represented as:

[0186]

[0187] wherein, is the spatial position of the second predicted translation parameter in the target coordinate system, is the spatial position of the lateral ray light source in the lateral ray light source coordinate system, is the spatial position of the lateral ray light source in the lateral ray light source coordinate system.

[0188] Based on the above and In the target coordinate system, two spatial straight lines L1 and L2 can be obtained, represented as:

[0189]

[0190] Through the solution of the homogeneous linear equations, the (pseudo) intersection point of the two spatial straight lines can be obtained

[0191] Finally, through the above first conversion matrix and second conversion matrix, the intersection point can be obtained from the target coordinate system to the frontal ray light source coordinate system, and the target predicted translation parameter of the frontal view angle, i.e., the frontal predicted translation parameter can be obtained. can be obtained from the target coordinate system to the lateral ray light source coordinate system, and the target predicted translation parameter of the lateral view angle, i.e., the predicted translation parameter

[0192]

[0193] According to the intersection point the frontal predicted translation parameter and the frontal predicted rotation matrix in the frontal ray light source coordinate system, the frontal predicted pose matrix can be constructed:

[0194]

[0195] Similarly, according to the intersection point The lateral prediction translation parameter and the lateral prediction rotation matrix in the lateral ray light source coordinate system can be used to construct a lateral prediction attitude matrix:

[0196]

[0197] S106, according to the error between at least one predicted attitude matrix and the standard attitude matrix of the target bone, the attitude prediction loss is calculated.

[0198] Wherein, the attitude prediction loss includes multi-view spatial conversion loss and 6D attitude prediction loss.

[0199] (1) Multi-view spatial conversion loss

[0200] In order to regress the target prediction translation parameter in the predicted attitude matrix, the target bone is regarded as a point, the initial position is at the origin, and the three-dimensional translation of the target bone is the target prediction translation parameter of the matrix.

[0201] For model training using two-view perspective image samples and three-dimensional image data samples, such as based on the above-mentioned frontal perspective image samples, lateral perspective image samples and three-dimensional image data samples, the predicted translation parameters T f , T l can be obtained. The regression obtained T f , T l can be approximated as the first projection point of the point in the frontal ray light source coordinate system pointA, and T l approximated as the second projection point of the point in the lateral ray light source coordinate system pointB. But the regression obtained pointA and pointB are not strictly projected by a point, only when the distance between pointA and pointB is 0, it meets the condition of projection by a point, at this time, the target prediction translation parameter of the matrix can be calculated according to T f , T l , that is, there is a double-view spatial conversion loss loss t , loss t can be expressed as:

[0202] loss t =||pointA-pointB||

[0203] The multi-view spatial conversion loss of three or more views includes multiple double-view spatial conversion losses. That is, according to the double-view spatial conversion loss loss tThe multi-view spatial conversion loss can be obtained by adding the plurality of double-view spatial conversion losses calculated by the calculation manner.

[0204] (2) 6D pose prediction loss

[0205] In addition to the parameterization of rotation and translation, the selection of the loss function is also crucial for 6D pose optimization. Most works do not directly utilize the rotation matrix and translation parameters, but instead adopt variants of point matching loss based on the ADD (Average Distance of Keypoints) (-S) metric to couple the rotation and translation estimates.

[0206] where the ADD metric is an index used to evaluate the accuracy of three-dimensional object pose estimation. It is widely used in the fields of robotics, computer vision, etc., especially in evaluating the effect of object pose estimation algorithms. ADD-S (Symmetric ADD) is an improved version of ADD, suitable for pose estimation evaluation of symmetric objects.

[0207] In this embodiment, a new unentangled 6D pose loss variant can be used by separately supervising the rotation matrix R, the scale-invariant 2D object center (δx, δy), and the translation parameter δz.

[0208] For model training using perspective image samples and three-dimensional image data samples of two views, such as based on the above-mentioned frontal perspective image samples, lateral perspective image samples, and three-dimensional image data samples, frontal predicted pose matrices and lateral predicted pose matrices are obtained. The 6D pose prediction loss can be represented as:

[0209]

[0210]

[0211] wherein, is the frontal pose prediction loss, is the lateral pose prediction loss, is the joint frontal-lateral pose prediction loss, M is the point cloud data sample of the target skeleton, and p is the spatial position of the edge voxel in the point cloud data sample. is the frontal standard pose matrix, is the lateral standard pose matrix, and E is the identity matrix.

[0212] For the 6D pose prediction loss of three or more views, the pose prediction loss of each view is calculated first, and then the joint pose prediction loss between each two views is calculated. The sum of the pose prediction loss of all views and the joint pose prediction loss can obtain the multi-view 6D pose prediction loss.

[0213] S107, updating the pose prediction network according to the pose prediction loss.

[0214] By continuously adjusting the parameters in the skeletal pose prediction model, such as network weights, biases, hyperparameters, learning rates, batch sizes, etc., the skeletal pose prediction model can learn the patterns and features in the sample data to reduce the prediction loss of the model.

[0215] Finally, the performance of the trained skeletal pose prediction model is evaluated by the sample data in the test set, and various performance indicators can be calculated as performance evaluation results, such as prediction loss, prediction accuracy, recall rate, etc. According to the performance evaluation results of the skeletal pose prediction model, it is selected to continue training or put into actual use.

[0216] After training the skeletal pose prediction model according to the above training process, enter the test process.

[0217] For example, the sample data includes an anteroposterior perspective image sample and a lateral perspective image sample of the target bone in the operation, and a three-dimensional image data sample of the target bone before the operation.

[0218] Input the N sample data in the test set into the skeletal pose prediction model to obtain the output anteroposterior predicted pose matrix and the lateral predicted pose matrix

[0219] According to the first conversion matrix the anteroposterior predicted pose matrix can be converted to the target coordinate system to obtain According to the second conversion matrix the lateral predicted pose matrix can be converted to the target coordinate system to obtain

[0220]

[0221] In theory, and The gap between them is very small, and one of them can be arbitrarily selected, or both can be used to evaluate the performance of the trained skeletal pose prediction model.

[0222] Among them, and Both include the predicted translation parameter of the predicted rotation parameter may be expressed as:

[0223]

[0224] The average absolute error of the predicted rotation parameters and the standard rotation parameters can be calculated by Euler angles:

[0225]

[0226] And the error of the predicted translation parameters and the standard translation parameters can be calculated by Euclidean distance:

[0227]

[0228] wherein Euler represents a function of converting a rotation matrix into Euler angles, the standard rotation parameters corresponding to the i-th sample data, is the standard translation parameters corresponding to the i-th sample data.

[0229] The standard rotation parameters and the standard translation parameters are obtained according to a standard pose matrix. The standard pose matrix can be a gold standard obtained by a mechanism interactive registration method. For example, the mechanism interactive registration of a vertebral body usually involves accurately registering the vertebral body in medical images of different modalities. In this process, key feature points or regions of the vertebral body are selected manually or semi-automatically, and finally the standard pose matrix of the vertebral body is obtained. This is a common knowledge in the art, and will not be described here.

[0230] The steps of performing the bone pose prediction by the bone pose prediction model trained based on the above process are as shown in Figure 3 , including:

[0231] Obtaining an anteroposterior perspective image and a lateral perspective image of a target bone in surgery, and three-dimensional image data of the target bone before surgery;

[0232] Performing point cloud processing on the three-dimensional image data sample to obtain point cloud data of the target bone;

[0233] Inputting the anteroposterior perspective image, the lateral perspective image and the point cloud data into the bone pose prediction model to respectively extract an anteroposterior geometric feature, a lateral geometric feature and a pose feature. Inputting the anteroposterior geometric feature, the lateral geometric feature and the pose feature into a pose prediction network, first fusing to obtain a target fusion feature, and then performing pose prediction based on the target fusion feature to obtain an anteroposterior predicted pose matrix and a lateral predicted pose matrix of the target bone.

[0234] It should be understood that, although Figure 1 and Figure 3The steps in the flowchart are displayed in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, Figure 1 And Figure 3 At least part of the steps in the above methods can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0235] Corresponding to the foregoing bone posture prediction method, the present application also provides a bone posture prediction device. In one embodiment, as shown in Figure 4 The bone posture prediction device comprises:

[0236] A data acquisition module 401 is configured to acquire perspective images of a target bone at at least two viewing angles during surgery and three-dimensional image data of the target bone before surgery.

[0237] A data preprocessing module 402 is configured to obtain point cloud data of the target bone according to the three-dimensional image data.

[0238] A bone posture prediction module 403 is configured to input the perspective images and the point cloud data into a pre-trained bone posture prediction model. The bone posture prediction model fuses features extracted from the perspective images and the point cloud data, respectively, and outputs a predicted posture matrix of the target bone corresponding to each viewing angle according to the fused features.

[0239] For specific limitations of the bone posture prediction device, refer to the limitations of the bone posture prediction method described above, which will not be repeated here. Each module in the above bone posture prediction device can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0240] The present application also provides a computer device. In one embodiment, the computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the bone posture prediction method in the above embodiments are implemented.

[0241] In one embodiment, the computer device can be a server, and its internal structure diagram can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store bone posture prediction data. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a bone posture prediction method.

[0242] In one embodiment, the computer device can be a terminal, and its internal structure diagram can be as shown in the figure. Figure 6 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a bone posture prediction method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0243] Those skilled in the art can understand that, Figure 4 and Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0244] The present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by the processor to implement the steps of the bone posture prediction method in the above embodiments.

[0245] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0246] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0247] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method of bone pose prediction, the method comprising: The method comprises the following steps: Obtaining perspective images of a target bone in at least two views during surgery and three-dimensional image data of the target bone before surgery; Obtaining point cloud data of the target bone according to the three-dimensional image data; Inputting the perspective images and the point cloud data into a pre-trained bone posture prediction model, wherein the bone posture prediction model is configured to extract geometric features of the target bone corresponding to each of the views according to the perspective images, extract posture features of the target bone according to the point cloud data, fuse the geometric features and the posture features, and output predicted posture matrices of the target bone corresponding to each of the views according to the fused features.

2. The skeletal pose prediction method of claim 1, wherein, The bone posture prediction model comprises a posture prediction network; Before the step of inputting the perspective images and the point cloud data into the pre-trained bone posture prediction model, the method further comprises the following steps: Obtaining sample data, wherein the sample data comprises perspective image samples of the target bone in at least two views during surgery and three-dimensional image data samples of the target bone before surgery; Obtaining point cloud data samples of the target bone according to the three-dimensional image data samples; Extracting geometric features of the target bone corresponding to each of the views according to the perspective image samples; Extracting posture features of the target bone according to the point cloud data samples; Inputting the geometric features and the posture features into the posture prediction network to output predicted posture matrices of the target bone corresponding to each of the views through the posture prediction network; Calculating a posture prediction loss according to errors between at least one of the predicted posture matrices and a standard posture matrix of the target bone; Updating the posture prediction network according to the posture prediction loss.

3. The skeletal pose prediction method of claim 2, wherein, The bone posture prediction model further comprises a posture feature extraction network; The step of extracting the posture features of the target bone according to the point cloud data samples comprises the following steps: Constructing a plurality of face structure features according to the point cloud data samples, wherein each of the face structure features comprises three spatial points connected to each other, a center point of the three spatial points, and a normal vector of a plane formed by the three spatial points; Inputting the plurality of face structure features into the posture feature extraction network to obtain the posture features through the posture feature extraction network.

4. The skeletal pose prediction method of claim 2, wherein, The bone posture prediction model further comprises a geometric feature extraction network; The step of extracting the geometric features of the target bone corresponding to each of the views comprises the following steps: Extracting target region images comprising the target bone from each of the perspective image samples; Inputting the target region images into the geometric feature extraction network to obtain predicted density features and predicted surface region attention features corresponding to each of the views through the geometric feature extraction network.

5. The skeletal pose prediction method of claim 4, wherein, The geometric feature extraction network is further configured to extract predicted contour mask features of each of the views according to the target region images; After the step of extracting the geometric features of the target bone corresponding to each of the views, the method further comprises the following steps: According to the predicted contour mask feature, the predicted density feature, and the predicted surface region attention feature, a geometric feature loss of each of the view angles is calculated respectively; According to the geometric feature losses of the plurality of view angles, a multi-view geometric feature loss is calculated; The geometric feature extraction network is updated according to the multi-view geometric feature loss.

6. The skeletal pose prediction method of claim 4, wherein, The step of outputting, by the pose prediction network, a predicted pose matrix of the target skeleton corresponding to each of the view angles includes: The predicted density features and the predicted surface region attention features of the plurality of view angles are fused to obtain a two-dimensional fusion feature; The two-dimensional fusion feature and the pose feature are fused to obtain a target fusion feature; According to the target fusion feature, a predicted pose matrix of the target skeleton corresponding to each of the view angles is determined.

7. The skeletal pose prediction method of claim 6, wherein, The step of determining the predicted pose matrix of the target skeleton corresponding to each of the view angles includes: The target fusion feature is input into a full connection module in the pose prediction network to obtain predicted transformation information of the target skeleton corresponding to each of the view angles, the predicted transformation information including a predicted rotation parameter and a predicted translation parameter of the target skeleton in an imaging coordinate system corresponding to the view angle; According to a space straight line formed by each of the predicted translation parameters and a corresponding ray light source, intersection points of a plurality of the space straight lines are determined; The intersection points are respectively converted into a ray light source coordinate system corresponding to each of the view angles to obtain corresponding target predicted translation parameters; Based on the predicted rotation parameter and the target predicted translation parameter, the predicted pose matrix of each of the view angles is determined respectively.

8. The skeletal pose prediction method of claim 7, wherein, The step of determining the intersection points of the plurality of space straight lines includes: A target coordinate system is constructed according to the ray light sources of at least two of the view angles and straight lines formed by the ray light sources to corresponding ray detectors; A conversion matrix between the target coordinate system and a ray light source coordinate system corresponding to each of the view angles is respectively obtained; Based on the conversion matrix, a space position of each of the predicted translation parameters and a corresponding ray light source in the target coordinate system is respectively obtained; According to each of the predicted translation parameters and the space position of the corresponding ray light source in the target coordinate system, a plurality of space straight lines are determined; Intersection points of the plurality of space straight lines are determined.

9. The skeletal pose prediction method of claim 8, wherein, The sample data includes perspective image samples of the target skeleton in at least two of the view angles in surgery and three-dimensional image data samples of the target skeleton before surgery; The step of outputting, by the pose prediction network, a predicted pose matrix of the target skeleton corresponding to each of the view angles includes: A frontal geometric feature, a lateral geometric feature, and a pose feature are input into the pose prediction network to obtain a frontal predicted pose matrix and a lateral predicted pose matrix output by the pose prediction network; The pose prediction loss includes a multi-view space conversion loss and a 6D pose prediction loss; The multi-view space conversion loss is represented as: ； wherein, is the multi-view spatial conversion loss for the target skeleton, is a first projection point of the target skeleton in the frontal radiographic light source coordinate system, is a second projection point of the target skeleton in the lateral radiographic light source coordinate system; The 6D pose prediction loss is represented as: ； ； wherein, is a frontal pose prediction loss, is a lateral pose prediction loss, is a frontal-lateral combined pose prediction loss, is a point cloud data sample of a target skeleton, is a spatial position of an edge voxel in the point cloud data sample, is a transformation matrix for transforming a spatial position in the target coordinate system into a frontal radiographic source coordinate system, is a transformation matrix for transforming a spatial position in the target coordinate system into a lateral radiographic source coordinate system, is a frontal standard pose matrix, is the frontal predicted pose matrix, is a lateral standard pose matrix, is the lateral predicted pose matrix, is an identity matrix.

10. A skeletal pose prediction apparatus, characterized by, The skeleton pose prediction device includes: a data acquisition module, configured to acquire perspective images of a target bone in at least two views in surgery and three-dimensional image data of the target bone before surgery; a data preprocessing module, configured to obtain point cloud data of the target bone according to the three-dimensional image data; a bone posture prediction module, configured to input the perspective images and the point cloud data into a pre-trained bone posture prediction model, the bone posture prediction model being configured to extract geometric features of the target bone corresponding to each of the views according to the perspective images, extract posture features of the target bone according to the point cloud data, fuse the geometric features and the posture features, and output a predicted posture matrix of the target bone corresponding to each of the views according to the fused features.

11. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 9.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 9.

Images