Bone posture prediction method and device, computer equipment and storage medium

Abstract

The invention relates to a skeleton posture prediction method and device, computer equipment and a storage medium. The skeleton posture prediction method comprises the steps that perspective images of a target skeleton in at least two view angles in an operation and three-dimensional image data of the target skeleton before the operation are acquired; obtaining point cloud data of the target skeleton according to the three-dimensional image data; and inputting the perspective image and the point cloud data into a skeleton attitude prediction model obtained by pre-training, fusing features extracted from the perspective image and the point cloud data, and outputting a prediction attitude matrix of the target skeleton corresponding to each view angle according to the fused features. According to the method, the attitude features extracted from the three-dimensional image data of the skeleton before the operation are taken as geometric guidance, and the attitude of the skeleton can be predicted based on the perspective images of the skeleton in the operation at multiple visual angles, so that the problem of insufficient depth information of single-visual-angle attitude prediction is solved, and the attitude prediction precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of skeletal posture prediction, and in particular to a skeletal posture prediction method, apparatus, computer equipment and storage medium. Background Art

[0002] When aligning 2D and 3D images of bones, there are large differences in the bone postures of different patients due to factors such as surgical posture, patient position, individual differences among patients, and image quality, making it difficult to ensure the accuracy of the alignment.

[0003] Traditional skeleton posture prediction methods are based on skeleton images from a single perspective, which is inaccurate for skeleton posture prediction. The main problems are:

[0004] Different skeletons have differences in 3D structures, so they are easily affected by occlusion and perspective changes and have poor robustness.

[0005] The bone image from a single perspective is not clear enough, the bone 3D texture is missing, and it cannot provide accurate depth information. Summary of the invention

[0006] In order to solve the above technical problems, the present application provides a skeleton posture prediction method, apparatus, computer device and storage medium.

[0007] In a first aspect, the present application provides a skeleton posture prediction method, the skeleton posture prediction method comprising:

[0008] Acquire perspective images of the target bone from at least two viewing angles during surgery, as well as three-dimensional image data of the target bone before surgery;

[0009] Obtaining point cloud data of the target bone according to the three-dimensional image data;

[0010] The perspective image and the point cloud data are input into a pre-trained skeletal posture prediction model, which fuses the features extracted from the perspective image and the point cloud data respectively, and outputs the predicted posture matrix of the target skeleton corresponding to each of the viewing angles based on the fused features.

[0011] In one embodiment, the skeletal posture prediction model includes a posture prediction network;

[0012] Before the step of inputting the perspective image and the point cloud data into the pre-trained skeleton posture prediction model, the method further includes:

[0013] Acquire sample data, wherein the sample data includes perspective image samples of the target bone during surgery at least two of the viewing angles, and three-dimensional image data samples of the target bone before surgery;

[0014] Obtaining a point cloud data sample of the target bone according to the three-dimensional image data sample;

[0015] Extracting, according to the perspective image samples, geometric features of the target bones corresponding to each of the perspectives;

[0016] Extracting the posture features of the target skeleton according to the point cloud data samples;

[0017] Inputting the geometric features and the posture features into the posture prediction network, and outputting the predicted posture matrix of the target skeleton corresponding to each of the viewing angles through the posture prediction network;

[0018] Calculating a posture prediction loss according to an error between at least one of the predicted posture matrices and a standard posture matrix of the target bone;

[0019] The posture prediction network is updated according to the posture prediction loss.

[0020] In one embodiment, the skeleton posture prediction model further includes a posture feature extraction network;

[0021] The step of extracting the posture feature of the target bone according to the point cloud data sample comprises:

[0022] Constructing a plurality of surface structure features according to the point cloud data sample, the surface structure features comprising three space points connected to each other, a center point of the three space points and a normal vector of a plane, the plane being constituted by the three space points;

[0023] A plurality of the face structure features are input into the posture feature extraction network, and the posture feature is obtained through the posture feature extraction network.

[0024] In one embodiment, the skeletal posture prediction model further includes a geometric feature extraction network;

[0025] The geometric features of the target skeleton corresponding to each of the viewing angles respectively extracted include:

[0026] Extracting a target area image including the target bone from each of the perspective image samples respectively;

[0027] The target area image is input into the geometric feature extraction network, and the predicted density feature and the predicted surface area attention feature corresponding to each of the viewing angles are obtained through the geometric feature extraction network.

[0028] In one embodiment, the geometric feature extraction network is further used to extract predicted contour mask features of each of the viewing angles according to the target area image;

[0029] After the step of respectively extracting the geometric features of the target skeleton corresponding to each of the viewing angles, the method further includes:

[0030] According to the predicted contour mask feature, the predicted density feature and the predicted surface area attention feature, respectively calculating the geometric feature loss of each of the viewing angles;

[0031] Calculating a multi-viewing angle geometric feature loss according to the geometric feature losses of the multiple viewing angles;

[0032] The geometric feature extraction network is updated according to the multi-view geometric feature loss.

[0033] In one embodiment, the step of outputting the predicted posture matrix of the target skeleton corresponding to each of the viewing angles through the posture prediction network includes:

[0034] Fusing the predicted density features and the predicted surface area attention features of the multiple perspectives to obtain a two-dimensional fused feature;

[0035] Fusing the two-dimensional fusion feature with the posture feature to obtain a target fusion feature;

[0036] According to the target fusion feature, a predicted posture matrix of the target skeleton corresponding to each of the viewing angles is determined.

[0037] In one embodiment, the step of determining the predicted posture matrix of the target skeleton corresponding to each of the viewing angles comprises:

[0038] Input the target fusion feature into the fully connected module in the posture prediction network to obtain the predicted transformation information of the target skeleton corresponding to each of the viewing angles, wherein the predicted transformation information includes a predicted rotation parameter and a predicted translation parameter of the target skeleton in the imaging coordinate system corresponding to the viewing angle;

[0039] Determine the intersection points of a plurality of the spatial straight lines according to the spatial straight lines formed by each of the predicted translation parameters and the corresponding ray light sources;

[0040] The intersection points are respectively converted into the ray light source coordinate system corresponding to each of the viewing angles to obtain corresponding target prediction translation parameters;

[0041] Based on the predicted rotation parameters and the target predicted translation parameters, the predicted posture matrix of each of the viewing angles is determined respectively.

[0042] In one embodiment, the step of determining the intersection points of the plurality of spatial straight lines comprises:

[0043] Constructing a target coordinate system according to the ray light sources of at least two of the viewing angles and the straight lines formed from the ray light sources to the corresponding ray detectors;

[0044] Respectively obtain the conversion matrix between the target coordinate system and the ray source coordinate system corresponding to each of the viewing angles;

[0045] Based on the transformation matrix, obtaining each predicted translation parameter and the corresponding spatial position of the ray light source in the target coordinate system respectively;

[0046] Determine a plurality of spatial straight lines according to each of the predicted translation parameters and the spatial position of the corresponding ray light source in the target coordinate system;

[0047] The intersection points of the plurality of spatial straight lines are determined.

[0048] In one embodiment, the sample data includes perspective image samples of the target bone during surgery at least two of the viewing angles, and three-dimensional image data samples of the target bone before surgery;

[0049] The step of outputting the predicted posture matrix of the target skeleton corresponding to each of the viewing angles through the posture prediction network comprises:

[0050] Inputting the anteroposterior geometric features, the lateral geometric features and the posture features into the posture prediction network to obtain the anteroposterior prediction posture matrix and the lateral prediction posture matrix output by the posture prediction network;

[0051] The posture prediction loss includes multi-view space conversion loss and 6D posture prediction loss;

[0052] The multi-view spatial conversion loss is expressed as:

[0053] loss t =||pointA-pointB||

[0054] Among them, loss t is the multi-view space conversion loss, pointA is the first projection point of the target bone in the orthogonal ray light source coordinate system, and pointB is the second projection point of the target bone in the lateral ray light source coordinate system;

[0055] The 6D pose prediction loss is expressed as:

[0056]

[0057] in, Predict loss for upright posture, Predict loss for side posture, is the joint posture prediction loss of the frontal and lateral positions, M is the point cloud data sample of the target skeleton, p is the spatial position of the edge voxel in the point cloud data sample, is a transformation matrix for transforming the spatial position in the target coordinate system into the orthogonal ray source coordinate system, is a transformation matrix for transforming the spatial position in the target coordinate system into the side ray source coordinate system, is the standard posture matrix, is the positive prediction posture matrix, is the standard posture matrix of the lateral position, is the lateral prediction posture matrix, and E is the unit matrix.

[0058] In a second aspect, the present application provides a skeleton posture prediction device, the device comprising:

[0059] A data acquisition module, used to acquire perspective images of the target bone from at least two viewing angles during surgery, and three-dimensional image data of the target bone before surgery;

[0060] A data preprocessing module, used for obtaining point cloud data of the target bone according to the three-dimensional image data;

[0061] The skeleton posture prediction module is used to input the perspective image and the point cloud data into a pre-trained skeleton posture prediction model. The skeleton posture prediction model fuses the features extracted from the perspective image and the point cloud data respectively, and outputs the predicted posture matrix of the target skeleton corresponding to each of the viewing angles based on the fused features.

[0062] In a third aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the skeletal posture prediction method described in the first aspect is implemented.

[0063] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the skeletal posture prediction method described in the first aspect is implemented.

[0064] On the basis of conforming to the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present application.

[0065] The above-mentioned skeletal posture prediction method, device, computer equipment and storage medium can achieve the following beneficial effects: extracting geometric features from perspective images obtained from at least two perspectives, extracting posture features from the three-dimensional image data of the skeleton before surgery, using the posture features as geometric guides to fuse the geometric features of multiple perspectives, and predicting the posture of the skeleton based on the fused features, which can solve the problem of insufficient depth information when predicting posture based on skeletal images from a single perspective, and improve the accuracy of posture prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a flow chart of a training process of a skeleton posture prediction model in one embodiment;

[0067] Figure 2 is a schematic diagram of a target coordinate system in one embodiment;

[0068] Figure 3 A schematic diagram of a flow chart of a skeleton posture prediction method in one embodiment;

[0069] Figure 4 A schematic diagram of a module of a skeleton posture prediction device in another embodiment;

[0070] Figure 5 is a first internal structure diagram of a computer device in one embodiment;

[0071] Figure 6 A second internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] It should be noted that the diagram provided in the present embodiment only illustrates the basic concept of the present application in a schematic manner, so the diagram only shows the components related to the present application rather than drawing according to the number, shape and size of the components during actual implementation. The form, quantity and ratio of each component during actual implementation can be a random change, and its component layout form may also be more complicated. The structure, proportion, size, etc. illustrated by the accompanying diagram of this specification are only used to match the content disclosed in the specification, so that people familiar with this technology can understand and read, and are not used to limit the restrictive conditions that can be implemented in this application, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size, without affecting the effect that can be produced by this application and the purpose that can be achieved, should still fall within the scope that the technical content disclosed in this application can cover. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are also only for the convenience of narration, rather than for limiting the scope that can be implemented in this application. The change or adjustment of its relative relationship, without substantial change of technical content, should also be regarded as the scope that can be implemented in this application.

[0074] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the text does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0075] As shown in this document, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural, unless the context clearly indicates an exception. Generally speaking, the terms "include" and "comprise" only indicate that the steps and elements that have been clearly identified are included, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0076] The definitions of inclusion in this document, such as the terms "having", "may have", "including" or "may include" used herein, indicate the existence of the corresponding functions, operations, elements, etc. herein, and do not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms "including" or "having" used herein indicate the existence of the features, numbers, steps, operations, elements, components, or a combination thereof described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, components, or a combination thereof.

[0077] In the embodiments of the present application, prefixes such as "first" and "second" are used only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0078] The present application provides a skeleton posture prediction method, comprising:

[0079] Acquire perspective images of the target bone from at least two viewing angles during surgery, as well as three-dimensional image data of the target bone before surgery;

[0080] Obtaining point cloud data of the target bone according to the three-dimensional image data;

[0081] The perspective image and point cloud data are input into the pre-trained skeleton posture prediction model, which fuses the features extracted from the perspective image and point cloud data respectively, and outputs the predicted posture matrix of the target skeleton corresponding to each viewing angle based on the fused features.

[0082] The target bones include but are not limited to vertebrae such as the lumbar vertebrae and cervical vertebrae, as well as bone blocks such as the femur, tibia and patella.

[0083] The present application uses the geometric features of the target bone extracted from at least two perspectives during surgery, and fuses them with the posture features obtained based on the three-dimensional image data of the target bone before surgery, to predict the posture information of the target bone in the perspective space of different perspectives. Based on the predicted posture information, it is possible to achieve registration from the preoperative three-dimensional image data of the target bone to the intraoperative two-dimensional image, so as to reduce the influence of different patients and different body positions on the registration accuracy, thereby improving the registration accuracy.

[0084] The predicted pose matrices of the target skeleton corresponding to each view actually express the same pose of the target skeleton. The predicted pose matrices of different viewpoints are expressed based on the coordinate system corresponding to the viewpoint. If the predicted pose matrices of the target skeleton corresponding to each viewpoint are converted to the same coordinate system for expression, the two are theoretically the same.

[0085] The skeletal posture prediction model includes a posture prediction network. Before using the skeletal posture prediction model, the posture prediction network can be iteratively trained using several sets of sample data, and the prediction loss is calculated based on the predicted posture matrix output by the skeletal posture prediction model and the standard posture matrix, and the training is stopped until the prediction loss reaches the expected range.

[0086] First, obtain several groups of sample data, and divide them into training sets and test sets according to the proportion. The specific proportion can be set and adjusted according to actual needs.

[0087] Each set of sample data includes perspective image samples of the target bone during surgery from at least two viewing angles, and three-dimensional image data samples of the target bone before surgery.

[0088] Then, the initial skeleton posture prediction model is trained using the sample data in the training set. The specific training process includes steps S101-S107:

[0089] S101. Obtain sample data.

[0090] S102: Obtain a point cloud data sample of the target skeleton according to the three-dimensional image data sample.

[0091] In one embodiment, step S102 includes:

[0092] S1021, segmenting the three-dimensional image data according to the contour of the target bone to obtain segmentation data;

[0093] S1022. Obtain a point cloud data sample of the target skeleton according to the segmentation data.

[0094] The principle of obtaining the point cloud data and point cloud data samples of the target bone is the same. First, the contour of the target bone in the three-dimensional image data is annotated to obtain the annotated data. Then, the target bone is segmented from the three-dimensional image data sample according to the annotated data to obtain the segmented data. Finally, the point cloud data sample of the target bone is obtained according to the segmented data.

[0095] The edge voxel set of point cloud data can be expressed as:

[0096] P={p1,p2,…,p n},n∈N *

[0097] Among them, p i =[x i ,y i ,z i ,1],i={1,2,…,n},p i represents the spatial position of the ith edge voxel, x i ,y i 、z i Respectively represent the x-axis, y-axis, and z-axis coordinates of the ith edge voxel, p i The last element of is 1, which means the type of edge voxel is point; if it is 0, it means the type of edge voxel is vector.

[0098] Since the size and shape of the target bones in different sample data are different, the number of edge voxels contained in the point cloud data obtained from the contour data is usually different. The farthest point sampling method can be used to ensure that the selected neighbor nodes are distributed globally, ensure the effective capture of the global features of the point cloud data, and convert the point cloud data of the target bones in different sample data into structured data with the same number of edge voxels.

[0099] The edge voxel set of the point cloud data after sampling by the farthest point sampling function can be expressed as:

[0100]

[0101] Among them, F fps is the farthest point sampling function, and n1 is the expected number of edge voxels, that is, the number of edge voxels after sampling by the farthest point sampling function.

[0102] S103: Extracting geometric features of the target skeleton corresponding to each viewing angle according to the perspective image samples.

[0103] In one embodiment, the skeleton posture prediction model also includes a geometric feature extraction network. Accordingly, step S103 includes:

[0104] S1031, extracting a target area image including a target bone from each perspective image sample;

[0105] S1032, input the target area image into a geometric feature extraction network, and obtain the predicted density feature and the predicted surface area attention feature corresponding to each viewing angle through the geometric feature extraction network. The predicted density feature includes the predicted 2D projection density feature and the predicted 3D space density feature.

[0106] That is, the geometric features include predicted density features and predicted surface area attention features.

[0107] In another embodiment, the geometric feature extraction network is further used to extract predicted contour mask features of each viewing angle according to the target area image, that is, the geometric features also include predicted contour mask features.

[0108] Accordingly, after step S1032, the method further includes:

[0109] S1033, calculating the geometric feature loss of each perspective respectively according to the predicted contour mask feature, the predicted density feature and the predicted surface area attention feature;

[0110] S1034. Calculate a multi-view geometric feature loss according to the geometric feature losses of the multiple viewpoints;

[0111] S1035. Update the geometric feature extraction network according to the multi-view geometric feature loss.

[0112] The multi-view geometric feature loss can be the sum of the geometric feature losses of multiple views. The multi-view geometric feature loss can be expressed as:

[0113]

[0114] The geometric feature loss of the s-th view can be expressed as:

[0115]

[0116] in, represents the geometric feature loss of the s-th view, Represents the standard contour mask feature of the target bone at the s-th view, Represents the predicted contour mask feature of the target skeleton at the s-th view, represents the predicted density feature of the target bone at the s-th view, represents the standard density feature of the target bone at the s-th perspective, CE represents the cross entropy loss function, is the predicted surface area attention feature of the s-th view, is the predicted surface area attention standard feature of the s-th perspective, and Θ represents the feature multiplication.

[0117] For example, see Figure 1 The sample data includes frontal perspective image samples and lateral perspective image samples of the target bone during surgery, as well as three-dimensional image data samples of the target bone before surgery.

[0118] Take the extraction of geometric features from frontal perspective image samples and lateral perspective image samples as an example:

[0119] The area corresponding to the target bone in the orthophoto image is taken as the region of interest (ROI), and it is scaled to a first target region image of 256×256 size. The first target region image is input into the ResNet network, and the orthophoto prediction contour mask feature, orthophoto prediction 2D projection density feature, orthophoto prediction 3D spatial density feature, and orthophoto prediction surface area attention feature can be extracted through the ResNet network.

[0120] Among them, the orthogonal prediction 2D projection density feature and the orthogonal prediction 3D space density are combined into the orthogonal prediction density feature, which is used to characterize the density correspondence of the target bone from the 2D projection to the 3D space under the orthogonal perspective. The orthogonal prediction surface area attention feature, as an additional fuzzy perception supervision signal, implicitly characterizes the orthogonal symmetry of the target bone, which can improve the prediction accuracy of the target bone in the orthogonal perspective space. That is, the orthogonal geometric features include the orthogonal prediction density feature and the orthogonal prediction surface area attention feature.

[0121] Similarly, the area corresponding to the target bone in the lateral perspective image is taken as the region of interest (ROI), and it is scaled to a second target area image of 256×256 size. The second target area image is input into the ResNet network, and the lateral prediction contour mask feature, lateral prediction 2D projection density feature, lateral prediction 3D spatial density feature and lateral prediction surface area attention feature are extracted through the ResNet network.

[0122] Among them, the lateral prediction 2D projection density feature and the lateral prediction 3D space density feature are combined into the lateral prediction density feature, which is used to characterize the density correspondence of the target bone from the 2D projection to the 3D space under the lateral perspective. That is, the lateral geometric feature includes the lateral prediction density feature and the lateral prediction surface area attention feature.

[0123] Correspondingly, the geometric feature regression loss includes the frontal geometric feature loss and the lateral geometric feature loss. The geometric feature regression loss can be expressed as:

[0124]

[0125] in, is the positive geometric feature loss, It is the loss of lateral geometric features. is the standard contour mask feature. is the positive prediction density feature, is the standard density characteristic of the positive position, To predict the contour mask feature in the positive position, To predict the surface area attention features for the positive position, It is the standard surface area attention feature of the positive position. is the standard contour mask feature of the side position, is the density feature for lateral prediction, is the lateral standard density feature, For side prediction contour mask features, To predict surface area attention features for laterality, It is the attention feature of the lateral standard surface area.

[0126] S104, extracting posture features of the target skeleton according to the point cloud data samples.

[0127] In one embodiment, the skeleton posture prediction model also includes a posture feature extraction network. Accordingly, step S104 includes:

[0128] S1041. Constructing a plurality of surface structure features according to the point cloud data sample, the surface structure features including three space points connected to each other, the center point of the three space points, and a normal vector of a plane, wherein the plane is composed of the three space points;

[0129] S1042. Input a plurality of face structure features into a posture feature extraction network, and obtain posture features through the posture feature extraction network.

[0130] The point cloud data sample includes multiple spatial points, and each spatial point has coordinate information in three-dimensional space.

[0131] Each spatial point can be mapped to a node of a graph structure. However, the nodes in the graph structure can only characterize the spatial structural features of the target bone, and cannot effectively characterize its spatial posture. Therefore, in this embodiment, a triangular face structure is used instead of a spatial point as a node input into a posture feature extraction network to obtain the posture features of the target bone.

[0132] Exemplarily, a plurality of surface structure features may be obtained by sampling a plurality of spatial points in a point cloud data sample using a surface structure feature sampling function.

[0133] Several surface structural features can be expressed as:

[0134]

[0135] Among them, F mesh is the surface structure feature sampling function, m is the number of spatial points in the point cloud data sample, is the mth surface structural feature, are three spatial points connected to each other in the mth surface structure feature, is the center point of the three spatial points in the mth surface structure feature, is the unit normal vector in the mth surface structure feature.

[0136] On the basis of the GCN network, by using triangular surface structures instead of spatial points as the surface nodes of the input posture feature extraction network, not only the number of nodes can be reduced, but also the topological structure of the point cloud data can be better maintained through the connection relationship of the triangular surface structure.

[0137] For a face-based graph neural network Face-GCN, its adjacency matrix A represents the connection relationship between face nodes. If face node i and face node j are connected by an edge, then A ij =1, otherwise A ij =0.

[0138] Assume that the adjacency matrix A is used to update the face node features. The specific update rules are as follows:

[0139]

[0140] Among them, H l is the surface node feature of the lth layer, σ is the activation function, I is the unit matrix, Add the adjacency matrix of self-loops to the adjacency matrix. for The degree matrix, W l is the weight matrix of the lth layer.

[0141] Through multi-layer graph convolution operations, the representation of face nodes can be gradually updated to obtain multi-scale feature information. The face node feature representation of each layer is obtained by the output of the previous layer through graph convolution operations. Therefore, after the above multi-layer graph convolution operations, the deep spatial features of the target skeleton, that is, the posture features, can be extracted.

[0142] S105, inputting the geometric features and posture features into a posture prediction network, and outputting the predicted posture matrix of the target skeleton corresponding to each viewing angle through the posture prediction network.

[0143] In one embodiment, step S105 includes:

[0144] S1051, fusing the predicted density features and the predicted surface area attention features of multiple perspectives to obtain a two-dimensional fused feature;

[0145] S1052, fusing the two-dimensional fusion feature with the posture feature to obtain a target fusion feature;

[0146] S1053. Determine the predicted posture matrix of the target skeleton corresponding to each viewpoint according to the target fusion feature.

[0147] Wherein, step S1053 specifically includes:

[0148] S201, inputting the target fusion feature into a fully connected module in a posture prediction network to obtain predicted transformation information of the target skeleton corresponding to each view angle, wherein the predicted transformation information includes predicted rotation parameters and predicted translation parameters of the target skeleton in an imaging coordinate system corresponding to the view angle;

[0149] S202, determining the intersection points of multiple space lines according to the space lines formed by each predicted translation parameter and the corresponding ray light source;

[0150] S203, converting the intersection points into the ray light source coordinate system corresponding to each viewing angle to obtain the corresponding target predicted translation parameters;

[0151] S204: Determine the predicted posture matrix of each viewing angle based on the predicted rotation parameter and the target predicted translation parameter.

[0152] Exemplarily, the frontal prediction density feature, the frontal prediction surface area attention feature, the lateral prediction density feature and the lateral prediction surface area attention feature are input into a posture prediction network, such as a Patch-PnP network, to obtain a frontal prediction posture matrix and a lateral prediction posture matrix. The Patch-PnP network includes a 2D convolution module and a fully connected module.

[0153] The 2D convolution module is used to fuse the frontal prediction density feature, the frontal prediction surface area attention feature, the lateral prediction density feature and the lateral surface area attention feature to obtain a two-dimensional fusion feature. The two-dimensional fusion feature can be expressed as:

[0154]

[0155] Among them, F cat is the feature fusion operation, is the positive prediction density feature, To predict the surface area attention features for the positive position, is the density feature for lateral prediction, Predicting surface area attentional features for laterality.

[0156] The two-dimensional fusion features are sequentially convolved and flattened, and then fused with the posture features of the target skeleton to obtain the target fusion features.

[0157] The target fusion feature can be expressed as:

[0158]

[0159] Among them, F flat For leveling operation, F conv is the convolution operation.

[0160] The target fusion features are input into the fully connected module of the Patch-PnP network, such as a multi-layer perceptron (MLP), to first obtain the predicted transformation information of the target skeleton corresponding to the frontal view and the side view.

[0161] The prediction transformation information can be expressed as:

[0162]

[0163] in, To predict the transformation information, F MLP is a multi-layer perceptron, R f is the positive prediction rotation matrix, R l is the side prediction rotation matrix, T f is the first predicted translation parameter of the target bone in the imaging coordinate system corresponding to the positive view, T l is a second predicted translation parameter in the imaging coordinate system corresponding to the lateral viewing angle.

[0164] Taking the X-ray image of the target bone as an example, the target bone needs to be placed between the X-ray light source and the X-ray detector. When the X-ray passes through the target bone, tissues of different densities and thicknesses absorb the X-rays to varying degrees. The remaining X-rays are absorbed by the X-ray detector, and imaging can be performed based on the remaining X-rays absorbed by the X-ray detector to obtain an X-ray image. Tissues with higher density (such as bones) absorb more X-rays and thus appear as bright areas on the image; while tissues with lower density (such as muscle and fat) absorb less X-rays and thus appear as dark areas on the image.

[0165] Since the fully connected modules of the Patch-PnP network directly output the first predicted translation parameters and the second predicted translation parameters in the imaging coordinate system, in order to determine the spatial straight line between the predicted translation parameters and the corresponding ray light source, the first predicted translation parameters and the second predicted translation parameters in the predicted transformation information need to be converted into a unified coordinate system.

[0166] In one embodiment, step S202 specifically includes S2021-S2024:

[0167] S2021. Construct a target coordinate system;

[0168] S2022, respectively obtaining the conversion matrix between the target coordinate system and the ray source coordinate system corresponding to each viewing angle;

[0169] S2022. Based on the transformation matrix, obtain each predicted translation parameter and the spatial position of the corresponding ray light source in the target coordinate system;

[0170] S2023, determining a plurality of spatial straight lines according to each predicted translation parameter and the spatial position of the corresponding ray light source in the target coordinate system;

[0171] S2024. Determine the intersection points of multiple spatial straight lines.

[0172] In step S2021, the X-axis and Y-axis of the target coordinate system may be determined by straight lines formed from ray light sources of any two viewing angles to corresponding ray detectors.

[0173] For example, see Figure 2 , the ray source of the normal viewing angle is the normal ray source The ray detector corresponding to the orthogonal ray source is the orthogonal ray detector. Similarly, the ray source of the side view is the side ray source. The ray detector corresponding to the side ray source is the side ray detector. The straight line formed by the positive X-ray detector is the X-axis. The straight line formed by the lateral X-ray detector is the Y axis, and the intersection point O between the two straight lines is int Construct the target coordinate system for the origin. The coordinate axis information of the target coordinate system can be expressed as:

[0174]

[0175] The coordinate system with the orthogonal ray source as the origin is the orthogonal ray source coordinate system. According to the target coordinate system and the orthogonal ray source coordinate system, the first transformation matrix can be obtained: Used to transform the spatial position in the target coordinate system to the orthographic ray source coordinate system.

[0176] Similarly, the coordinate system with the side ray source as the origin is the side ray source coordinate system. According to the target coordinate system and the side ray source coordinate system, the second transformation matrix can be obtained: Used to transform the spatial position in the target coordinate system into the side ray source coordinate system.

[0177] In step S2023, the spatial positions of the predicted translation parameters in the corresponding ray source coordinate system are first determined, and then the spatial positions of the predicted translation parameters and the corresponding ray source in the target coordinate system are determined.

[0178] According to the distance dis between the positive ray source and the positive ray detector f , and the distance dis between the side ray source and the side ray detector l , you can determine:

[0179]

[0180] in, is the spatial position of the first predicted translation parameter in the orthogonal ray source coordinate system, is the spatial position of the second predicted translation parameter in the lateral ray light source coordinate system.

[0181] According to the spatial position of the predicted translation parameter in the corresponding ray source coordinate system, the predicted translation parameter and the spatial position of the corresponding ray source in the target coordinate system are determined respectively.

[0182] For example, the first predicted translation parameter and the spatial position of the orthogonal ray source in the target coordinate system can be expressed as:

[0183]

[0184] in, is the spatial position of the first predicted translation parameter in the target coordinate system, is the spatial position of the orthogonal ray light source in the orthogonal ray light source coordinate system, is the spatial position of the orthographic ray light source in the target coordinate system.

[0185] Similarly, the second predicted translation parameter and the spatial position of the side ray light source in the target coordinate system can be expressed as:

[0186]

[0187] in, is the spatial position of the second predicted translation parameter in the target coordinate system, is the spatial position of the side ray source in the side ray source coordinate system, is the spatial position of the side ray light source in the side ray light source coordinate system.

[0188] Based on the above and In the target coordinate system, two space lines L1 and L2 can be obtained, which are expressed as:

[0189]

[0190] By solving the homogeneous linear equations, we can get the (pseudo) intersection of two spatial lines

[0191] Finally, through the above first transformation matrix and second transformation matrix, we can get the intersection point By converting the target coordinate space system into the orthogonal ray source coordinate system, we can obtain the target prediction translation parameters of the orthogonal perspective, that is, the orthogonal prediction translation parameters And the intersection By converting the target coordinate system into the side ray source coordinate system, the target predicted translation parameters of the side view can be obtained, that is, the predicted translation parameters

[0192]

[0193] According to the intersection The orthogonal prediction translation parameters and the orthogonal prediction rotation matrix in the orthogonal ray source coordinate system can be used to construct the orthogonal prediction attitude matrix:

[0194]

[0195] Similarly, according to the intersection The lateral prediction translation parameters and lateral prediction rotation matrix in the lateral ray source coordinate system can be used to construct the lateral prediction attitude matrix:

[0196]

[0197] S106. Calculate a posture prediction loss according to an error between at least one predicted posture matrix and a standard posture matrix of the target skeleton.

[0198] Among them, the posture prediction loss includes multi-view spatial conversion loss and 6D posture prediction loss.

[0199] (1) Multi-view spatial conversion loss

[0200] In order to regress the target prediction translation parameters in the prediction posture matrix, the target bone is regarded as a mass point with the initial position at the origin. The three-dimensional translation of the target bone is the target prediction translation parameter of the matrix.

[0201] Model training is performed for perspective image samples and three-dimensional image data samples using two viewing angles. For example, based on the above-mentioned frontal perspective image samples, lateral perspective image samples and three-dimensional image data samples, the predicted translation parameter T is obtained. f 、T l The T obtained by regression can be f 、T l Approximately, the first projection point A of the particle in the coordinate system of the orthogonal ray source, and T l It is approximated as the second projection point B of the particle on the lateral ray source coordinate system. However, the point A and point B obtained by regression are not strictly obtained by projecting from one point. Only when the distance between point A and point B is 0, the condition of being projected from one particle is satisfied. At this time, according to T f 、T l Calculate the target prediction translation parameters of the matrix, that is, there is a dual-view space conversion loss loss t , loss t It can be expressed as:

[0202] loss t =||pointA-pointB||

[0203] Multi-view spatial conversion loss of three or more views, including multiple dual-view spatial conversion losses. tThe dual-view spatial conversion loss between each two perspectives is calculated separately, and the multi-view spatial conversion loss is obtained by adding up multiple dual-view spatial conversion losses.

[0204] (2) 6D pose prediction loss

[0205] In addition to the parameterization of rotation and translation, the choice of loss function is also crucial for 6D pose optimization. Most works do not directly use the rotation matrix and translation parameters, but adopt a variant of the point matching loss based on the ADD (Average Distance of Keypoints) (-S) metric to couple the rotation and translation estimates.

[0206] Among them, the ADD metric is a metric used to evaluate the accuracy of 3D object pose estimation. It is widely used in robotics, computer vision and other fields, especially in evaluating the effectiveness of object pose estimation algorithms. ADD-S (Symmetric ADD) is an improved version of ADD, suitable for pose estimation evaluation of symmetrical objects.

[0207] In this embodiment, a new variant of the disentangled 6D pose loss can be adopted by separately supervising the rotation matrix R, the scale-invariant 2D object center (δx, δy), and the translation parameter δz.

[0208] Model training is performed for perspective image samples and 3D image data samples using two viewing angles. For example, based on the above-mentioned frontal perspective image samples, lateral perspective image samples and 3D image data samples, the frontal predicted posture matrix and the lateral predicted posture matrix are obtained. The 6D posture prediction loss can be expressed as:

[0209]

[0210]

[0211] in, Predict loss for upright posture, Predict loss for side posture, is the joint posture prediction loss of the frontal and lateral positions, M is the point cloud data sample of the target skeleton, p is the spatial position of the edge voxel in the point cloud data sample, is the standard posture matrix, is the standard lateral posture matrix, and E is the unit matrix.

[0212] For the 6D posture prediction loss of three or more perspectives, it is necessary to first calculate the posture prediction loss of each perspective separately, and then calculate the joint posture prediction loss between every two perspectives separately. The posture prediction loss of all perspectives and the joint posture prediction loss are added together to obtain the multi-perspective 6D posture prediction loss.

[0213] S107. Update the posture prediction network according to the posture prediction loss.

[0214] By continuously adjusting the parameters in the skeletal posture prediction model, such as network weights, biases, hyperparameters, learning rate, batch size, etc., the skeletal posture prediction model can learn the patterns and features in the sample data to reduce the prediction loss of the model.

[0215] Finally, the performance of the trained skeletal posture prediction model is evaluated through the sample data in the test set, and various performance indicators can be calculated as performance evaluation results, such as prediction loss, prediction accuracy, recall rate, etc. According to the performance evaluation results of the skeletal posture prediction model, you can choose to continue training or put it into actual use.

[0216] After the skeleton posture prediction model is trained according to the above training process, the testing process begins.

[0217] Exemplarily, the sample data includes an AP fluoroscopic image sample and a lateral fluoroscopic image sample of the target bone during surgery, and a three-dimensional image data sample of the target bone before surgery.

[0218] Input the N sample data in the test set into the skeleton posture prediction model to obtain the output positive prediction posture matrix and the lateral predicted posture matrix

[0219] According to the first transformation matrix The predicted posture matrix can be transformed into the target coordinate system to obtain According to the second transformation matrix The lateral predicted posture matrix can be converted to the target coordinate system to obtain

[0220]

[0221] In theory, and The gap between them is very small, and you can choose either one of them or use both to evaluate the performance of the trained skeleton posture prediction model.

[0222] in, and The predicted rotation parameters are included in The predicted translation parameter It can be expressed as:

[0223]

[0224] The mean absolute error between the predicted rotation parameters and the standard rotation parameters can be calculated using the Euler angles:

[0225]

[0226] And the error between the predicted translation parameters and the standard translation parameters can be calculated by the Euclidean distance:

[0227]

[0228] Among them, Euler represents the function of converting the rotation matrix into Euler angles. The standard rotation parameter corresponding to the i-th sample data, is the standard translation parameter corresponding to the i-th sample data.

[0229] The standard rotation parameters and the standard translation parameters are obtained according to the standard posture matrix. The standard posture matrix can be a gold standard obtained by a mechanism interactive registration method. For example, the mechanism interactive registration of vertebrae usually involves the precise registration of vertebrae in medical images of different modalities. In this process, the key feature points or regions of the vertebrae are selected manually or semi-automatically, and finally the standard posture matrix of the vertebrae is obtained. This is common knowledge in the art and will not be repeated here.

[0230] Based on the skeleton posture prediction model trained by the above process, the steps of skeleton posture prediction are as follows: Figure 3 As shown, including:

[0231] Acquire anteroposterior and lateral perspective images of the target bone during surgery, as well as three-dimensional image data of the target bone before surgery;

[0232] Performing point cloud processing on the three-dimensional image data samples to obtain point cloud data of the target bone;

[0233] The anteroposterior perspective image, lateral perspective image and point cloud data are input into the bone posture prediction model to extract the anteroposterior geometric features, lateral geometric features and posture features respectively. The anteroposterior geometric features, lateral geometric features and posture features are input into the posture prediction network, first fused to obtain the target fusion features, and then posture prediction is performed based on the target fusion features to obtain the anteroposterior prediction posture matrix and lateral prediction posture matrix of the target bone.

[0234] It should be understood that although Figure 1 and Figure 3The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0235] Corresponding to the above-mentioned skeleton posture prediction method, the present application also provides a skeleton posture prediction device. In one embodiment, Figure 4 As shown, the skeleton posture prediction device includes:

[0236] A data acquisition module 401 is used to acquire perspective images of the target bone from at least two viewing angles during surgery, as well as three-dimensional image data of the target bone before surgery;

[0237] A data preprocessing module 402 is used to obtain point cloud data of the target bone according to the three-dimensional image data;

[0238] The skeleton posture prediction module 403 is used to input the perspective image and point cloud data into a pre-trained skeleton posture prediction model. The skeleton posture prediction model fuses the features extracted from the perspective image and point cloud data respectively, and outputs the predicted posture matrix of the target skeleton corresponding to each viewing angle based on the fused features.

[0239] The specific limitation about the skeleton posture prediction device can refer to the limitation for the skeleton posture prediction method in the above text, which will not be repeated here. Each module in the above-mentioned skeleton posture prediction device can be realized in whole or in part by software, hardware and combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls the operation corresponding to each of the above modules.

[0240] The present application also provides a computer device. In one embodiment, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the skeletal posture prediction method in the above embodiment are implemented.

[0241] In one embodiment, the computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store skeletal posture prediction data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a skeletal posture prediction method is implemented.

[0242] In one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a skeletal posture prediction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0243] Those skilled in the art will understand that Figure 4 and Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0244] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the skeleton posture prediction method in the above embodiment are implemented.

[0245] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0246] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0247] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A skeleton posture prediction method, characterized in that: The skeleton posture prediction method comprises: Acquire perspective images of the target bone from at least two viewing angles during surgery, as well as three-dimensional image data of the target bone before surgery; Obtaining point cloud data of the target bone according to the three-dimensional image data; The perspective image and the point cloud data are input into a pre-trained skeletal posture prediction model, which fuses the features extracted from the perspective image and the point cloud data respectively, and outputs the predicted posture matrix of the target skeleton corresponding to each of the viewing angles based on the fused features.

2. The skeleton posture prediction method according to claim 1, characterized in that: The skeleton posture prediction model includes a posture prediction network; Before the step of inputting the perspective image and the point cloud data into the pre-trained skeleton posture prediction model, the method further includes: Acquire sample data, wherein the sample data includes perspective image samples of the target bone during surgery at least two of the viewing angles, and three-dimensional image data samples of the target bone before surgery; Obtaining a point cloud data sample of the target bone according to the three-dimensional image data sample; Extracting, according to the perspective image samples, geometric features of the target bones corresponding to each of the perspectives; Extracting the posture features of the target skeleton according to the point cloud data samples; Inputting the geometric features and the posture features into the posture prediction network, and outputting the predicted posture matrix of the target skeleton corresponding to each of the viewing angles through the posture prediction network; Calculating a posture prediction loss according to an error between at least one of the predicted posture matrices and a standard posture matrix of the target bone; The posture prediction network is updated according to the posture prediction loss.

3. The skeleton posture prediction method according to claim 2, characterized in that: The skeleton posture prediction model also includes a posture feature extraction network; The step of extracting the posture feature of the target bone according to the point cloud data sample comprises: Constructing a plurality of surface structure features according to the point cloud data sample, the surface structure features comprising three space points connected to each other, a center point of the three space points and a normal vector of a plane, the plane being constituted by the three space points; A plurality of the face structure features are input into the posture feature extraction network, and the posture feature is obtained through the posture feature extraction network.

4. The skeleton posture prediction method according to claim 2, characterized in that: The skeleton posture prediction model also includes a geometric feature extraction network; The geometric features of the target skeleton corresponding to each of the viewing angles respectively extracted include: Extracting a target area image including the target bone from each of the perspective image samples respectively; The target area image is input into the geometric feature extraction network, and the predicted density feature and the predicted surface area attention feature corresponding to each of the viewing angles are obtained through the geometric feature extraction network.

5. The skeleton posture prediction method according to claim 4, characterized in that: The geometric feature extraction network is also used to extract predicted contour mask features of each of the viewing angles according to the target area image; After the step of respectively extracting the geometric features of the target skeleton corresponding to each of the viewing angles, the method further includes: According to the predicted contour mask feature, the predicted density feature and the predicted surface area attention feature, respectively calculating the geometric feature loss of each of the viewing angles; Calculating a multi-viewing angle geometric feature loss according to the geometric feature losses of the multiple viewing angles; The geometric feature extraction network is updated according to the multi-view geometric feature loss.

6. The skeleton posture prediction method according to claim 2, characterized in that: The step of outputting the predicted posture matrix of the target skeleton corresponding to each of the viewing angles through the posture prediction network comprises: Fusing the predicted density features and the predicted surface area attention features of the multiple perspectives to obtain a two-dimensional fused feature; Fusing the two-dimensional fusion feature with the posture feature to obtain a target fusion feature; According to the target fusion feature, a predicted posture matrix of the target skeleton corresponding to each of the viewing angles is determined.

7. The skeleton posture prediction method according to claim 6, characterized in that: The step of determining the predicted posture matrix of the target skeleton corresponding to each of the viewing angles comprises: Input the target fusion feature into the fully connected module in the posture prediction network to obtain the predicted transformation information of the target skeleton corresponding to each of the viewing angles, wherein the predicted transformation information includes a predicted rotation parameter and a predicted translation parameter of the target skeleton in the imaging coordinate system corresponding to the viewing angle; Determine the intersection points of a plurality of spatial straight lines according to the spatial straight lines formed by each of the predicted translation parameters and the corresponding ray light sources; The intersection points are respectively converted into the ray light source coordinate system corresponding to each of the viewing angles to obtain corresponding target prediction translation parameters; Based on the predicted rotation parameters and the target predicted translation parameters, the predicted posture matrix of each of the viewing angles is determined respectively.

8. The method for predicting skeleton posture according to claim 7, characterized in that: The step of determining the intersection points of the plurality of space straight lines comprises: Constructing a target coordinate system according to the ray light sources of at least two of the viewing angles and the straight lines formed from the ray light sources to the corresponding ray detectors; Respectively obtain the conversion matrix between the target coordinate system and the ray source coordinate system corresponding to each of the viewing angles; Based on the transformation matrix, obtaining each predicted translation parameter and the corresponding spatial position of the ray light source in the target coordinate system respectively; Determine a plurality of spatial straight lines according to each of the predicted translation parameters and the spatial position of the corresponding ray light source in the target coordinate system; The intersection points of the plurality of spatial straight lines are determined.

9. The skeleton posture prediction method according to claim 7, characterized in that: The sample data includes perspective image samples of the target bone at least two viewing angles during surgery, and three-dimensional image data samples of the target bone before surgery; The step of outputting the predicted posture matrix of the target skeleton corresponding to each of the viewing angles through the posture prediction network comprises: Inputting the anteroposterior geometric features, the lateral geometric features and the posture features into the posture prediction network to obtain the anteroposterior prediction posture matrix and the lateral prediction posture matrix output by the posture prediction network; The posture prediction loss includes multi-view space conversion loss and 6D posture prediction loss; The multi-view spatial conversion loss is expressed as: loss t =||pointA-pointB|| Among them, loss t is the multi-view space conversion loss, pointA is the first projection point of the target bone in the orthogonal ray light source coordinate system, and pointB is the second projection point of the target bone in the lateral ray light source coordinate system; The 6D pose prediction loss is expressed as: in, Predict loss for upright posture, Predict loss for side posture, is the joint posture prediction loss of the frontal and lateral positions, M is the point cloud data sample of the target skeleton, p is the spatial position of the edge voxel in the point cloud data sample, is a transformation matrix for transforming the spatial position in the target coordinate system into the orthogonal ray source coordinate system, is a transformation matrix for transforming the spatial position in the target coordinate system into the side ray source coordinate system, is the standard posture matrix, is the positive prediction posture matrix, is the standard posture matrix of the lateral position, is the lateral prediction posture matrix, and E is the unit matrix.

10. A skeleton posture prediction device, characterized in that: The skeleton posture prediction device comprises: A data acquisition module, used to acquire perspective images of the target bone from at least two viewing angles during surgery, and three-dimensional image data of the target bone before surgery; A data preprocessing module, used for obtaining point cloud data of the target bone according to the three-dimensional image data; The skeleton posture prediction module is used to input the perspective image and the point cloud data into a pre-trained skeleton posture prediction model. The skeleton posture prediction model fuses the features extracted from the perspective image and the point cloud data respectively, and outputs the predicted posture matrix of the target skeleton corresponding to each of the viewing angles based on the fused features.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images