Three-dimensional reconstruction method and device and surgical robot system

Abstract

The invention relates to a three-dimensional reconstruction method and device and a surgical robot system. The method comprises the following steps: acquiring a two-dimensional image obtained by shooting a target part from two mutually vertical directions; determining a virtual bounding box of the target part according to the two-dimensional image; determining a transformation relation between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part; and performing transformation processing on the standard three-dimensional model according to the transformation relation to obtain a three-dimensional reconstruction result of the target part. By adopting the method, the three-dimensional reconstruction of the target part is not limited by two-dimensional image calibration and synchronous exposure, the three-dimensional reconstruction is accurately carried out under the condition that the two-dimensional image is not calibrated and is not synchronously exposed, and the accuracy of a three-dimensional reconstruction result is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional reconstruction method, device and surgical robot system. Background Art

[0002] With the development of medical imaging technology, X-ray-based three-dimensional reconstruction technology has emerged. This technology uses two X-ray two-dimensional images (for example, frontal and lateral views) obtained by photographing the target part from different angles to reconstruct the target part in three dimensions.

[0003] However, when two X-ray 2D images are not calibrated or are not exposed synchronously, direct 3D reconstruction can easily lead to inaccurate 3D reconstruction results. For example, when there is no calibration, the 3D spatial relationship between the anteroposterior and lateral views is unknown, and the spatial transformation matrix between the two cannot be accurately determined, which can easily lead to inaccurate 3D reconstruction results; when exposed asynchronously, due to breathing, position movement, etc., the anteroposterior and lateral views of the target part may not correspond to each other, which in turn affects the accuracy of the 3D reconstruction results.

[0004] Therefore, the current three-dimensional reconstruction technology has the problem of low accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a three-dimensional reconstruction method, device, computer equipment, computer readable storage medium, computer program product and surgical robot system with higher accuracy to address the above technical problems.

[0006] In a first aspect, the present application provides a three-dimensional reconstruction method. The method comprises:

[0007] Acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other;

[0008] Determining a virtual bounding box of the target part according to the two-dimensional image;

[0009] Determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part;

[0010] The standard three-dimensional model is transformed according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0011] In a second aspect, the present application also provides a three-dimensional reconstruction device. The device comprises:

[0012] An image acquisition module is used to acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other;

[0013] An image determination module, used to determine a virtual bounding box of the target part according to the two-dimensional image;

[0014] A relationship determination module, used to determine the transformation relationship between the standard bounding box of the target part and the virtual bounding box; the standard bounding box is the bounding box corresponding to the standard three-dimensional model of the target part;

[0015] The three-dimensional reconstruction module is used to transform the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0016] In a third aspect, the present application also provides a surgical robot system. The system includes a processor and a surgical robot;

[0017] The processor is used to obtain a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other, determine a virtual bounding box of the target part according to the two-dimensional image, determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box, and transform the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part; the standard bounding box is a bounding box corresponding to the standard three-dimensional model of the target part;

[0018] The surgical robot is used to perform a predetermined surgical operation on the target part according to the three-dimensional reconstruction result.

[0019] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0020] Acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other;

[0021] Determining a virtual bounding box of the target part according to the two-dimensional image;

[0022] Determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part;

[0023] The standard three-dimensional model is transformed according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0024] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0025] Acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other;

[0026] Determining a virtual bounding box of the target part according to the two-dimensional image;

[0027] Determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part;

[0028] The standard three-dimensional model is transformed according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0029] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0030] Acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other;

[0031] Determining a virtual bounding box of the target part according to the two-dimensional image;

[0032] Determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part;

[0033] The standard three-dimensional model is transformed according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0034] The above-mentioned 3D reconstruction method, apparatus, computer equipment, storage medium, computer program product and surgical robot system obtain a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other, determine the virtual bounding box of the target part according to the two-dimensional image, determine the transformation relationship between the standard bounding box of the target part and the virtual bounding box, transform the standard 3D model according to the transformation relationship, and obtain the 3D reconstruction result of the target part; the standard 3D model of the target part can be obtained, the standard bounding box of the standard 3D model can be determined, and the virtual bounding box of the target part can be determined according to the two-dimensional image, and the coordinate transformation relationship between the virtual bounding box and the standard bounding box can be used to transform the standard 3D model, and the standard 3D model after the coordinate transformation is used as the 3D reconstruction result of the target part, so that the 3D reconstruction of the target part is not limited by the calibration and synchronous exposure of the 2D image, and the 3D reconstruction can be accurately performed even when the 2D image is not calibrated and the exposure is asynchronous, thereby improving the accuracy of the 3D reconstruction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic flow chart of a three-dimensional reconstruction method in one embodiment;

[0036] Figure 2 is a schematic diagram of a virtual bounding box in one embodiment;

[0037] Figure 3 is a schematic diagram of an original positive angle point in one embodiment;

[0038] Figure 4 A schematic diagram of aligning an original lateral angle point to an original anteroposterior angle point in one embodiment;

[0039] Figure 5 is a schematic diagram of a three-dimensional image of a vertebra in one embodiment;

[0040] Figure 6 is a schematic diagram of an original three-dimensional model in one embodiment;

[0041] Figure 7 is a schematic diagram of an original bounding box in one embodiment;

[0042] Figure 8 A schematic diagram of determining a first transformation relationship in an embodiment;

[0043] Fig. 9 is a schematic diagram of an original bounding box in another embodiment;

[0044] Fig.10 A schematic diagram of a three-dimensional reconstruction result of a human spine in one embodiment;

[0045] Fig.11 is a schematic flow chart of a vertebral reconstruction method based on X-rays in one embodiment;

[0046] Fig.12 is a schematic flow chart of a vertebral reconstruction method based on X-rays in another embodiment;

[0047] Fig.13 is a structural block diagram of a three-dimensional reconstruction device in one embodiment;

[0048] Fig.14 is a structural block diagram of a surgical robot system in one embodiment;

[0049] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In one embodiment, Figure 1 As shown, a 3D reconstruction method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0052] Step S110 , obtaining a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other.

[0053] The target part may be a part to be three-dimensionally reconstructed, for example, a vertebra in the human spine that needs to be three-dimensionally reconstructed.

[0054] The two-dimensional image may be two two-dimensional medical images, for example, an anteroposterior view and a lateral view taken by an X-ray fluoroscopy device.

[0055] In a specific implementation, for a target part that needs to be three-dimensionally reconstructed, it can be photographed from two directions perpendicular to each other to obtain two two-dimensional images, which are input into a terminal. The terminal obtains two two-dimensional images obtained by photographing the target part from directions perpendicular to each other.

[0056] For example, when a target vertebra in the human spine needs to be reconstructed in three dimensions, an X-ray fluoroscopy device can be used to take two shots from the front and side of the human body to obtain an AP and lateral view, respectively. Both the AP and lateral views contain the target vertebra. The AP and lateral views are input into the terminal, and the terminal obtains two two-dimensional images of the target vertebra.

[0057] Step S120: determining a virtual bounding box of the target part according to the two-dimensional image.

[0058] The virtual bounding box may be a bounding box determined according to a two-dimensional image of the target part, and the number of the virtual bounding boxes may be two.

[0059] In a specific implementation, the terminal can respectively determine the corner points of the target part from the two two-dimensional images, and project rays perpendicular to the corresponding two-dimensional images from the corner points to form a first prism and a second prism, and use the two planes corresponding to the second prism to intercept the first prism to obtain a virtual bounding box, and use the two planes corresponding to the first prism to intercept the second prism to obtain another virtual bounding box.

[0060] Figure 2 A schematic diagram of a virtual bounding box is provided. Figure 2The spatial position relationship between the AP and lateral films can be vertically intersecting. The vertices of the target vertebrae are extracted from the AP and lateral films, and the four corner points A, B, C, and D in the AP and the four corner points E, F, G, and H in the lateral film are obtained respectively. Four rays AI, BJ, CK, and DL perpendicular to the AP are projected from the four corner points of the AP. The quadrangular prism formed by AI, BJ, CK, and DL is defined as the first prism. The first prism is intercepted by the plane FNOG formed by the corner points F and G in the lateral film and the plane EJKH formed by the corner points E and H. A virtual bounding box is obtained, recorded as the first bounding box, and four rays EJ, FN, GO, and HK perpendicular to the lateral film are projected from the four corner points of the lateral film. The quadrangular prism formed by EJ, FN, GO, and HK is defined as the second prism. The second prism is intercepted by the plane ADLI formed by the corner points A and D in the orthographic film and the plane BCKJ formed by the corner points B and C, and another virtual bounding box is obtained, recorded as the second bounding box. The first bounding box and the second bounding box are used as virtual bounding boxes of the target part. Both the first bounding box and the second bounding box can be hexahedral structures.

[0061] Step S130, determining the transformation relationship between the standard bounding box of the target part and the virtual bounding box; the standard bounding box is the bounding box corresponding to the standard three-dimensional model of the target part.

[0062] The standard three-dimensional model may be a three-dimensional model of a target part located at a predetermined standard position and oriented toward a predetermined standard orientation.

[0063] The transformation relationship may be a transformation relationship of space coordinates, including but not limited to at least one of a rotation relationship, a translation relationship and a scaling relationship.

[0064] In a specific implementation, the terminal may predetermine a standard 3D model of the target part and determine a bounding box of the standard 3D model to obtain a standard bounding box of the target part. After determining a virtual bounding box, the terminal may determine a coordinate transformation relationship between the standard bounding box and the virtual bounding box.

[0065] For example, the original three-dimensional model of the target vertebra is obtained, and the points on the original three-dimensional model are represented by P_origin. According to the coordinate transformation matrix T1 between the original three-dimensional model and the standard three-dimensional model, the original three-dimensional model is moved to a predetermined standard position and rotated to a predetermined standard orientation to obtain the standard three-dimensional model of the target vertebra, which can be represented by P_origin*T1, where T1 includes a rotation matrix and a translation. The bounding box of the standard three-dimensional model is determined, and the bounding box is determined as the standard bounding box of the target vertebra. The standard bounding box can also be a hexahedral structure. After determining the first bounding box and the second bounding box of the target vertebra, the coordinate transformation matrix T2 and the scaling amount s of the virtual bounding box relative to the standard bounding box are determined according to the first bounding box and the second bounding box, where T2 includes a rotation matrix and a translation. The obtained T2 and s are determined as the transformation relationship between the standard bounding box and the virtual bounding box.

[0066] Step S140, transforming the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0067] In a specific implementation, since the transformation relationship between the standard bounding box and the virtual bounding box can be considered to be equivalent to the transformation relationship between the standard three-dimensional model and the three-dimensional reconstruction result, after determining the transformation relationship between the standard bounding box and the virtual bounding box, the standard three-dimensional model of the target part can be coordinate-transformed according to the transformation relationship to obtain the standard three-dimensional model after coordinate transformation, and the standard three-dimensional model after coordinate transformation can be determined as the three-dimensional reconstruction result of the target part.

[0068] For example, the coordinate transformation matrix T2 of the virtual bounding box relative to the standard bounding box and the scaling amount s are used to perform an affine transformation on the standard three-dimensional model P_origin*T1 of the target vertebra to obtain a three-dimensional reconstruction result P_trans=P_origin*T1*T2*s of the target vertebra.

[0069] The above-mentioned 3D reconstruction method obtains a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other, determines a virtual bounding box of the target part according to the two-dimensional image, determines a transformation relationship between a standard bounding box of the target part and the virtual bounding box, and transforms the standard 3D model according to the transformation relationship to obtain a 3D reconstruction result of the target part; the standard 3D model of the target part can be obtained, a standard bounding box of the standard 3D model can be determined, and a virtual bounding box of the target part can be determined according to the two-dimensional image, and a coordinate transformation relationship between the virtual bounding box and the standard bounding box can be used to transform the standard 3D model, and the standard 3D model after the coordinate transformation is used as the 3D reconstruction result of the target part, so that the 3D reconstruction of the target part is not restricted by the calibration and synchronous exposure of the 2D image, and the 3D reconstruction can be accurately performed even when the 2D image is not calibrated and the exposure is not synchronous, thereby improving the accuracy of the 3D reconstruction result.

[0070] In one embodiment, the two-dimensional image includes an orthotopic image and a lateral image, and the virtual bounding box includes a first bounding box and a second bounding box; the above step S120 may specifically include: determining the orthotopic angle point of the target part according to the orthotopic image, and determining the lateral angle point of the target part according to the lateral image; determining a first prism formed by vertically projecting from the plane where the orthotopic image is located with the orthotopic angle point as a reference, and intercepting the first prism with the first plane and the second plane to obtain a first bounding box; the first plane and the second plane pass through the two lateral angle points respectively, the first plane and the second plane are both perpendicular to the plane where the lateral image is located, and the first plane and the second plane both intersect with the first prism; determining a second prism formed by vertically projecting from the plane where the lateral image is located with the lateral angle point as a reference, and intercepting the second prism with the third plane and the fourth plane to obtain a second bounding box; the third plane and the fourth plane pass through the two orthotopic angle points respectively, the third plane and the fourth plane are both perpendicular to the plane where the orthotopic image is located, and the third plane and the fourth plane both intersect with the second prism.

[0071] The anteroposterior image may be a two-dimensional anteroposterior film obtained by photographing the target part from an anteroposterior position, and the lateral image may be a two-dimensional lateral film obtained by photographing the target part from a lateral position.

[0072] The anteroposterior angle point may be an angle point determined based on the anteroposterior radiograph, and the lateral angle point may be an angle point determined based on the lateral radiograph.

[0073] The first bounding box may be a virtual bounding box determined according to the frontal slice, and the second bounding box may be a virtual bounding box determined according to the lateral slice.

[0074] The first plane (or the second plane) may be a plane passing through two lateral angle points, perpendicular to the plane where the lateral image is located, and intersecting with the first prism. The third plane (or the fourth plane) may be a plane passing through two orthogonal angle points, perpendicular to the plane where the orthogonal image is located, and intersecting with the second prism.

[0075] In the specific implementation, according to Figure 2 , make the anteroposterior film and the lateral film intersect vertically in three-dimensional space, determine the anteroposterior angle points A, B, C, D of the target part according to the anteroposterior film, determine the lateral angle points E, F, G, H of the target part according to the lateral film, project rays AI, BJ, CK, DL perpendicular to the anteroposterior film from the anteroposterior angle points, AI, BJ, CK, DL form a first prism, use the first plane FNOG and the second plane EJKH to intercept the first prism, and obtain hexahedrons M1, N1, O1 , P1, I1, J1, K1, L1, recorded as the first bounding box, project rays EJ, FN, GO, HK perpendicular to the lateral film from the lateral angle point, EJ, FN, GO, HK form a second prism, use the third plane ADLI and the fourth plane BCKJ to intercept the second prism, and get the hexahedron M2, N2, O2, P2, I2, J2, K2, L2, recorded as the second bounding box, and use the first bounding box and the second bounding box as virtual bounding boxes.

[0076] In this embodiment, the orthogonal angle point of the target part is determined according to the orthogonal image, and the lateral angle point of the target part is determined according to the lateral image; a first prism formed by vertically projecting from the plane where the orthogonal image is located with the orthogonal angle point as a reference is determined, and the first prism is intercepted by the first plane and the second plane to obtain a first bounding box; the first plane and the second plane pass through two lateral angle points respectively, the first plane and the second plane are both perpendicular to the plane where the lateral image is located, and the first plane and the second plane both intersect with the first prism; a second prism formed by vertically projecting from the plane where the lateral image is located with the lateral angle point as a reference is determined, and the second prism is intercepted by the third plane and the fourth plane to obtain a second bounding box; the third plane and the fourth plane pass through two orthogonal angle points respectively, the third plane and the fourth plane are both perpendicular to the plane where the orthogonal image is located, and the third plane and the fourth plane both intersect with the second prism, so that the virtual bounding box of the target part can be quickly determined using the two two-dimensional images of the orthogonal film and the lateral film, thereby improving the efficiency of three-dimensional reconstruction of the target part.

[0077] In one embodiment, the above-mentioned steps of determining the anteroposterior corner point of the target part according to the anteroposterior image, and determining the lateral corner point of the target part according to the lateral image, may specifically include: obtaining the original anteroposterior corner point of the target part in the anteroposterior image, and the original lateral corner point of the target part in the lateral image; aligning the original lateral corner points according to the original anteroposterior corner points to obtain the aligned lateral corner points; and performing back-projection processing on the original anteroposterior corner points and the aligned lateral corner points, respectively, to obtain the anteroposterior corner points and the lateral corner points.

[0078] The original anteroposterior angle point may be the vertex of the target part in the anteroposterior image, and the original lateral angle point may be the vertex of the target part in the lateral image.

[0079] The alignment operation may be an operation for removing the influence of position movement and aligning the original lateral angle point with the original frontal angle point.

[0080] In a specific implementation, the vertices of the target part can be extracted from the anteroposterior image to obtain a two-dimensional original anteroposterior angle point, and the vertices of the target part can be extracted from the lateral image to obtain a two-dimensional original lateral angle point. The original anteroposterior angle point is aligned with the center point of the aligned lateral angle point as a reference, so that the center point of the aligned lateral angle point matches the center point of the original anteroposterior angle point. The original anteroposterior angle point and the aligned lateral angle point are back-projected into three-dimensional space, the original anteroposterior angle point is transformed from two-dimensional coordinates to three-dimensional coordinates to obtain an anteroposterior angle point, and the aligned lateral angle point is transformed from two-dimensional coordinates to three-dimensional coordinates to obtain a lateral angle point.

[0081] For example, the vertices of the target vertebrae of the human spine are extracted from the anteroposterior and lateral views to obtain the original anteroposterior angle points A0, B0, C0, D0, and the original lateral angle points E0, F0, G0, H0. Figure 3 A schematic diagram of an original anteroposterior angle point is provided, wherein the original anteroposterior angle point and the original lateral angle point can be, but are not limited to, automatically extracted by a deep learning method.

[0082] Figure 4 A schematic diagram is provided for aligning the original lateral angle point to the original anteroposterior angle point. Figure 4 In the case of asynchronous exposure of the anteroposterior and lateral views, there may be a position shift between the two, resulting in the target vertebra in the anteroposterior view not corresponding to the target vertebra in the lateral view in the height direction (the direction of the head and feet of the human body), so it is necessary to align the original lateral angle point and the original anteroposterior angle point. Specifically, the center coordinates of the original anteroposterior angle point in the anteroposterior view can be calculated, and the original lateral angle point in the lateral view can be moved in the height direction to obtain the aligned lateral angle point, and the center coordinates of the aligned lateral angle point are the same as the center coordinates of the original anteroposterior angle point.

[0083] Afterwards, the two-dimensional original anteroposterior angle point and the aligned lateral angle point can be back-projected into three-dimensional space to obtain the anteroposterior angle point and the lateral angle point. The mapping relationship of the back-projection processing can be: assuming that the two-dimensional coordinates of the original anteroposterior angle point are (x1, y1), it is converted into three-dimensional coordinates (-x1, y1, 0), or (x1, -y1, 0) to obtain the anteroposterior angle point; assuming that the two-dimensional coordinates of the aligned lateral angle point are (x2, y2), it is converted into three-dimensional coordinates (0, y2, x2), or (0, -y2, -x2) to obtain the lateral angle point.

[0084] In this embodiment, the original anterior corner points of the target part in the anteroposterior image and the original lateral corner points of the target part in the lateral image are obtained; the original lateral corner points are aligned according to the original anterior corner points to obtain the aligned lateral corner points; the original anterior corner points and the aligned lateral corner points are back-projected to obtain the anteroposterior corner points and the lateral corner points, respectively. The original corner points of the target part can be aligned and back-projected to obtain relatively accurate anteroposterior corner points and lateral corner points, and three-dimensional reconstruction is performed to improve the accuracy of three-dimensional reconstruction.

[0085] In one embodiment, the above-mentioned step of aligning the original lateral angle points according to the original anteroposterior angle points to obtain the aligned posterior lateral angle points may specifically include: taking the center point of the original anteroposterior angle points as the reference center point; adjusting the original lateral angle points according to the reference center point to obtain the aligned posterior lateral angle points; and matching the center point of the aligned posterior lateral angle points with the reference center point.

[0086] The reference center point may be a reference point for the alignment operation.

[0087] In a specific implementation, the center point of the original frontal angle point can be determined as the reference center point of the alignment operation, and the original lateral angle point can be adjusted up and down according to the reference center point. During the adjustment process, the relative positions of the original lateral angle points remain unchanged, and the center point of the aligned lateral angle point obtained after adjustment coincides with the reference center point.

[0088] In this embodiment, the center point of the original anteroposterior corner point is used as the reference center point; the original lateral corner point is adjusted according to the reference center point to obtain the aligned lateral corner point; the center point of the aligned lateral corner point is matched with the reference center point, so that the anteroposterior corner point and the lateral corner point can be made consistent in the height direction, avoiding the inaccurate determination of the virtual bounding box caused by position movement during the shooting of the anteroposterior and lateral films, thereby further improving the accuracy of three-dimensional reconstruction.

[0089] In one embodiment, the above-mentioned steps of back-projecting the original anteroposterior angle point and the aligned lateral angle point to obtain the anteroposterior angle point and the lateral angle point may specifically include: back-projecting the original anteroposterior angle point into three-dimensional space to obtain the anteroposterior angle point, and back-projecting the aligned lateral angle point into three-dimensional space to obtain the lateral angle point.

[0090] In a specific implementation, the original anteroposterior angle point and the aligned lateral angle point can be back-projected into three-dimensional space, so that the original anteroposterior angle point and the aligned lateral angle point are transformed from two-dimensional coordinates to three-dimensional coordinates to obtain three-dimensional anteroposterior angle points and three-dimensional lateral angle points.

[0091] For example, by performing back projection, the two-dimensional coordinates (x1, y1) of the original anteroposterior angle point can be converted into three-dimensional coordinates (-x1, y1, 0) to obtain the anteroposterior angle point, and the two-dimensional coordinates (x2, y2) of the lateral angle point after alignment can be converted into three-dimensional coordinates (0, y2, x2) to obtain the lateral angle point; or, the two-dimensional coordinates (x1, y1) of the original anteroposterior angle point can be converted into three-dimensional coordinates (x1, -y1, 0) to obtain the anteroposterior angle point, and the two-dimensional coordinates (x2, y2) of the lateral angle point after alignment can be converted into three-dimensional coordinates (0, -y2, -x2) to obtain the lateral angle point.

[0092] In this embodiment, the original anteroposterior corner points are back-projected into three-dimensional space to obtain anteroposterior corner points, and the aligned lateral corner points are back-projected into three-dimensional space to obtain lateral corner points. The two-dimensional anteroposterior image and lateral image can be transformed into three dimensions, thereby improving the efficiency of three-dimensional reconstruction.

[0093] In one embodiment, before the above-mentioned step S130, it can also specifically include: obtaining the original three-dimensional model of the target part, and determining the original bounding box of the original three-dimensional model; determining the first transformation relationship corresponding to the original bounding box according to the normal vectors of each face of the original bounding box; the first transformation relationship is used to characterize the rotation relationship and translation relationship between the original bounding box and the standard bounding box; transforming the original bounding box according to the first transformation relationship to obtain the standard bounding box of the target part.

[0094] The original three-dimensional model may be an originally acquired three-dimensional model, for example, a three-dimensional model obtained by three-dimensionally reconstructing a CT image of the target part, or a three-dimensional model of the target part directly downloaded from the Internet.

[0095] The original bounding box may be the bounding box of the original three-dimensional model.

[0096] The standard bounding box may be a bounding box of a standard three-dimensional model.

[0097] In a specific implementation, the original three-dimensional image containing the target part can be segmented to obtain the original three-dimensional model of the target part, and the original bounding box of the original three-dimensional model can be determined. Then, the translation amount for moving the original bounding box to a predetermined standard position and the rotation matrix for rotating the original bounding box to a predetermined standard orientation can be determined. The product of the translation amount and the rotation matrix is ​​used as the first transformation relationship, and the coordinates of the original bounding box are transformed according to the first transformation relationship to obtain the standard bounding box of the target part.

[0098] Figure 5 A schematic diagram of a three-dimensional image of a vertebra is provided. Figure 5 The three-dimensional image of the human vertebra can be a CT image, or it can be any three-dimensional image of the human vertebra downloaded from the Internet. The position and orientation of each vertebra in the three-dimensional image may be inconsistent. It is necessary to perform translation and rotation operations on each vertebra to make it consistent with the standard position and standard orientation, thereby obtaining a standard three-dimensional model of each vertebra.

[0099] Figure 6 A schematic diagram of the original 3D model is provided. Figure 6 , the vertebral three-dimensional image can be segmented based on the deep learning method to obtain the original three-dimensional model of each vertebra. For example, the bone plate outside the transverse process and spinous process can be segmented.

[0100] Figure 7 A schematic diagram of the original bounding box is provided. Figure 7 , the directed bounding box corresponding to the bone plate can be calculated to obtain the original bounding box of the hexahedral structure.

[0101] Figure 8 A schematic diagram for determining the first transformation relationship is provided. Figure 8 , the first transformation matrix for converting the original bounding box to the standard bounding box can be calculated based on the 8 vertices of the original bounding box. Specifically, the first normal vector of the face formed by vertices 1, 2, and 4, the second normal vector of the face formed by vertices 5, 6, and 8 can be obtained, and the average value of the first normal vector and the second normal vector can be obtained as the direction vector of the x-axis of the rotation matrix; the third normal vector of the face formed by vertices 2, 6, and 8, and the fourth normal vector of the face formed by vertices 1, 5, and 7 can be obtained, and the average value of the third normal vector and the fourth normal vector can be obtained as the direction vector of the z-axis of the rotation matrix; according to the constraint that the three direction vectors of the rotation matrix are orthogonal to each other, the direction vector of the z-axis can be cross-multiplied with the direction vector of the x-axis to obtain the direction vector of the y-axis of the rotation matrix, thereby obtaining the rotation matrix. Afterwards, the center coordinates of the 8 vertices of the original bounding box can be used as the translation amount, and the product of the rotation matrix and the translation amount can be used as the first transformation relationship between the original three-dimensional model and the standard three-dimensional model, represented by T1.

[0102] Fig. 9 Another schematic diagram of the original bounding box is provided. In practical applications, it is also possible to manually select feature points in the vertebral 3D image without segmenting the vertebral 3D image, and obtain the original bounding box by converting the feature points. For example, according to Fig. 9 , since it is impossible to directly select the vertices of the original bounding box from the three-dimensional image, feature points 9 to 16 can be manually selected on the vertebrae in the three-dimensional image, corresponding to the most anterior, posterior, leftmost and rightmost points on the upper and lower end surfaces of the vertebral bone plates, respectively, which basically correspond to the midpoints of each edge in the upper and lower surfaces of the original bounding box. Then, the vertices 1 to 8 of the original bounding box can be determined based on feature points 9 to 16. For example, two straight lines 1-2 and 5-6 passing through feature points 10 and 13 and parallel to the diagonal line 9-14 can be drawn in the surface formed by 9, 10, 13, and 14, and two straight lines 1-5 and 2-6 passing through feature points 9 and 14 and parallel to the diagonal line 10-13 can be drawn, and the intersection of straight lines 1-2, 5-6, 1-5, and 2-6 is used as the upper vertex of the original bounding box; at 11, 12, 15, 16 In the formed surface, two straight lines 3-4 and 7-8 are drawn passing through the feature points 11 and 16 and parallel to the diagonal line 12-15, and two straight lines 3-7 and 4-8 are drawn passing through the feature points 12 and 15 and parallel to the diagonal line 11-16, and the intersection of the straight lines 3-4, 7-8, 3-7, and 4-8 is used as the lower vertex of the original bounding box; the original bounding box is determined according to the upper vertex and the lower vertex of the original bounding box, thereby reducing the complexity of determining the original bounding box and improving the efficiency of determining the original bounding box.

[0103] In this embodiment, the original three-dimensional model of the target part is obtained, and the original bounding box of the original three-dimensional model is determined; according to the normal vectors of each face of the original bounding box, the first transformation relationship corresponding to the original bounding box is determined; the first transformation relationship is used to characterize the rotation relationship and translation relationship between the original bounding box and the standard bounding box; the original bounding box is transformed according to the first transformation relationship to obtain the standard bounding box of the target part. By determining the standard bounding box, the target part can be three-dimensionally reconstructed using the standard three-dimensional model, thereby improving the accuracy of the three-dimensional reconstruction of the target part.

[0104] In one embodiment, the above step S130 may specifically include: determining the roll angle direction vector of the target part according to the first bounding box; determining the pitch angle direction vector of the target part according to the second bounding box; determining the center coordinates of the first bounding box and the second bounding box; obtaining the second transformation relationship between the virtual bounding box and the standard bounding box according to the roll angle direction vector, the pitch angle direction vector and the center coordinates; the second transformation relationship is used to characterize the rotation transformation and translation transformation between the virtual bounding box and the standard bounding box.

[0105] In a specific implementation, the x-axis direction vector of the first bounding box can be used as the direction vector of the roll angle of the target part, and the z-axis direction vector of the second bounding box can be used as the direction vector of the pitch angle of the target part. The x-axis direction vector and the z-axis direction vector are cross-producted to obtain the y-axis direction vector. According to the x-axis direction vector, the y-axis direction vector and the z-axis direction vector, the rotation matrix of the virtual bounding box relative to the standard bounding box is obtained. The translation amount of the virtual bounding box relative to the standard bounding box can also be determined according to the average value of the center coordinates of the first bounding box and the center coordinates of the second bounding box. The rotation matrix is ​​multiplied by the translation amount to obtain the second transformation relationship between the virtual bounding box and the standard bounding box. The second transformation relationship can be represented by T2.

[0106] In this embodiment, the roll angle direction vector of the target part is determined according to the first bounding box; the pitch angle direction vector of the target part is determined according to the second bounding box; the center coordinates of the first bounding box and the second bounding box are determined; according to the roll angle direction vector, the pitch angle direction vector and the center coordinates, a second transformation relationship between the virtual bounding box and the standard bounding box is obtained; the second transformation relationship is used to characterize the rotation transformation and translation transformation between the virtual bounding box and the standard bounding box, and the rotation amount and translation amount between the virtual bounding box and the standard bounding box can be determined, which is convenient for converting the standard three-dimensional model into the three-dimensional reconstruction result of the target part, thereby improving the efficiency of three-dimensional model reconstruction.

[0107] In one embodiment, the above step S130 may further specifically include: determining a first expansion amount of the first bounding box relative to the standard bounding box, and a second expansion amount of the second bounding box relative to the standard bounding box; obtaining a third transformation relationship between the virtual bounding box and the standard bounding box based on an average of the first expansion amount and the second expansion amount; the third transformation relationship is used to characterize the scaling relationship between the virtual bounding box and the standard bounding box.

[0108] In a specific implementation, the expansion and contraction amounts of the first bounding box and the second bounding box relative to the standard bounding box can be calculated respectively to obtain the first expansion and contraction amounts, and the average value of the first expansion and contraction amounts can be determined as the expansion and contraction amount of the virtual bounding box relative to the standard bounding box, that is, the third transformation relationship, which can be represented by s.

[0109] In practical applications, the calculation of the expansion and contraction amount can be obtained by iterating the Pugh transformation, and the calculation formula is:

[0110] argmin s，R，T ||sRp T +Tq T ||,

[0111] Among them, R, T, and s respectively represent the rotation, translation, and expansion of the first bounding box or the second bounding box relative to the standard bounding box. Only the expansion can be taken. p can be a point on the standard bounding box, and q can be a point on the first bounding box or the second bounding box.

[0112] In this embodiment, by determining a first expansion amount of the first bounding box relative to the standard bounding box, and a second expansion amount of the second bounding box relative to the standard bounding box; and according to the average value of the first expansion amount and the second expansion amount, a third transformation relationship between the virtual bounding box and the standard bounding box is obtained; the third transformation relationship is used to characterize the scaling relationship between the virtual bounding box and the standard bounding box, and the expansion amount between the virtual bounding box and the standard bounding box can be determined, so as to facilitate the conversion of the standard three-dimensional model into a three-dimensional reconstruction result of the target part, thereby improving the efficiency of three-dimensional model reconstruction.

[0113] In one embodiment, the above step S140 may specifically include: transforming the standard three-dimensional model according to the second transformation relationship and the third transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0114] In a specific implementation, each point in the standard three-dimensional model may be multiplied by the second transformation relationship and the third transformation relationship respectively to obtain a three-dimensional reconstruction result of the target part.

[0115] In practical applications, the point P_origin on the original three-dimensional model of the target vertebra can be affine transformed to obtain the point P_origin*T1 on the standard three-dimensional model, where T1 is the first transformation relationship between the original three-dimensional model and the standard three-dimensional model; the point P_origin*T1 on the standard three-dimensional model is radially transformed to obtain the three-dimensional reconstruction result of the target vertebra P_trans=P_origin*T1*T2*s, where T2 and s are the second transformation relationship and the third transformation relationship between the virtual bounding box and the standard bounding box, respectively. The above three-dimensional reconstruction method is used to perform three-dimensional reconstruction on each vertebra in the human spine in turn to obtain the three-dimensional reconstruction result of the human spine. Fig.10 A schematic diagram of the results of three-dimensional reconstruction of the human spine is provided.

[0116] It should be noted that P_origin*T1 can be calculated in advance and stored in a database. When 3D reconstruction is required, after T2 and s are determined, P_origin*T1 stored in the database is directly used to perform 3D reconstruction according to P_trans=P_origin*T1*T2*s.

[0117] In this embodiment, the standard three-dimensional model is transformed according to the second transformation relationship and the third transformation relationship to obtain the three-dimensional reconstruction result of the target part. The three-dimensional reconstruction can be directly performed on the standard three-dimensional model of the target part to obtain the three-dimensional reconstruction result, thereby improving the efficiency of the three-dimensional reconstruction.

[0118] In order to facilitate those skilled in the art to have a deeper understanding of the embodiments of the present application, a specific example will be described below.

[0119] The present application provides a method suitable for three-dimensional reconstruction of non-calibrated, non-synchronous exposure X-ray films. The physiological parameters of the Cobb angle (lateral curvature angle, used to assess the severity of scoliosis) corresponding to the reconstructed three-dimensional model are highly consistent with the Cobb angle corresponding to the X-ray film, with high accuracy, and do not require training processes such as deep learning, are not data-dependent, are simple to calculate, and are easy to implement.

[0120] Fig.11 A schematic diagram of the process of vertebral reconstruction based on X-ray film is provided. Fig.12 A schematic diagram of another method for vertebral reconstruction based on radiographs is provided. Fig.11 and 12 The two inputs of the vertebra reconstruction method based on X-rays are the original 3D model of the vertebra and two mutually perpendicular X-rays, and the output is the actual 3D model of the vertebra corresponding to the X-rays. The original 3D model is a 3D mesh model, which can be obtained by three-dimensional reconstruction of CT data or downloaded from the Internet. Figure 5 As shown, the position angle of each vertebra in the original 3D model may be inconsistent, and can be processed through steps S211-S213 to obtain a standard 3D model, which is stored in the database and used for 3D reconstruction of different X-rays, that is, the standard 3D model may not be changed each time a 3D reconstruction is performed. The X-ray film may be a 2D image based on X-rays such as DR (Digital Radiography) or small C, which includes the vertebral segments to be reconstructed, and may be a non-calibrated X-ray film or two X-ray films exposed asynchronously.

[0121] The vertebral reconstruction method based on X-rays may specifically include the following steps:

[0122] Step S210, processing the original 3D model to produce a standard 3D model, including:

[0123] Step S211, segment the original 3D model using deep learning or other methods to obtain the transverse process and the bone plate outside the spinous process, such as Figure 6 shown.

[0124] Step S212, calculate the directed bounding box corresponding to the bone plate, and obtain 8 vertices. Specifically, the three-dimensional points on the bone plate grid can be represented as a one-dimensional array (x, y, z), and the covariance matrix of the array can be obtained. The eigenvalues ​​and eigenvectors can be obtained by the Jacobi iteration method, wherein the eigenvectors constitute a column vector matrix; after the three-dimensional points on the grid are decentralized, they are multiplied by the column vector matrix for rotation; for the rotated point coordinates, the maximum and minimum values ​​in the three directions of x, y, and z are obtained to obtain the center coordinates and length of the directed bounding box; the center coordinates are multiplied by the inverse matrix of the column vector matrix to obtain the coordinate values ​​of the center coordinates in the original grid coordinate system of the bone plate, and the coordinate values ​​of the 8 vertices of the directed bounding box in the original grid coordinate system can be obtained in the same way.

[0125] Step S213, based on the eight vertices of the directed bounding box, calculate the transformation matrix T1 for rotating and translating the original 3D model to a standard orientation and standard position. Specifically, Figure 8 , take the first normal vector of the surface formed by points 1, 2, and 4, and the second normal vector of the surface formed by points 5, 6, and 8, and calculate the average of the first normal vector and the second normal vector as the direction vector of the x-axis; take the third normal vector of the surface formed by points 2, 6, and 8, and the fourth normal vector of the surface formed by points 1, 5, and 7, and calculate the average of the third normal vector and the fourth normal vector as the direction vector of the z-axis; since the three direction vectors of the rotation matrix are orthogonal, the z-axis direction vector can be cross-multiplied by the x-axis direction vector to obtain the y-axis direction vector, and the rotation matrix is ​​obtained according to the x-axis direction vector, the y-axis direction vector, and the z-axis direction vector, and the center coordinates of the 8 vertices of the directed bounding box are used as the translation, and the product of the rotation matrix and the translation is used as the transformation matrix T1.

[0126] Step S220, X-ray film processing step, includes:

[0127] Step S221, extracting corner points, extracting the vertebral vertices in the X-ray film. Each vertebral segment has 4 corner points in each X-ray film, and obtaining the coordinates P_corner_i (i=1, 2, 3, 4) of each corner point in the X-ray film image coordinate system, such as Figure 3 As shown in the figure, the extraction of corner points can be automated through a spine corner point extraction algorithm based on deep learning.

[0128] Step S222, align the corner points of the anteroposterior and lateral X-rays. In the case of asynchronous exposure, the person may move when taking two X-rays, resulting in each vertebra not corresponding to the other in the height direction (head-to-foot direction) of the two X-rays, and position alignment is required. Specifically, the center coordinates of the four corner points in each X-ray can be calculated as the center coordinates of the vertebra in the X-ray. The height direction values ​​of the vertebra center coordinates in the lateral film are replaced with the height direction values ​​of the vertebra center coordinates in the anteroposterior film based on the center coordinates of the vertebra in the anteroposterior film, such as Figure 4 shown.

[0129] Step S223, generates 3D coordinate points based on the back projection of the 2D anterior and lateral angle points. Specifically, the 2D coordinates (x, y) of the anteroposterior film can be converted to (-x, y, 0), and the 2D coordinates (x, y) of the lateral film can be converted to (0, y, x); the 2D coordinates (x, y) of the anteroposterior film can be converted to (x, -y, 0), and the 2D coordinates (x, y) of the lateral film can be converted to (0, -y, -x), etc.

[0130] Step S224, according to Figure 2 , project rays AI, BJ, CK, DL perpendicular to the plane of the orthotopic film from the four corner points A, B, C, D of the orthotopic film to obtain the first prism, and project rays EJ, FN, GO, HK perpendicular to the plane of the lateral film from the four corner points E, F, G, H of the lateral film to obtain the second prism; use the two vertical planes FNOG and EJKH corresponding to the four corner points E, F, G, H of the lateral film to intercept the first prism to obtain the first hexahedron, whose eight vertices are M1, N1, O1, P1, I1, J1, K1, L1; similarly, use the two vertical planes ADLI and BCKJ corresponding to the four corner points A, B, C, D of the orthotopic film to intercept the second prism to obtain the second hexahedron, whose eight vertices are M2, N2, O2, P2, I2, J2, K2, L2.

[0131] Step S225, take the x-axis direction vector of the first hexahedron obtained by projecting the frontal film as the direction vector of the roll angle corresponding to the vertebral body; take the z-axis direction vector of the second hexahedron obtained by projecting the lateral film as the direction vector of the pitch angle corresponding to the vertebral body. The direction vector calculation method is the same as step S213.

[0132] Step S226, according to the center coordinates of the first hexahedron and the second hexahedron, the direction vector of the roll angle and the direction vector of the pitch angle, a matrix T2 including rotation and translation can be obtained, and the calculation method is the same as step S213.

[0133] Step S227: According to the vertices of the first hexahedron and the second hexahedron and the eight vertices of the directed bounding box in step S213, two expansion amounts are obtained, and the average of the two expansion amounts is taken to obtain the final expansion amount s. The expansion amount can be calculated by iterating the Prototype Transform, and the specific formula is:

[0134] argmin s，R，T ||sRp T +Tq T ||,

[0135] Among them, R represents the rotation amount, T represents the translation amount, and s represents the expansion amount, and only the expansion amount is taken.

[0136] Step S230, actual model calculation, specifically, the above steps calculate the transformation matrix T1, transformation matrix T2, and expansion amount s. Among them, T1 is the transformation matrix when each vertebra model corresponds to the standard orientation, T2 is the transformation matrix from the standard orientation to the actual position of each vertebra, and s is the expansion amount between each vertebra model and the actual vertebra. Among them, T1 can be performed alone, that is, the standard 3D model transformed by T1 can be saved and used directly.

[0137] The 3D coordinates of each point P_origin of the original 3D model are affine transformed to obtain Fig.10 The actual 3D model shown is calculated using the formula P_trans=s*T2*T1*P_origin.

[0138] The above-mentioned vertebral reconstruction method based on X-rays matches the two-dimensional and three-dimensional vertebrae from the perspective of spatial geometric transformation, and improves the consistency between the three-dimensional model and the two-dimensional X-ray by decoupling the three-dimensional scaling, angle, and translation, including the separate projection of the roll angle and pitch angle of the 3D model corresponding to the X-ray. The high consistency between the corner points of the three-dimensional model and the corner points of the two-dimensional X-ray film ultimately ensures the accuracy of the physiological parameters such as the Cobb angle and position of the three-dimensional model.

[0139] Moreover, the automated production of the three-dimensional standard model uses the angles and positions of the segmented bone plates to represent the angles and positions of each vertebra, thereby simplifying the model while ensuring accuracy. The model is replaced with a simplified bounding box model to improve computational efficiency. At the same time, the same bounding box calculation method is used to ensure the consistency and accuracy of the reconstruction effect.

[0140] Furthermore, the X-ray vertebrae alignment method is applicable to non-calibrated X-rays and non-synchronously exposed X-ray data, and has no special requirements for internal and external calibration, exposure synchronization, etc. of the X-ray shooting equipment.

[0141] In addition, the above process can be automated without the need for interaction, and the calculation process is relatively mature, simple, reliable, and easy to implement, avoiding the restrictive dependence on deep learning methods.

[0142] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a 3D reconstruction device and a surgical robot system for implementing the 3D reconstruction method involved above. The implementation scheme for solving the problem provided by the device and the surgical robot system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more 3D reconstruction device embodiments provided below can refer to the limitations on the 3D reconstruction method above, and will not be repeated here.

[0144] In one embodiment, Fig.13 As shown, a three-dimensional reconstruction device is provided, including: an image acquisition module 310, an image determination module 320, a relationship determination module 330 and a three-dimensional reconstruction module 340, wherein:

[0145] An image acquisition module 310 is used to acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other;

[0146] An image determination module 320, configured to determine a virtual bounding box of the target part according to the two-dimensional image;

[0147] A relationship determination module 330 is used to determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part;

[0148] The three-dimensional reconstruction module 340 is used to transform the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

[0149] In one embodiment, the above-mentioned image determination module 320 is also used to determine the orthogonal angle point of the target part according to the orthogonal image, and to determine the lateral angle point of the target part according to the lateral image; determine a first prism formed by vertically projecting from the plane where the orthogonal image is located with the orthogonal angle point as a reference, and use the first plane and the second plane to intercept the first prism to obtain the first bounding box; the first plane and the second plane pass through the two orthogonal angle points respectively, the first plane and the second plane are both perpendicular to the plane where the lateral image is located, and the first plane and the second plane both intersect with the first prism; determine a second prism formed by vertically projecting from the plane where the lateral image is located with the lateral angle point as a reference, and use the third plane and the fourth plane to intercept the second prism to obtain the second bounding box; the third plane and the fourth plane pass through the two orthogonal angle points respectively, the third plane and the fourth plane are both perpendicular to the plane where the orthogonal image is located, and the third plane and the fourth plane both intersect with the second prism.

[0150] In one embodiment, the above-mentioned image determination module 320 is also used to obtain the original anterior angle point of the target part in the anterior image, and the original lateral angle point of the target part in the lateral image; align the original lateral angle points according to the original anterior angle points to obtain the aligned lateral angle points; and perform back-projection processing on the original anterior angle points and the aligned lateral angle points respectively to obtain the anterior angle points and the lateral angle points.

[0151] In one embodiment, the above-mentioned image determination module 320 is also used to use the center point of the original frontal angle point as the reference center point; according to the reference center point, adjust the original lateral angle point to obtain the aligned posterior lateral angle point; the center point of the aligned posterior lateral angle point matches the reference center point.

[0152] In one embodiment, the image determination module 320 is further used to back-project the original anteroposterior angle point into three-dimensional space to obtain the anteroposterior angle point, and to back-project the aligned lateral angle point into three-dimensional space to obtain the lateral angle point.

[0153] In one embodiment, the above three-dimensional reconstruction device further includes:

[0154] An original bounding box module, used to obtain an original three-dimensional model of the target part and determine an original bounding box of the original three-dimensional model;

[0155] A first transformation relationship module, used to determine a first transformation relationship corresponding to the original bounding box according to normal vectors of each face of the original bounding box; the first transformation relationship is used to characterize the rotation relationship and translation relationship between the original bounding box and the standard bounding box;

[0156] The standard bounding box module is used to transform the original bounding box according to the first transformation relationship to obtain the standard bounding box of the target part.

[0157] In one embodiment, the above-mentioned relationship determination module 330 is also used to determine the roll angle direction vector of the target part according to the first bounding box; determine the pitch angle direction vector of the target part according to the second bounding box; determine the center coordinates of the first bounding box and the second bounding box; obtain the second transformation relationship between the virtual bounding box and the standard bounding box according to the roll angle direction vector, the pitch angle direction vector and the center coordinates; the second transformation relationship is used to characterize the rotation transformation and translation transformation between the virtual bounding box and the standard bounding box.

[0158] In one embodiment, the relationship determination module 330 is further used to determine a first expansion amount of the first bounding box relative to the standard bounding box, and a second expansion amount of the second bounding box relative to the standard bounding box; a third transformation relationship between the virtual bounding box and the standard bounding box is obtained based on an average value of the first expansion amount and the second expansion amount; the third transformation relationship is used to characterize the scaling relationship between the virtual bounding box and the standard bounding box.

[0159] In one embodiment, the three-dimensional reconstruction module 340 is further used to transform the standard three-dimensional model according to the second transformation relationship and the third transformation relationship to obtain the three-dimensional reconstruction result of the target part.

[0160] In one embodiment, Fig.14 As shown, a surgical robot system is provided, including: a processor 410 and a surgical robot 420, wherein:

[0161] The processor 410 is used to obtain a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other, determine a virtual bounding box of the target part according to the two-dimensional image, determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box, and transform the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part; the standard bounding box is a bounding box corresponding to the standard three-dimensional model of the target part;

[0162] The surgical robot 420 is used to perform a predetermined operation on the target part according to the three-dimensional reconstruction result.

[0163] Each module in the above-mentioned three-dimensional reconstruction device and surgical robot system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0164] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a three-dimensional reconstruction method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0165] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0167] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0168] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0171] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other; Determining a virtual bounding box of the target part according to the two-dimensional image; Determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box; the standard bounding box is a bounding box corresponding to a standard three-dimensional model of the target part; The standard three-dimensional model is transformed according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

2. The method according to claim 1, characterized in that The two-dimensional image includes an orthotopic image and a lateral image, and the virtual bounding box includes a first bounding box and a second bounding box; and determining the virtual bounding box of the target part according to the two-dimensional image includes: Determining an anteroposterior corner point of the target part according to the anteroposterior image, and determining a lateral corner point of the target part according to the lateral image; Determine a first prism formed by vertically projecting from the plane where the orthogonal image is located with the orthogonal angle point as a reference, and intercept the first prism with the first plane and the second plane to obtain the first bounding box; the first plane and the second plane pass through the two lateral angle points respectively, the first plane and the second plane are both perpendicular to the plane where the lateral image is located, and the first plane and the second plane both intersect the first prism; Determine a second prism formed by vertically projecting from the plane where the lateral image is located with the lateral angle point as a reference, and use the third plane and the fourth plane to intercept the second prism to obtain the second bounding box; the third plane and the fourth plane pass through the two orthogonal angle points respectively, the third plane and the fourth plane are both perpendicular to the plane where the orthogonal image is located, and the third plane and the fourth plane both intersect with the second prism.

3. The method according to claim 2, characterized in that The determining of the anteroposterior corner point of the target part according to the anteroposterior image, and the determining of the lateral corner point of the target part according to the lateral image, comprises: Acquire the original anteroposterior corner point of the target part in the anteroposterior image, and the original lateral corner point of the target part in the lateral image; Performing an alignment operation on the original lateral angle point according to the original frontal angle point to obtain an aligned lateral angle point; The original anteroposterior angle point and the aligned lateral angle point are respectively back-projected to obtain the anteroposterior angle point and the lateral angle point.

4. The method according to claim 3, characterized in that The step of performing an alignment operation on the original lateral angle point according to the original frontal angle point to obtain the aligned lateral angle point includes: Taking the center point of the original positive angle point as the reference center point; According to the reference center point, the original lateral angle point is adjusted to obtain the aligned lateral angle point; the center point of the aligned lateral angle point matches the reference center point.

5. The method according to claim 3, characterized in that: The back-projection processing is performed on the original anteroposterior angle point and the aligned lateral angle point to obtain the anteroposterior angle point and the lateral angle point, respectively, including: The original anteroposterior angle point is back-projected into three-dimensional space to obtain the anteroposterior angle point, and the aligned lateral angle point is back-projected into three-dimensional space to obtain the lateral angle point.

6. The method according to claim 1, characterized in that Before determining the transformation relationship between the standard bounding box of the target part and the virtual bounding box, the method further includes: Acquire an original three-dimensional model of the target part, and determine an original bounding box of the original three-dimensional model; Determine a first transformation relationship corresponding to the original bounding box according to the normal vectors of each face of the original bounding box; the first transformation relationship is used to characterize the rotation relationship and translation relationship between the original bounding box and the standard bounding box; The original bounding box is transformed according to the first transformation relationship to obtain the standard bounding box of the target part.

7. The method according to claim 2, characterized in that: The step of determining the transformation relationship between the standard bounding box of the target part and the virtual bounding box includes: Determining a roll angle direction vector of the target part according to the first bounding box; Determining the pitch angle direction vector of the target part according to the second bounding box; Determine the center coordinates of the first bounding box and the second bounding box; A second transformation relationship between the virtual bounding box and the standard bounding box is obtained according to the roll angle direction vector, the pitch angle direction vector and the center coordinate; the second transformation relationship is used to characterize the rotation transformation and translation transformation between the virtual bounding box and the standard bounding box.

8. The method according to claim 7, characterized in that The step of determining the transformation relationship between the standard bounding box of the target part and the virtual bounding box further includes: Determine a first expansion amount of the first bounding box relative to the standard bounding box, and a second expansion amount of the second bounding box relative to the standard bounding box; A third transformation relationship between the virtual bounding box and the standard bounding box is obtained according to an average value of the first expansion amount and the second expansion amount; the third transformation relationship is used to characterize the scaling relationship between the virtual bounding box and the standard bounding box.

9. The method according to claim 8, characterized in that The step of transforming the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part includes: According to the second transformation relationship and the third transformation relationship, the standard three-dimensional model is transformed to obtain the three-dimensional reconstruction result of the target part.

10. A three-dimensional reconstruction device, characterized in that: The device comprises: An image acquisition module is used to acquire a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other; An image determination module, used to determine a virtual bounding box of the target part according to the two-dimensional image; A relationship determination module, used to determine the transformation relationship between the standard bounding box of the target part and the virtual bounding box; the standard bounding box is the bounding box corresponding to the standard three-dimensional model of the target part; The three-dimensional reconstruction module is used to transform the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part.

11. A surgical robot system, characterized in that: The system includes a processor and a surgical robot; The processor is used to obtain a two-dimensional image obtained by photographing the target part from two directions perpendicular to each other, determine a virtual bounding box of the target part according to the two-dimensional image, determine a transformation relationship between a standard bounding box of the target part and the virtual bounding box, and transform the standard three-dimensional model according to the transformation relationship to obtain a three-dimensional reconstruction result of the target part; the standard bounding box is a bounding box corresponding to the standard three-dimensional model of the target part; The surgical robot is used to perform a predetermined operation on the target part according to the three-dimensional reconstruction result.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images