Method for positioning an operating target by a mandibular reconstruction robot

Abstract

The application provides a method for positioning an operation target by a mandibular reconstruction robot, and the method comprises the following steps: (1) establishing a virtual model of the operation target according to a surgical plan to obtain a first oriented bounding box; (2) collecting at least one set of point clouds of the operation target by a camera; (3) obtaining a second oriented bounding box according to the point clouds; and (4) performing coarse registration on the first oriented bounding box and the second oriented bounding box to obtain the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera. Compared with the prior art, the application can realize the positioning of the operation target without the need for markers, avoids the errors in the positioning of the operation target caused by the difficulty in fixing the markers during the surgery, does not need an expensive tracking navigation system, greatly reduces the cost of the surgery, and can reduce the operation difficulty of doctors, save bone cutting working hours, and improve bone cutting precision.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a method for locating and manipulating targets using a mandibular reconstruction robot. Background Technology

[0002] Mandibular resection is a common treatment for benign tumors of the mandible. Patients who undergo mandibular resection will lose at least part of their mandible, which will affect or even eliminate basic functions such as speaking and chewing, seriously impacting their daily lives. In such cases, a mandibular prosthesis is needed to fill the patient's mandible.

[0003] However, unlike bone prostheses in other parts of the body, the mandible is exposed to the oral cavity's bacterial environment. If materials such as metal-polyethylene, ceramic, or titanium alloy are implanted, they will be affected by the oral environment and oxidize and deteriorate. In this case, a living bone that can survive in the oral cavity for a long time is needed. Usually, the patient's ribs or fibulas are cut off, reshaped into the shape of the mandible, and then implanted into the missing mandible. It is necessary to ensure that the blood vessels of the ribs or fibulas are anastomosed with the carotid artery to restore the function and shape of the mandible.

[0004] In the past, the cutting and shaping of ribs or fibulas were mostly done manually, which had problems such as low osteotomy accuracy and long working time. With the development of automation technology, the shaping of the mandible can reduce the difficulty of operation for doctors, save osteotomy time, and improve osteotomy accuracy. The positioning of the target bone is an important preparatory work for using a robot to reconstruct the mandible.

[0005] In existing technologies, the localization of the target is mostly achieved through a navigation tracking system combined with markers. The markers need to be fixed to the target object beforehand, and the tracking system identifies the markers to locate the target. However, in actual surgery, due to the limited operating space, it is difficult to fix the markers, which can easily become misaligned. This results in an error between the target identified and located by the navigation tracking system and the actual target, affecting the surgical outcome. Furthermore, the navigation tracking system is expensive. Therefore, a new localization method is needed that can locate the target without the participation of markers. Summary of the Invention

[0006] The purpose of this invention is to provide a method for locating a target using a mandibular reconstruction robot, which can achieve the location of the target without the need for markers.

[0007] To address the aforementioned technical problems, this invention provides a method for locating a target using a mandibular reconstruction robot, characterized in that the method includes the following steps:

[0008] (1) Establish a virtual model of the operation target based on the surgical plan to obtain the first directed bounding box;

[0009] (2) Acquire at least one set of point clouds of the target object using a camera;

[0010] (3) Obtain the second directed bounding box based on the point cloud;

[0011] (4) Perform coarse registration on the first directed bounding box and the second directed bounding box to obtain the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera.

[0012] Optionally, the camera is a structured light camera, and step (2) includes:

[0013] (2-1) Define the initial position of the camera as the zero point, and the camera acquires the point cloud of the target being operated on;

[0014] (2-2) Starting from the zero point, rotate the camera and collect the point cloud of the target object with each rotation to obtain at least two sets of point clouds.

[0015] Optionally, step (3) includes:

[0016] (3-1) Identify the planes in each group of point clouds and segment the planes from the point clouds;

[0017] (3-2) Remove outliers from each point cloud group;

[0018] (3-3) Project each group of point clouds onto the plane as a whole;

[0019] (3-4) Distinguish point cloud clusters based on the distances between point cloud projections on the plane;

[0020] (3-5) Obtain the second directed bounding box through the point cloud clusters.

[0021] Optionally, the coarse registration of the first directed bounding box and the second directed bounding box specifically involves:

[0022] The positional relationship between the target and the virtual model is obtained through the pose transformation matrix: ,

[0023] Where M1 is the pose matrix of the first directed bounding box, and M2 is the pose matrix of the second directed bounding box.

[0024] Optionally, the method further includes:

[0025] (5) Merge the point cloud to obtain the first point set, and obtain the second point set according to the virtual model;

[0026] (6) Perform fine registration on the first point set and the second point set to obtain the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera.

[0027] Optionally, step (5) specifically includes:

[0028] (5-1) Merge point clouds based on coordinate transformation information;

[0029] (5-2) Define the first point set and the second point set according to the following formula:

[0030] ;

[0031] ,

[0032] Where P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

[0033] Optionally, the fine registration of the first point set and the second point set includes the following steps:

[0034] (6-1) Center the first point set and the second point set;

[0035] (6-2) Calculate the rotation matrix and translation vector based on the first point set and the second point set;

[0036] (6-3) Determine whether the error is less than the preset threshold. If yes, output the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera. Otherwise, continue with step (6-1).

[0037] Optionally, step (6-1) specifically includes:

[0038] The first and second point sets are centered according to the following formula:

[0039] ,

[0040] Where P is the first set of points, Q is the second set of points, i is the number of coordinates, and N is the maximum number of coordinates.

[0041] Optionally, step (6-2) specifically includes:

[0042] Calculate the rotation matrix and translation vector for the following function when it reaches its minimum:

[0043] ,

[0044] Where P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

[0045] Optionally, step (6-2) specifically includes the following steps:

[0046] (6-2-1) Perform singular value decomposition of the matrix rows according to the following formula:

[0047] ,

[0048] get Where G is a matrix, T is a translation vector, i is the number of coordinates, N is the maximum number of coordinates, U is the first orthogonal matrix, and V is the second orthogonal matrix;

[0049] (6-2-2) Calculate the rotation matrix using the following formula:

[0050] ,

[0051] Where R is the rotation matrix, T is the translation vector, U is the first orthogonal matrix, and V is the second orthogonal matrix;

[0052] (6-2-3) Calculate the translation vector using the following formula:

[0053] ,

[0054] Where R is the rotation matrix and T is the translation vector.

[0055] Optionally, determining whether the error is less than a preset threshold specifically includes:

[0056] (6-3-1) Set thresholds according to surgical needs;

[0057] (6-3-2) Calculate the error using the following formula:

[0058] ,

[0059] Where E is the error, P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

[0060] Compared with existing technologies, the present invention provides a method for locating the target of the operation using a mandibular reconstruction robot, which can achieve the positioning of the target of the operation without the need for markers, thus avoiding the positioning error caused by the difficulty in fixing the markers during actual surgery; at the same time, it eliminates the need for an expensive tracking and navigation system, greatly reducing surgical costs; compared with manual operation, it can reduce the difficulty of operation for doctors, save osteotomy time, and improve osteotomy accuracy. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the method for locating the target operation using a mandibular reconstruction robot in Embodiment 1 of the present invention.

[0062] Figure 2 This is a schematic diagram of camera-acquired point cloud data for the method of locating the target of the robot by reconstructing the mandible in Embodiment 1 of the present invention.

[0063] Figure 3 This is a schematic diagram of the fine registration process of the method for locating the target of the operation by the mandibular reconstruction robot in Embodiment 1 of the present invention.

[0064] Explanation of reference numerals in the attached figures:

[0065] 1-Operation target; 2-Camera. Detailed Implementation

[0066] The method for locating the target using a mandibular reconstruction robot, as proposed in this invention, will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0067] As used herein, the singular forms “a,” “an,” and “the” include plural objects unless otherwise expressly indicated. As used herein, the term “or” is generally used to include “and / or” unless otherwise expressly indicated. As used herein, the term “a number” is generally used to include “at least one” unless otherwise expressly indicated. As used herein, the term “at least two” is generally used to include “two or more” unless otherwise expressly indicated. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature.

[0068] As described in the background section, existing technologies typically use navigation tracking systems in conjunction with markers to locate the target of the operation. However, in actual surgical procedures, due to the limited operating space, it is difficult to fix the markers, which can easily lead to misalignment of manually fixed markers. This results in an error between the target identified and located by the navigation tracking system and the actual target, affecting the surgical outcome. Furthermore, navigation tracking systems are expensive.

[0069] Therefore, a completely new positioning method is needed to locate the operational target without the need for markers.

[0070] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0071] Figure 1 This is a flowchart illustrating the method for locating the target using a mandibular reconstruction robot in Embodiment 1 of the present invention. Figure 1 As shown, the present invention provides a method for locating a target for manipulation using a robot based on mandibular bone reconstruction, characterized in that the method includes the following steps:

[0072] (1) Establish a virtual model of operation target 1 according to the surgical plan to obtain the first directed bounding box;

[0073] (2) Acquire at least one set of point clouds of the target 1 using camera 2;

[0074] (3) Obtain the second directed bounding box based on the point cloud;

[0075] (4) Perform coarse registration on the first directed bounding box and the second directed bounding box to obtain the positional relationship between the operation target 1 and the virtual model and the positional relationship between the operation target 1 and the camera 2.

[0076] It should be noted that the target of mandibular reconstruction surgery is usually the ribs or fibula of the human body. Those skilled in the art can choose different targets of operation 1 according to the actual surgical needs. If the ribs or fibula are mentioned in the following embodiments, it is not a limitation on the target of operation 1.

[0077] In one specific embodiment, medical personnel need to determine the surgical plan through preoperative planning. Generally, doctors will obtain the shape information of the patient's ribs or fibula through CT scans and build a three-dimensional virtual model based on this. After that, medical personnel can define the cutting path and cutting start point on the virtual model and plan the specific surgical procedure.

[0078] In a more basic embodiment, after the surgical plan is determined and the virtual model is established, the camera 2 needs to capture at least a set of point clouds. The point clouds are used to describe the outer surface features of the operation target 1. The coordinate system can be determined by defining a source point cloud. By associating the coordinate system of the operation target 1 acquired by the camera 2 with the coordinate system of the virtual model, the first directed bounding box of the virtual model coincides with the second directed bounding box of the operation target 1. In this way, the robot can implement the surgical plan set on the virtual model on the operation target 1.

[0079] Figure 2 This is a schematic diagram of the camera-acquired point cloud of the method for locating the target of the operation by a robot through mandibular reconstruction in Embodiment 1 of the present invention, as shown below. Figure 2 As shown, further, the camera 2 is a structured light camera, and step (2) includes:

[0080] (2-1) Define the initial position of the camera 2 as the zero point, and the camera 2 acquires the point cloud of the target 1;

[0081] (2-2) Starting from the zero point, rotate the camera 2, and collect the point cloud of the target 1 each time it rotates, so as to obtain at least two sets of point clouds.

[0082] In a preferred embodiment, a fixed robot delta platform and a RealSense D435 depth camera can be used. This camera 2 projects a dot matrix onto the surface of the target 1, and infrared light is used to identify the dot matrix to obtain surface features described by a point cloud. Point clouds in different poses are acquired by rotating the robot axis carrying the camera 2. Each rotation uses the camera 2 to acquire point cloud information of the target 1 in the current pose. Theoretically, the more rotations, the more accurate the point cloud information. Generally, the initial position is defined as the zero point, and each rotation uses a fixed angle. Approximately 10 rotations are needed to obtain relatively ideal point cloud information.

[0083] Furthermore, step (3) includes:

[0084] (3-1) Identify the planes in each group of point clouds and segment the planes from the point clouds;

[0085] (3-2) Remove outliers from each point cloud group;

[0086] (3-3) Project each group of point clouds onto the plane as a whole;

[0087] (3-4) Distinguish point cloud clusters based on the distances between point cloud projections on the plane;

[0088] (3-5) Obtain the second directed bounding box through the point cloud clusters.

[0089] In a specific embodiment, the planar point cloud of the desktop can be used as a point cloud feature for coarse registration. After segmenting the plane representing the desktop from the point cloud, the point cloud information of the operation target 1 and the fixture is retained. Outlier removal is performed on the segmented point cloud to achieve a filtering effect. Since the operation target 1 and the fixture are parallel to the desktop, in order to find the pose relationship between the point clouds, the point cloud of the operation target 1 can be projected onto the planar point cloud representing the desktop. In order to distinguish different point cloud clusters based on distance, Euclidean clustering is required to segment the projected two-dimensional point cloud clusters. Then, the second directed bounding box can be obtained, and the homogeneous transformation matrix of each point cloud relative to the source point cloud can also be obtained.

[0090] Furthermore, the coarse registration of the first directed bounding box and the second directed bounding box specifically involves:

[0091] The positional relationship between the target 1 and the virtual model is obtained through the pose transformation matrix: ,

[0092] Where M1 is the pose matrix of the first directed bounding box, and M2 is the pose matrix of the second directed bounding box.

[0093] Since the directed bounding box can only describe the general direction, further fine registration can be performed to improve the progress of the operation. Figure 3 This is a schematic diagram of the fine registration process for the method of locating the target using a mandibular reconstruction robot in Embodiment 1 of the present invention, as shown below. Figure 3 As shown, the method further includes:

[0094] (5) Merge the point cloud to obtain the first point set, and obtain the second point set according to the virtual model;

[0095] (6) Perform fine registration on the first point set and the second point set to obtain the positional relationship between the operation target 1 and the virtual model and the positional relationship between the operation target 1 and the camera 2.

[0096] Fine registration builds upon coarse registration, using the coarse registration result as the initial registration value. Then, the Iterative Nearest Point (ICP) algorithm is employed to finely register the point clouds acquired under different robot postures, ultimately obtaining the relative pose relationship between the target 1 and camera 2. In principle, the ICP algorithm can be roughly divided into two steps: first, searching for the closest pair of points in the two point clouds to be registered; and second, using the nearest points as corresponding points for registration transformation.

[0097] Furthermore, step (5) specifically includes:

[0098] (5-1) Merge point clouds based on coordinate transformation information;

[0099] (5-2) Define the first point set and the second point set according to the following formula:

[0100] ;

[0101] ,

[0102] Where P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

[0103] In the above formula, Let P represent the first point set P and the second point set Q, which contain N coordinate points acquired by camera 2. Each point in the second point set Q, after spatial transformation, corresponds one-to-one with a point in the first point set P. The relationship of the single-point transformation is as follows: .

[0104] Furthermore, the fine registration of the first point set and the second point set includes the following steps:

[0105] (6-1) Center the first point set and the second point set;

[0106] (6-2) Calculate the rotation matrix and translation vector based on the first point set and the second point set;

[0107] (6-3) Determine whether the error is less than the preset threshold. If so, output the positional relationship between the operation target 1 and the virtual model and the positional relationship between the operation target 1 and the camera 2. Otherwise, continue with step (6-1).

[0108] In a specific embodiment, it is necessary to perform an initial estimation and remove the translation vector T, obtain the center of the first point set P and the second point set Q, and translate the first point set P and the second point set Q to the center point.

[0109] Furthermore, step (6-1) specifically includes:

[0110] The first and second point sets are centered according to the following formula:

[0111] ,

[0112] Where P is the first set of points, Q is the second set of points, i is the number of coordinates, and N is the maximum number of coordinates.

[0113] Since the center coordinates of the second point set Q of the first point set P also satisfy... After obtaining the decentralized first point set P and second point set Q, rigid body transformations between point clouds can be constructed using vectors. express.

[0114] Furthermore, step (6-2) specifically includes:

[0115] Calculate the rotation matrix and translation vector for the following function when it reaches its minimum:

[0116] ,

[0117] Where P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates. This formula can also describe the error of the ICP iterative closest point algorithm to some extent.

[0118] Furthermore, step (6-2) specifically includes the following steps:

[0119] (6-2-1) Perform singular value decomposition on the matrix according to the following formula:

[0120] ,

[0121] get Where G is a matrix, T is a translation vector, i is the number of coordinates, N is the maximum number of coordinates, U is the first orthogonal matrix, and V is the second orthogonal matrix;

[0122] (6-2-2) Calculate the rotation matrix using the following formula:

[0123] ,

[0124] Where R is the rotation matrix, T is the translation vector, U is the first orthogonal matrix, and V is the second orthogonal matrix;

[0125] (6-2-3) Calculate the translation vector using the following formula:

[0126] ,

[0127] Where R is the rotation matrix and T is the translation vector.

[0128] In a specific embodiment, the above steps can also be described as follows:

[0129] Let the function be:

[0130] ,

[0131] When the value of H is at its maximum, the value of error E is at its minimum. Therefore, when H = trace(RG) is at its maximum, singular value decomposition can be performed on G. Let:

[0132] ;

[0133] ,

[0134] In the above formula, U is the first orthogonal matrix, and V is the second orthogonal matrix. If X is a non-negative diagonal matrix and X is also an orthogonal matrix, then for any orthogonal matrix Y:

[0135] ,

[0136] Therefore, when R=X, the function The maximum value can be obtained, thereby solving for the rotation matrix R, and further obtaining the translation vector T.

[0137] Furthermore, determining whether the error is less than a preset threshold specifically includes:

[0138] (6-3-1) Set thresholds according to surgical needs;

[0139] (6-3-2) Calculate the error using the following formula:

[0140] ,

[0141] Where E is the error, P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

[0142] It should be noted that the threshold setting is determined based on the actual surgical needs, and those skilled in the art can improve the surgical progress by lowering the threshold.

[0143] In summary, compared with the prior art, the method for locating the operation target using a mandibular reconstruction robot provided by the present invention can achieve the positioning of the operation target without the need for markers, avoiding the positioning error caused by the difficulty in fixing markers during actual surgery; at the same time, it eliminates the need for an expensive tracking navigation system, greatly reducing surgical costs; compared with manual operation, it can reduce the difficulty of operation for doctors, save osteotomy time, and improve osteotomy accuracy.

[0144] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A method for locating a target for robotic manipulation using mandibular reconstruction, characterized in that, The method includes the following steps: (1) Establish a virtual model of the operation target based on the surgical plan to obtain the first directed bounding box; (2) Acquire at least one set of point clouds of the target object using a camera; (3) Obtain the second directed bounding box based on the point cloud; (4) Perform coarse registration on the first directed bounding box and the second directed bounding box to obtain the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera; (5) Merge the point cloud to obtain the first point set, and obtain the second point set according to the virtual model; (6) Perform fine registration on the first point set and the second point set to obtain the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera; in, Step (2) includes: (2-1) Define the initial position of the camera as the zero point, and the camera acquires the point cloud of the target being operated on; (2-2) Starting from the zero point, rotate the camera and collect the point cloud of the target object with each rotation to obtain at least two sets of point clouds; Step (3) includes: (3-1) Identify the planes in each group of point clouds and segment the planes from the point clouds; (3-2) Remove outliers from each point cloud group; (3-3) Project each group of point clouds onto the plane as a whole; (3-4) Distinguish point cloud clusters based on the distance between point cloud projections on the plane; (3-5) Obtain the second directed bounding box through the point cloud clusters.

2. The method for locating and manipulating a target using a mandibular reconstruction robot as described in claim 1, characterized in that, The coarse registration of the first directed bounding box and the second directed bounding box specifically involves: The positional relationship between the target and the virtual model is obtained through the pose transformation matrix: , Where M1 is the pose matrix of the first directed bounding box, and M2 is the pose matrix of the second directed bounding box.

3. The method for locating and manipulating a target using a mandibular reconstruction robot as described in claim 1, characterized in that, Step (5) specifically includes: (5-1) Merge point clouds based on coordinate transformation information; (5-2) Define the first point set and the second point set according to the following formula: ； ， Where P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

4. The method for locating the target using a robot via mandibular reconstruction as described in claim 1, characterized in that, The fine registration of the first point set and the second point set includes the following steps: (6-1) Center the first point set and the second point set; (6-2) Calculate the rotation matrix and translation vector based on the first point set and the second point set; (6-3) Determine whether the error is less than the preset threshold. If yes, output the positional relationship between the operation target and the virtual model and the positional relationship between the operation target and the camera. Otherwise, continue with step (6-1).

5. The method for locating the target using a robot via mandibular reconstruction as described in claim 4, characterized in that, The specific steps (6-1) are as follows: The first and second point sets are centered according to the following formula: ， Where P is the first set of points, Q is the second set of points, i is the number of coordinates, and N is the maximum number of coordinates.

6. The method for locating the target using a mandibular reconstruction robot as described in claim 5, characterized in that, The specific steps (6-2) are as follows: Calculate the rotation matrix and translation vector for the following function when it reaches its minimum: ， Where P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

7. The method for locating and manipulating a target using a mandibular reconstruction robot as described in claim 6, characterized in that, Step (6-2) specifically includes the following steps: (6-2-1) Perform singular value decomposition on the matrix according to the following formula: ， get Where G is a matrix, T is a translation vector, i is the number of coordinates, N is the maximum number of coordinates, U is the first orthogonal matrix, and V is the second orthogonal matrix; (6-2-2) Calculate the rotation matrix using the following formula: ， Where R is the rotation matrix, T is the translation vector, U is the first orthogonal matrix, and V is the second orthogonal matrix; (6-2-3) Calculate the translation vector using the following formula: ， Where R is the rotation matrix and T is the translation vector.

8. The method for locating the target using a mandibular reconstruction robot as described in claim 4, characterized in that, The determination of whether the error is less than a preset threshold specifically includes: (6-3-1) Set thresholds according to surgical needs; (6-3-2) Calculate the error using the following formula: ， Where E is the error, P is the first point set, Q is the second point set, R is the rotation matrix, T is the translation vector, i is the number of coordinates, and N is the maximum number of coordinates.

Images