Surgical system based on structured light registration, registration control method, and storage medium

Abstract

The application provides a surgical system based on structured light registration, a registration control method and a storage medium, and the surgical system based on structured light registration comprises a first marker, a structured light camera and a control device; the first marker is used for being attached to a target object; the structured light camera is used for identifying a pose of the first marker in a structured light camera coordinate system; the control device is configured to acquire a medical image of the target object to which the first marker is attached, to obtain a pose of the first marker in a medical image coordinate system, to further acquire a first conversion matrix of the structured light camera coordinate system and the medical image coordinate system, and to register the first marker based on the first conversion matrix. In this way, the structured light camera no longer needs to perform overall point cloud reconstruction on the target object, but only needs to identify the first marker, the imaging frequency can be greatly improved, the cost is reduced, and the calculation speed is accelerated.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a surgical system, registration control method and storage medium based on structured light registration. Background Technology

[0002] Currently, the application of structured light in medical robots is mostly in target detection, 3D reconstruction, etc. In some applications, the structured light registration method used in surgical robots is mostly point cloud registration. The registration result of this method depends on the quality of point cloud imaging, and the calculation time is long and the algorithm is complex. When controlling the robotic arm, there is a lack of feedback on the pose of the robotic arm end, which may lead to unsatisfactory final absolute positioning accuracy. Summary of the Invention

[0003] The purpose of this invention is to provide a surgical system, registration control method, and storage medium based on structured light registration, so as to solve the problem that the existing structured light registration uses overall point cloud registration which is too complicated.

[0004] To solve the above-mentioned technical problems, the present invention provides a surgical system based on structured light registration, which includes: a first marker, a structured light camera, and a control device;

[0005] The first marker is used to be attached to the target object;

[0006] The structured light camera is used to identify the pose of the first marker in the structured light camera coordinate system;

[0007] The control device is configured to acquire a medical image of the target object with the first marker attached, obtain the pose of the first marker in the medical image coordinate system; then acquire a first transformation matrix between the structured light camera coordinate system and the medical image coordinate system; and register the first marker based on the first transformation matrix.

[0008] Optionally, in the structured light registration-based surgical system, the structured light camera scans the first marker at a preset frequency, and the control device is configured to track the pose change of the first marker. If the pose change of the first marker exceeds a first preset value in the scan results of two consecutive frames, it indicates that the target object has moved.

[0009] Optionally, the structured light registration-based surgical system further includes a surgical robot, which includes a robotic arm. The control device is further configured to compensate the motion control unit of the robotic arm for the pose change of the first marker if the pose change of the first marker exceeds a first preset value in the scan results of two consecutive frames.

[0010] Optionally, the first marker includes a radiopaque marker for medical image scanning and an image marker for structured light camera scanning, and the relative positions of the radiopaque marker and the image marker are fixed.

[0011] Optionally, the structured light registration-based surgical system further includes a surgical robot and a second marker, wherein the surgical robot includes a robotic arm on which surgical instruments are connected;

[0012] The second marker is used to be attached to the end of the robotic arm;

[0013] The structured light camera is also used to identify the pose of the second marker in the structured light camera coordinate system;

[0014] The control device is further configured to acquire a second conversion matrix between the second marker and the surgical instrument, and register the second marker based on the second conversion matrix;

[0015] In the structured light image acquired by the structured light camera, the images corresponding to the second marker and the first marker are not the same.

[0016] Optionally, in the structured light registration-based surgical system, the control device is further configured to acquire the first pose of the surgical instrument recorded by the robotic arm controller, obtain the second pose of the surgical instrument based on the pose of the second marker, and compensate the difference between the first pose and the second pose to the motion control unit of the robotic arm of the surgical robot.

[0017] Optionally, in the structured light registration-based surgical system, the structured light registration-based surgical system further includes a support device and a third marker;

[0018] The third marker is used to be attached to the carrier device;

[0019] The structured light camera is also used to identify the pose of the third marker in the structured light camera coordinate system, and to scan the first marker and the third marker at a preset frequency;

[0020] The control device is further configured to acquire a third transformation matrix between the pose of the first marker and the pose of the third marker; and track the changes in the third transformation matrix. If the change in the third transformation matrix exceeds a second preset value in the scanning results of two consecutive frames, the device will indicate that the target object has moved.

[0021] In the structured light image acquired by the structured light camera, the third marker is not the same as the image corresponding to the first marker.

[0022] Optionally, the structured light registration-based surgical system further includes a surgical robot, which includes a robotic arm. The control device is further configured to compensate the change in the third transformation matrix to the motion control unit of the robotic arm if the change in the third transformation matrix exceeds a second preset value in the scan results of two consecutive frames.

[0023] To address the aforementioned technical problems, this invention also provides a registration control method based on structured light, comprising:

[0024] Attach the first marker to the target object;

[0025] Acquire a medical image of the target object with the first marker affixed to it;

[0026] The first marker is scanned using a structured light camera, and its pose in the structured light camera coordinate system is identified.

[0027] Based on the medical image, the pose of the first marker in the medical image coordinate system is obtained; then, the first transformation matrix between the structured light camera coordinate system and the medical image coordinate system is obtained; and the first marker is registered based on the first transformation matrix.

[0028] Optionally, the structured light-based registration control method further includes:

[0029] The structured light camera scans the first marker at a preset frequency;

[0030] The pose change of the first marker is tracked. If the pose change of the first marker exceeds a first preset value in the scan results of two consecutive frames, it is indicated that the target object has moved.

[0031] Optionally, in the structured light-based registration control method, if the pose change of the first marker exceeds a first preset value in the scanning results of two consecutive frames, the pose change of the first marker is compensated to the motion control unit of the robotic arm.

[0032] Optionally, the structured light-based registration control method further includes:

[0033] Attach the second marker to the end of the robotic arm;

[0034] The second marker is scanned using a structured light camera, and its pose in the structured light camera coordinate system is identified.

[0035] Obtain the second transformation matrix between the second marker and the surgical instrument, and register the second marker based on the second transformation matrix.

[0036] Optionally, the registration control method based on structured light further includes: acquiring the first pose of the surgical instrument recorded by the robotic arm controller, obtaining the second pose of the surgical instrument based on the pose of the second marker, and compensating the difference between the first pose and the second pose to the motion control unit of the robotic arm of the surgical robot.

[0037] Optionally, the structured light-based registration control method further includes:

[0038] Affix the third marker to the carrier device;

[0039] The third marker is scanned using a structured light camera to identify its pose in the structured light camera coordinate system, and the first marker and the third marker are scanned at a preset frequency.

[0040] Obtain the third transformation matrix of the pose of the first marker and the pose of the third marker; and track the changes of the third transformation matrix. If the change of the third transformation matrix exceeds a second preset value in the scanning results of two consecutive frames, the target object is prompted to move.

[0041] Optionally, in the structured light-based registration control method, if the change in the third transformation matrix exceeds a second preset value in the scanning results of two consecutive frames, the change in the third transformation matrix is ​​compensated to the motion control unit of the robotic arm.

[0042] To address the aforementioned technical problems, the present invention also provides a readable storage medium storing a program thereon, which, when executed, implements the steps of the structured light-based registration control method as described above.

[0043] In summary, in the structured light registration-based surgical system, structured light registration control method, and readable storage medium provided by this invention, the structured light registration-based surgical system includes: a first marker, a structured light camera, and a control device; the first marker is used to be attached to a target object; the structured light camera is used to identify the pose of the first marker in the structured light camera coordinate system; the control device is configured to acquire a medical image of the target object with the first marker attached, obtain the pose of the first marker in the medical image coordinate system; then acquire a first transformation matrix between the structured light camera coordinate system and the medical image coordinate system; and register the first marker based on the first transformation matrix.

[0044] This configuration, by attaching a specific first marker to the target object, eliminates the need for the structured light camera to reconstruct the entire point cloud of the target object; it only needs to identify the first marker, significantly increasing the imaging frequency. Compared to traditional NDI tracking and navigation, the structured light registration control scheme greatly reduces costs. Furthermore, compared to traditional structured light positioning, the use of the first marker accelerates calculations, significantly improving real-time performance and accuracy. The first marker is attached to the target object, avoiding invasive positioning methods like bone screws required in traditional NDI tracking and navigation, essentially achieving non-invasive registration and greatly reducing patient trauma. Attached Figure Description

[0045] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0046] Figure 1 This is a schematic diagram of a surgical system based on structured light registration according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the first marker being attached to the target object according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the first marker in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the second marker being attached to the robotic arm according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the second marker in an embodiment of the present invention;

[0051] Figure 6 This is a flowchart of a preferred example of a surgical system based on structured light registration according to an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the third marker being attached to the carrier device according to an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the third marker in an embodiment of the present invention;

[0054] Figure 9 This is a flowchart of another preferred example of a surgical system based on structured light registration, which is an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to facilitate and clarify the explanation of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and may sometimes use different scales.

[0056] As used in this invention, the singular forms “a,” “an,” and “the” include plural objects; the term “or” is generally used to mean “and / or”; the term “a number” is generally used to mean “at least one”; and the term “at least two” is generally used to mean “two or more”. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature; “one end” and “the other end,” and “proximal end” and “distal end” generally refer to two corresponding parts, which include not only endpoints. Furthermore, the terms "installed," "connected," and "attached," as used in this invention, and the term "set" on one element from another, should be interpreted broadly. They generally only indicate a connection, coupling, cooperation, or transmission relationship between the two elements, which can be direct or indirect through an intermediate element. They should not be construed as indicating or implying a spatial relationship between the two elements, meaning one element can be located inside, outside, above, below, or to one side of another element, unless otherwise explicitly stated. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Additionally, directional terms such as above, below, up, down, upward, downward, left, and right are used relative to exemplary embodiments as shown in the figures, with upward or upper directions pointing towards the top of the corresponding figure, and downward or lower directions pointing towards the bottom of the corresponding figure.

[0057] The purpose of this invention is to provide a surgical system based on structured light registration, a registration control method based on structured light, and a readable storage medium to solve the problem that existing structured light registration methods using overall point cloud registration are too complex.

[0058] The following description refers to the accompanying drawings.

[0059] Figure 1An application scenario of a structured light registration-based surgical system is illustrated. This system includes a first marker 11, a structured light camera 20, and a control device (not shown). Preferably, the system further includes a surgical robot. The surgical robot includes a surgical execution component, which comprises a robotic arm 33 and surgical instruments 34 connected to the robotic arm 33. The main controller includes a motion control unit for controlling the movement of the robotic arm 33 and the surgical instruments 34. In an alternative example, the surgical robot is a puncture surgical robot, which includes a main controller, a carriage 31, a surgical execution component mounted on the carriage 31, and a support device 32 (e.g., an operating table) for carrying the surgical object and performing the surgery. It should be noted that in some embodiments, the support device 32 can be replaced with other surgical operating platforms. Optionally, the surgical instruments 34 include puncture devices, etc., for performing specific puncture operations.

[0060] The following explanation uses the surgical subject as the target object 35 as an example. It should be noted that the target object 35 is not limited to the surgical subject; it can also be objects such as model prostheses, which can be used for operational training and registration purposes. The control device can be set up separately or integrated into or attached to the structured light camera 20 or the puncture surgical robot. For example, it can be integrated into the main controller of the puncture surgical robot. This invention does not limit the location of the control device. Furthermore, the application scenarios of the structured light registration-based surgical system of this invention are not limited to puncture surgical robots; they can also be applied to other surgical robots, such as single-ended surgical robots (e.g., orthopedic surgical robots). Optionally, the structured light camera 20 uses an infrared light source to avoid visible light interfering with the surgery.

[0061] Please refer to the reference. Figure 1 and Figure 2 The first marker 11 is used to be attached to the target object 35; the structured light camera 20 is used to identify the pose of the first marker 11 in the structured light camera coordinate system; the control device is configured to acquire a medical image of the target object 35 with the first marker 11 attached, and obtain the pose of the first marker 11 in the medical image coordinate system; then acquire a first transformation matrix between the structured light camera coordinate system and the medical image coordinate system; and register the first marker 11 based on the first transformation matrix.

[0062] In one example, five first markers 11 are attached to the head of the target object 35. The target object 35, with the first markers 11 attached, undergoes a medical image scan (such as a CT scan), obtaining a medical image (e.g., a multi-planar reconstructed MPR view), and simultaneously, the pose of the first markers 11 in the medical image coordinate system. Then, after the target object 35 lies on the support device 32, the structured light camera 20 takes a picture of the target object 35, obtaining a structured light image, and simultaneously, the pose of the first markers 11 in the structured light camera coordinate system. In an optional embodiment, the structured light image can be displayed on navigation software, and the five first markers 11 on the structured light image can be further sorted. The medical image can also be displayed on the navigation software, and the operator can select on the medical image according to the sorted order of the five first markers 11, associating and matching the first markers 11 in the medical image with the first markers 11 in the structured light image. Of course, the associative matching process is not limited to manual selection by the operator; it can also be implemented by the navigation software program. It should be noted that the number of first markers 11 is not limited to 5, and their placement is not limited to the head of the target object 35. The number and placement of the first markers 11 can be configured according to actual needs. Preferably, the center points of any 3 first markers 11 are not collinear to avoid anomalies when calculating the registration first transformation matrix.

[0063] Furthermore, the poses of the five first markers 11 can be stored in array format. By solving the equations of the pose arrays of the first markers 11 in the structured light camera coordinate system and the pose arrays of the first markers 11 in the medical image coordinate system, the first transformation matrix between the structured light camera coordinate system and the medical image coordinate system can be calculated. Based on the first transformation matrix, the registration of the first markers 11 can be completed, that is, the registration of the target object 35 can be completed.

[0064] Optionally, the first marker 11 includes a imaging marker for medical image scanning and an image marker for structured light camera scanning, and the relative positions of the imaging marker and the image marker are fixed. Please refer to... Figure 3 In an alternative example, the first marker 11 is a square marker with dimensions of 30mm x 30mm and a thickness of 1mm. A developing mark 111 is located at one corner (e.g., the upper left corner). The developing mark 111 can be a raised tungsten alloy ball with a diameter of 1mm. The front of the first marker 11 has a square QR code pattern 112 that can be recognized by OpenCV. The QR code pattern 112 is 25mm x 25mm. The coordinate system direction vector of the developing mark 111 relative to the center of the QR code pattern 112 is (-1, 1, 0). The design of the pattern on the first marker 11 is not based on... Figure 3 The above is a limited list. Figure 3The pattern shown is an ArUco marker, from the open-source ArUco library. The imaging marker 111 can be used for medical image recognition, and the QR code pattern 112 can be used for recognition by the structured light camera 20. After the structured light camera 20 completes scanning, based on the intrinsic parameters and depth map of the structured light camera 20, the coordinates of the center of the QR code pattern 112 of the five first markers 11 in the structured light camera coordinate system can be calculated. Based on the coordinate system direction vector of the imaging marker 111 relative to the center of the QR code pattern 112, the coordinates of the imaging marker 111 in the structured light camera coordinate system can be calculated. After associating and matching the imaging marker 111 of the first markers 11 in the medical image and the imaging marker 111 of the first markers 11 in the structured light image, the first markers 11 in the medical image and the first markers 11 in the structured light image can be mutually registered, establishing the relative relationship between the structured light camera 20 and the target object 35, thus completing the registration of the first markers 11 and the target object 35.

[0065] Optionally, the structured light camera 20 scans the first marker 11 at a preset frequency. The control device is configured to track the pose changes of the first marker 11. If the pose change of the first marker 11 exceeds a first preset value in two consecutive scan frames, it indicates that the target object 35 has moved. This configuration allows for real-time determination of whether the target object 35 has moved during surgery. If the target object 35 moves, the operator can be reminded to re-register. It should be noted that the preset frequency and the first preset value can be set according to actual conditions; for example, the preset frequency could be three frames per second.

[0066] Furthermore, the control device is also configured to compensate the motion control unit of the robotic arm 33 for the change in pose of the first marker 11 exceeding a first preset value in two consecutive scan frames. If the change in pose of the first marker 11 exceeds the first preset value in two consecutive scan frames, it can be considered that the target object 35 has moved. In this case, compensating the motion control unit of the robotic arm 33 for the change in pose of the first marker 11 can make the movement of the robotic arm 33 follow the movement of the target object 35, thereby improving the final positioning accuracy.

[0067] For preferred options, please refer to [the following]. Figure 4 and Figure 5The structured light registration-based surgical system further includes a second marker 12, which is attached to the end of the robotic arm 33. The structured light camera 20 is also used to identify the pose of the second marker 12 in the structured light camera coordinate system. The control device is further configured to acquire a second transformation matrix between the second marker 12 and the surgical instrument 34, and register the second marker 12 based on the second transformation matrix, that is, to complete the registration of the robotic arm 33. Notably, the images corresponding to the second marker 12 and the first marker 11 in the structured light images acquired by the structured light camera 20 are not identical.

[0068] Please refer to Figure 5 In an alternative example, the second marker 12 is a square marker with dimensions of 60mm x 60mm, its surface printed with a combination of a checkerboard pattern and a QR code. The checkerboard is a 2x2 black and white alternating grid, with a white square in the upper left corner; each square measures 28mm x 28mm. Each white square contains a 25mm x 25mm QR code. The design of the second marker 12 is not limited to... Figure 5 The above is a limited list. Figure 5 The pattern shown is an ArUco marker, from the open-source ArUco library. Since the second marker 12 contains two QR codes at known locations, the structured light camera 20 can obtain the pose of the second marker 12 when scanning it. This is equivalent to knowing the pose of the robotic arm 33. Based on the second transformation matrix, the pose of the surgical instrument 34 can then be obtained.

[0069] In one example, a second marker 12 is affixed to the end of the robotic arm 33, and there should be no obstruction between the second marker 12 and the structured light camera 20. Optionally, the position of the second marker 12 on the robotic arm 33 is immovable. With this configuration, the second transformation matrix between the second marker 12 and the surgical instrument 34 is known (e.g., it can be obtained through calibration or preset design parameters). Preferably, the second marker 12 is affixed at the time of manufacture of the robotic arm 33, and the second transformation matrix between the second marker 12 and the surgical instrument 34 is pre-stored in the control device. Thus, after the structured light camera 20 captures the surgical area, the pose of the second marker 12 in the structured light camera coordinate system can be extracted. Multiplying this pose by the second transformation matrix yields the pose of the surgical instrument 34 in the structured light camera coordinate system, thereby completing the registration of the second marker 12, which in turn completes the registration of the surgical instrument 34.

[0070] Furthermore, the control device is also configured to acquire the first pose of the surgical instrument 34 recorded by the robotic arm controller, obtain the second pose of the surgical instrument 34 based on the pose of the second marker 12, and compensate the difference between the first pose and the second pose to the motion control unit of the robotic arm 33 of the surgical robot. Optionally, the main controller of the surgical robot also includes a robotic arm controller, which is used to control the movement of the robotic arm 33 and the surgical instrument 34 on the one hand, and to feed back and record the actual poses of the robotic arm 33 and the surgical instrument 34 on the other hand. In this way, by subtracting the first pose directly read from the robotic arm controller from the second pose obtained by scanning according to the structured light camera 20, and using this difference as a compensation amount in the control calculation of the robotic arm 33, closed-loop control of the robotic arm 33 can be realized, effectively eliminating the tracking and driving errors of the robotic arm 33, and further improving the tracking and driving accuracy.

[0071] Please refer to Figure 6 It illustrates the workflow of a preferred example of a structured light registration-based surgical system, which includes:

[0072] Step SA1: Preoperative medical imaging scan; understandably, before scanning the target object 35 in this step, the target object 35 has been affixed with the first marker 11;

[0073] Step SA2: Position the trolley 31 to ensure that the scanning range of the structured light camera 20 can cover the first marker 11 and the second marker 12;

[0074] Step SA3: Register robotic arm 33 and target object 35;

[0075] Step SA4: Automatic positioning of robotic arm 33; The main controller of the puncture surgery robot provides a recommended puncture path for the operator to select. After the operator confirms the path, the main controller controls the robotic arm 33 to perform automatic positioning through the motion control unit.

[0076] Step SA5: The structured light camera 20 acquires the pose of the second marker 12;

[0077] Step SA6: The structured light camera 20 acquires the pose of the first marker 11;

[0078] Step SA7: Track the pose changes of the first marker 11;

[0079] Step SA8: Determine whether the pose change of the first marker 11 in two consecutive frames exceeds the first preset value; if yes, prompt that the target object 35 has moved and return to step SA3; if no, proceed to step SA9.

[0080] Step SA9: Determine if the surgery is over. If yes, the surgery is over. If not, return to step SA6.

[0081] In another embodiment, please refer to Figure 7 and Figure 8 The structured light registration-based surgical system further includes a third marker 13; the third marker 13 is attached to the support device 32; the structured light camera 20 is also used to identify the pose of the third marker 13 in the structured light camera coordinate system, and scan the first marker 11 and the third marker 13 at a preset frequency; the control device is further configured to acquire a third transformation matrix between the poses of the first marker 11 and the third marker 13; and track the changes in the third transformation matrix. If the change in the third transformation matrix exceeds a second preset value in two consecutive scan frames, the target object 35 is prompted to move. In the structured light image acquired by the structured light camera 20, the images corresponding to the third marker 13 and the first marker 11 are not the same. The pose of the first marker 11 obtained by the structured light camera 20 reflects the pose of the target object 35, and the pose of the third marker 13 obtained by the structured light camera 20 reflects the pose of the supporting device 32. In each frame of the scan results obtained by the structured light camera 20 at a preset frequency, the third transformation matrix of the first marker 11 relative to the third marker 13 is calculated, and the difference between the third transformation matrix obtained in each calculation and the third transformation matrix obtained in the previous calculation is taken to obtain the movement matrix of the target object 35, which represents the movement direction and movement distance of the target object 35. The second preset value can be a preset allowable matrix. If the movement matrix of the target object 35 exceeds the preset allowable matrix, it is determined that the target object 35 has moved.

[0082] Please refer to Figure 8 In an alternative example, the third marker 13 is a rectangular marker with dimensions of 60mm x 120mm, its surface printed with a combination of a checkerboard pattern and a QR code. The checkerboard is a 2x4 black and white alternating grid, with a white square in the upper left corner. Each square measures 28mm x 28mm; each white square contains a 25mm x 25mm QR code. The design of the third marker 13 is not based on... Figure 8 The above is a limited list. Figure 8 The pattern shown is an ArUco marker, from the open-source ArUco library. Preferably, the third marker 13 is affixed to the area of ​​the support device 32 near the surgical site, and there should be no obstruction between the third marker 13 and the structured light camera 20. The affixing direction of the third marker 13 is arbitrary.

[0083] Furthermore, the control device is also configured to compensate for the change in the third transformation matrix to the motion control unit of the robotic arm 33 if the change in the third transformation matrix exceeds a second preset value in two consecutive scan frames. If the change in the third transformation matrix exceeds the second preset value in two consecutive scan frames, it can be considered that the target object 35 has moved. In this case, compensating for the change in the third transformation matrix to the motion control unit of the robotic arm 33 can make the movement of the robotic arm 33 follow the movement of the target object 35, thereby improving the final positioning accuracy.

[0084] Please refer to Figure 9 This illustrates the workflow of another preferred example of a structured light registration-based surgical system, which includes:

[0085] Step SB1: Preoperative medical imaging scan; understandably, before scanning the target object 35, the target object 35 has been affixed with the first marker 11;

[0086] Step SB2: Position the trolley 31 to ensure that the scanning range of the structured light camera 20 can cover the first marker 11, the second marker 12 and the third marker 13;

[0087] Step SB3: Register the robotic arm 33, support device 32, and target object 35;

[0088] Step SB4: Automatic positioning of robotic arm 33; The main controller of the puncture surgery robot provides a recommended puncture path for the operator to select. After the operator confirms the path, the main controller controls the robotic arm 33 to perform automatic positioning through the motion control unit.

[0089] Step SB5: The structured light camera 20 acquires the pose of the second marker 12;

[0090] Step SB6: The structured light camera 20 acquires the poses of the first marker 11 and the third marker 13;

[0091] Step SB7: Track the changes in the third transformation matrix;

[0092] Step SB8: Determine whether the change in the third conversion matrix in two consecutive frames exceeds the second preset value; if yes, compensate the change in the third conversion matrix to the motion control unit of the robotic arm 33 and return to execute step SB4; if no, proceed to step SB9.

[0093] Step SB9: Determine if the surgery is over. If yes, the surgery is over. If not, return to step SB6.

[0094] Based on the structured light registration-based surgical system described above, this embodiment of the invention also provides a structured light registration control method, which includes:

[0095] Step S1: Attach the first marker 11 to the target object 35;

[0096] Step S2: Obtain a medical image of the target object 35 with the first marker 11 attached;

[0097] Step S3: Use the structured light camera 20 to scan the first marker 11 and identify the pose of the first marker 11 in the structured light camera coordinate system;

[0098] Step S4: Based on the medical image, obtain the pose of the first marker 11 in the medical image coordinate system; then obtain the first transformation matrix between the structured light camera coordinate system and the medical image coordinate system; and register the first marker 11 based on the first transformation matrix.

[0099] Furthermore, the structured light-based registration control method also includes:

[0100] Step S5: The structured light camera scans the first marker at a preset frequency;

[0101] Step S6: Track the pose change of the first marker. If the pose change of the first marker exceeds a first preset value in the scanning results of two consecutive frames, then indicate that the target object has moved.

[0102] Optionally, in step S6, if the pose change of the first marker exceeds a first preset value in the scanning results of two consecutive frames, the pose change of the first marker is compensated to the motion control unit of the robotic arm.

[0103] Optionally, the structured light-based registration control method further includes:

[0104] Step S7: Attach the second marker to the end of the robotic arm;

[0105] Step S8: Use a structured light camera to scan the second marker and identify the pose of the second marker in the structured light camera coordinate system;

[0106] Step S9: Obtain the second transformation matrix between the second marker and the surgical instrument, and register the second marker based on the second transformation matrix.

[0107] Optionally, after the registration of the second marker is completed in step S9, the structured light-based registration control method further includes:

[0108] Step S10: Obtain the first pose of the surgical instrument recorded by the robotic arm controller, obtain the second pose of the surgical instrument based on the pose of the second marker, and compensate the difference between the first pose and the second pose to the motion control unit of the robotic arm of the surgical robot.

[0109] Optionally, the structured light-based registration control method further includes:

[0110] Step S11: Attach the third marker to the carrier device;

[0111] Step S12: Use a structured light camera to scan the third marker, identify the pose of the third marker in the structured light camera coordinate system, and scan the first marker and the third marker at a preset frequency;

[0112] Step S13: Obtain the third transformation matrix of the pose of the first marker and the pose of the third marker; and track the changes of the third transformation matrix. If the change of the third transformation matrix exceeds the second preset value in the scanning results of two consecutive frames, then indicate that the target object has moved.

[0113] Optionally, in step S13, if the change in the third transformation matrix exceeds the second preset value in the scanning results of two consecutive frames, the change in the third transformation matrix is ​​compensated to the motion control unit of the robotic arm.

[0114] The specific implementation principles of steps S1 to S13 above can be found in the text above, and will not be repeated here. It should be noted that steps S1 to S13 are not necessarily all required to be executed; rather, they can be selectively executed as needed, and their execution order is not limited to the sequential order of steps S1 to S13. Those skilled in the art can understand this based on the text above.

[0115] Furthermore, embodiments of the present invention also provide a readable storage medium storing a program thereon, which, when executed, implements the steps of the structured light-based registration control method as described above. Even further, embodiments of the present invention also provide a computer device including a processor and the readable storage medium as described above, the processor being used to execute the program stored on the readable storage medium. The readable storage medium can be set independently or integrated into a structured light-based registration surgical system, such as integrating it into a control device; the present invention is not limited thereto.

[0116] In summary, the structured light registration-based surgical system, structured light registration control method, and readable storage medium provided by this invention include: a first marker, a structured light camera, and a control device; the first marker is attached to a target object; the structured light camera is used to identify the pose of the first marker in the structured light camera coordinate system; the control device is configured to acquire a medical image of the target object with the first marker attached, obtain the pose of the first marker in the medical image coordinate system; then acquire a first transformation matrix between the structured light camera coordinate system and the medical image coordinate system; and register the first marker based on the first transformation matrix. With this configuration, by installing a specific first marker on the target object, the structured light camera no longer needs to perform overall point cloud reconstruction of the target object, but only needs to identify the first marker, significantly increasing the imaging frequency. Compared to traditional NDI tracking and navigation, the structured light registration control scheme greatly reduces costs, and compared to traditional structured light positioning, the setting of the first marker accelerates the calculation speed, greatly improving real-time performance and accuracy. The first marker is attached to the target object, avoiding the invasive positioning required by traditional NDI tracking and navigation through bone screws, and can basically achieve non-invasive registration, greatly reducing damage to patients.

[0117] It should be noted that the above embodiments can be combined with each other. The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A surgical system based on structured light registration, characterized in that, include: First marker, second marker, surgical robot, structured light camera and control device; The first marker is used to be attached to the target object; The structured light camera is used to identify the pose of the first marker in the structured light camera coordinate system; The control device is configured to acquire a medical image of the target object with the first marker attached, obtain the pose of the first marker in the medical image coordinate system, then acquire a first transformation matrix between the structured light camera coordinate system and the medical image coordinate system, and register the first marker based on the first transformation matrix. The surgical robot includes a robotic arm on which surgical instruments are attached; The second marker is used to be attached to the end of the robotic arm; The structured light camera is also used to identify the pose of the second marker in the structured light camera coordinate system; The control device is further configured to acquire a second conversion matrix between the second marker and the surgical instrument, and register the second marker based on the second conversion matrix; Wherein, in the structured light image acquired by the structured light camera, the images corresponding to the second marker and the first marker are not the same; The first marker includes a radiopaque marker for medical image scanning and an image marker for structured light camera scanning, and the relative positions of the radiopaque marker and the image marker are fixed.

2. The surgical system based on structured light registration according to claim 1, characterized in that, The structured light camera scans the first marker at a preset frequency, and the control device is configured to track the pose change of the first marker. If the pose change of the first marker exceeds a first preset value in the scanning results of two consecutive frames, the device will indicate that the target object has moved.

3. The surgical system based on structured light registration according to claim 2, characterized in that, The structured light registration-based surgical system also includes a surgical robot, which includes a robotic arm. The control device is further configured to compensate the motion control unit of the robotic arm for the change in pose of the first marker if the change in pose of the first marker exceeds a first preset value in two consecutive frames of scanning results.

4. The surgical system based on structured light registration according to claim 1, characterized in that, The control device is further configured to acquire the first pose of the surgical instrument recorded by the robotic arm controller, obtain the second pose of the surgical instrument based on the pose of the second marker, and compensate the difference between the first pose and the second pose to the motion control unit of the robotic arm of the surgical robot.

5. The surgical system based on structured light registration according to claim 1, characterized in that, The structured light registration-based surgical system also includes a support device and a third marker; The third marker is used to be attached to the carrier device; The structured light camera is also used to identify the pose of the third marker in the structured light camera coordinate system, and to scan the first marker and the third marker at a preset frequency; The control device is further configured to acquire a third transformation matrix between the pose of the first marker and the pose of the third marker; and track the changes in the third transformation matrix. If the change in the third transformation matrix exceeds a second preset value in the scanning results of two consecutive frames, the device will indicate that the target object has moved. In the structured light image acquired by the structured light camera, the third marker is not the same as the image corresponding to the first marker.

6. The surgical system based on structured light registration according to claim 5, characterized in that, The structured light registration-based surgical system also includes a surgical robot, which includes a robotic arm. The control device is further configured to compensate the change in the third transformation matrix to the motion control unit of the robotic arm if the change in the third transformation matrix exceeds a second preset value in two consecutive scan frames.

7. A registration control method based on structured light, characterized in that, include: Attach the first marker to the target object; Acquire a medical image of the target object with the first marker affixed to it; The first marker is scanned using a structured light camera, and its pose in the structured light camera coordinate system is identified. Based on the medical image, the pose of the first marker in the medical image coordinate system is obtained; then, a first transformation matrix between the structured light camera coordinate system and the medical image coordinate system is obtained; and the first marker is registered based on the first transformation matrix. Attach the second marker to the end of the robotic arm; The second marker is scanned using a structured light camera, and its pose in the structured light camera coordinate system is identified. Obtain the second transformation matrix between the second marker and the surgical instrument, and register the second marker based on the second transformation matrix; The first marker includes a imaging marker for medical image scanning and an image marker for structured light camera scanning, and the relative positions of the imaging marker and the image marker are fixed.

8. The registration control method based on structured light according to claim 7, characterized in that, The structured light-based registration control method further includes: The structured light camera scans the first marker at a preset frequency; The pose change of the first marker is tracked. If the pose change of the first marker exceeds a first preset value in the scan results of two consecutive frames, it is indicated that the target object has moved.

9. The registration control method based on structured light according to claim 8, characterized in that, If the pose change of the first marker exceeds a first preset value in the scanning results of two consecutive frames, the pose change of the first marker will be compensated to the motion control unit of the robotic arm.

10. The registration control method based on structured light according to claim 9, characterized in that, The structured light-based registration control method further includes: acquiring the first pose of the surgical instrument recorded by the robotic arm controller, obtaining the second pose of the surgical instrument based on the pose of the second marker, and compensating the difference between the first pose and the second pose to the motion control unit of the robotic arm of the surgical robot.

11. The registration control method based on structured light according to claim 7, characterized in that, The structured light-based registration control method further includes: Affix the third marker to the carrier device; The third marker is scanned using a structured light camera to identify its pose in the structured light camera coordinate system, and the first marker and the third marker are scanned at a preset frequency. Obtain the third transformation matrix of the pose of the first marker and the pose of the third marker; and track the changes of the third transformation matrix. If the change of the third transformation matrix exceeds a second preset value in the scanning results of two consecutive frames, the target object is prompted to move.

12. The registration control method based on structured light according to claim 11, characterized in that, If the change in the third transformation matrix exceeds the second preset value in the scan results of two consecutive frames, the change in the third transformation matrix will be compensated to the motion control unit of the robotic arm.

13. A readable storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the steps of the registration control method based on structured light according to any one of claims 7 to 12.

Images