Robotic arm positioning method, readable storage medium and surgical robot system

Abstract

The present invention provides a robotic arm positioning method, a readable storage medium, and a surgical robot system. The robotic arm positioning method includes: obtaining a virtual three-dimensional model of a surgical object, pre-drilled hole locations on the surgical object, and a robotic arm on a patient-side operating table; obtaining a positioning configuration of an adjustment arm of the robotic arm based on the virtual three-dimensional model; performing virtual-real fusion coordinate registration of a patient-side operating table coordinate system and a real-body surface coordinate system of the surgical object based on the current positioning configuration of the adjustment arm; obtaining a positioning path plan for the adjustment arm based on the real-time position of an AR device and the positioning configuration of the adjustment arm; and prompting the positioning path plan via the AR device. This configuration can conveniently assist and guide the operator in positioning the adjustment arm, thereby improving the accuracy and efficiency of preoperative positioning.

Description

Technical Field

[0001] The present invention relates to the field of robot-assisted surgery systems and methods, and in particular to a robotic arm positioning method, a readable storage medium, and a surgical robot system. Background Art

[0002] The emergence of surgical robots aligns with the development trend of precision surgery. Surgical robots have become a powerful tool to assist doctors in completing surgeries. Their design concept is to precisely perform complex surgical procedures using minimally invasive methods. Faced with the various limitations of traditional surgery, surgical robots have been developed to replace them. Surgical robots transcend the limitations of the human eye and utilize stereoscopic imaging technology to present internal organs more clearly to the operator. In areas previously inaccessible to human hands, robotic arms can rotate, move, swing, and grip 360 degrees while avoiding shaking. Because surgical robots cause minimal harm to patients, resulting in less bleeding and faster recovery, they significantly shorten postoperative hospital stays and significantly improve survival and recovery rates. They are favored by both doctors and patients and are now widely used as high-end medical devices in various clinical surgeries.

[0003] Existing surgical robots often position the robotic arm before surgery by manually pulling or adjusting it using the control end. It is impossible to determine whether the position is in the expected optimal position. The preoperative positioning operation is complicated and inefficient. Summary of the Invention

[0004] The purpose of the present invention is to provide a robotic arm positioning method, a readable storage medium and a surgical robot system to solve the problem of inaccurate preoperative positioning of existing surgical robots.

[0005] To solve the above technical problems, according to a first aspect of the present invention, a method for positioning a robotic arm is provided, which includes:

[0006] Acquire a virtual three-dimensional model of the surgical object, pre-drilled hole locations on the surgical object, and a robotic arm of a patient-side surgical operating table;

[0007] Obtaining a positioning configuration of an adjustment arm of the robotic arm according to the virtual three-dimensional model;

[0008] According to the current positioning configuration of the adjustment arm, the coordinate system of the patient's operating table is aligned with the real body surface coordinate system of the surgical object by virtual-real fusion coordinate registration;

[0009] Obtaining a positioning path plan for the adjustment arm according to the real-time position of the AR device and the positioning configuration of the adjustment arm; and

[0010] The positioning path planning is prompted by the AR device.

[0011] Optionally, the robotic arm positioning method further includes:

[0012] Obtaining the actual position of the adjustment arm after positioning according to the positioning path planning;

[0013] Obtaining a planned position of the adjustment arm calculated according to the positioning path planning;

[0014] Calculate the deviation between the planned position and the actual position. If the deviation exceeds the preset standard, execute again: based on the current positioning configuration of the adjustment arm, perform virtual-real fusion coordinate alignment on the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object; based on the real-time position of the AR device, obtain the positioning path planning of the adjustment arm; and prompt the positioning path planning step through the AR device; if the deviation is within the preset standard, determine that the positioning of the adjustment arm is completed.

[0015] Optionally, in the robotic arm positioning method, a virtual three-dimensional model of the surgical object, the pre-drilling position and the robotic arm is established based on parallax information fed back by a binocular vision device.

[0016] Optionally, in the robotic arm positioning method, a method for performing virtual-real fusion coordinate registration of the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object according to the current positioning configuration of the adjustment arm includes:

[0017] Establishing coordinate mapping relationships between the binocular vision device coordinate system and the patient-side operating table coordinate system and the AR device coordinate system in the world coordinate system;

[0018] According to the mapping relationship between the real body surface coordinate system of the surgical object and the coordinate system of the binocular vision device, a coordinate mapping relationship between the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object is established.

[0019] Optionally, in the robotic arm positioning method, the relative coordinate relationship between the binocular vision device coordinate system and the AR device coordinate system is fixed.

[0020] Optionally, in the robotic arm positioning method, the method of prompting the positioning path planning by using an AR device includes:

[0021] The AR device displays a plurality of planned positions of the adjustment arm and displays a plurality of positioning paths of the adjustment arm for selection to reach the planned positions;

[0022] Prompt the joint of the selected adjustment arm that needs to be adjusted; and

[0023] Prompts the amount of adjustment required for the joint.

[0024] Optionally, in the robotic arm positioning method, the method of prompting the positioning path planning through an AR device further includes:

[0025] Obtain the adjustment difference value fed back during the adjustment process of the joint that needs to be adjusted;

[0026] If the adjustment difference exceeds the range of the preset target value, the step of prompting the adjustment amount of the joint to be adjusted is performed again until the adjustment difference is within the range of the preset target value.

[0027] Optionally, in the robotic arm positioning method, the method of prompting the positioning path planning through an AR device further includes:

[0028] After detecting that the positioning of all the adjustment arms is completed, a prompt is given that the current adjustment is completed, and the operation time is displayed and / or recorded.

[0029] Optionally, in the robotic arm positioning method, in the step where the AR device displays the planned positions of multiple adjustment arms and displays multiple positioning paths of the adjustment arms for selection, the nearest adjustment arm is recommended as the starting adjustment arm based on the real-time position of the AR device and the relative positions between the multiple adjustment arms.

[0030] Optionally, in the robotic arm positioning method, before prompting the joints that need to be adjusted for the selected adjustment arm, the joint with the highest collision probability is recommended as the starting joint for adjustment based on the positions of the joints of the selected adjustment arm and the adjacent joints.

[0031] Optionally, the current positioning configuration of the adjustment arm is obtained based on historical surgical data under the same surgical procedure and the current virtual three-dimensional model.

[0032] In order to solve the above technical problems, according to the second aspect of the present invention, a readable storage medium is further provided, on which a program is stored. When the program is executed, the robot arm positioning method as described above is implemented.

[0033] In order to solve the above technical problems, according to the third aspect of the present invention, a surgical robot system is also provided, which includes a patient-side surgical operating table, an AR device and a control device. The patient-side surgical operating table includes a robotic arm, and the robotic arm includes an adjustment arm. The control device uses the robotic arm positioning method as described above to control the AR device to prompt the positioning path planning of the adjustment arm.

[0034] Optionally, the surgical robot system also includes a binocular vision device, which is used to obtain parallax information of the surgical object, the pre-drilling position and the robotic arm, and the control device is used to establish a virtual three-dimensional model of the surgical object, the drilling position and the robotic arm based on the parallax information.

[0035] Optionally, in the surgical robot, the binocular vision device and the AR device are integrally or separately arranged.

[0036] Optionally, in the surgical robot, the robotic arm also includes a tool arm, one end of the tool arm is used to connect to the surgical instrument, and the other end is used to connect to the adjustment arm, the adjustment arm includes multiple joints, and the control device controls the AR device to prompt the joint that needs to be adjusted in the selected adjustment arm according to the positioning path planning.

[0037] Optionally, in the surgical robot, the multiple joints of the adjustment arm include at least three rotating joints and one movable joint, and the control device controls the AR device to prompt at least one of the rotating joints and movable joints that need to be adjusted for the selected adjustment arm according to the positioning path planning.

[0038] In summary, in the robotic arm positioning method, readable storage medium and surgical robot system provided by the present invention, the robotic arm positioning method includes: obtaining a virtual three-dimensional model of the surgical object, the pre-punched position on the surgical object and the robotic arm of the patient-side surgical operating table; obtaining the positioning configuration of the adjustment arm of the robotic arm based on the virtual three-dimensional model; performing virtual-real fusion coordinate alignment on the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object based on the current positioning configuration of the adjustment arm; obtaining the positioning path planning of the adjustment arm based on the real-time position of the AR device and the positioning configuration of the adjustment arm; and prompting the positioning path planning through the AR device.

[0039] With such a configuration, based on the virtual-reality fusion coordinate registration and according to the real-time position of the AR device, the obtained positioning path planning of the adjustment arm will be prompted through the AR device, which can conveniently assist and guide the operator to perform the positioning operation of the adjustment arm, thereby improving the accuracy and execution efficiency of preoperative positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0041] Figure 1 is a schematic diagram of an application scenario of a surgical robot according to an embodiment of the present invention;

[0042] Figure 2is a schematic diagram of a patient-side surgical operating table according to an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of an adjustment arm joint of a robotic arm according to an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of the acquisition principle of a binocular vision device according to an embodiment of the present invention;

[0045] Figure 5 1 is a schematic diagram of the principle of a binocular vision device according to an embodiment of the present invention;

[0046] Figure 6 is a flow chart of a method for positioning a robotic arm according to an embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of an application scenario of path planning according to an embodiment of the present invention;

[0048] Figure 8 2 is a schematic diagram of the principle of virtual-real fusion coordinate registration according to an embodiment of the present invention, wherein the binocular vision device and the AR device are separately provided;

[0049] Figure 9 This is a schematic diagram of a binocular vision device and an AR device that are separately configured according to an embodiment of the present invention;

[0050] Figure 10 2 is a schematic diagram of the principle of virtual-real fusion coordinate registration according to an embodiment of the present invention, wherein a binocular vision device and an AR device are integrated;

[0051] Figure 11 This is a schematic diagram of an integrated arrangement of a binocular vision device and an AR device according to an embodiment of the present invention;

[0052] Figure 12 Schematic diagram of the coordinate mapping relationship between a binocular vision device and an AR device according to an embodiment of the present invention;

[0053] Figure 13 This is a flowchart of positioning path planning using an AR device according to an embodiment of the present invention;

[0054] Figure 14 is a schematic diagram of a path planning method according to an embodiment of the present invention;

[0055] Figure 15 This is a schematic diagram of a scenario in which an AR device prompts positioning path planning according to an embodiment of the present invention;

[0056] Figure 16 is a schematic diagram of a real-time path planning method according to an embodiment of the present invention;

[0057] Figure 17FIG. 4 is a schematic diagram of an optimal path selection method according to an embodiment of the present invention.

[0058] In the attached figure:

[0059] 1-First rotational joint 1; 2-Second rotational joint; 3-Moving joint; 4-Third rotational joint; 10-Doctor-side console; 20-Patient-side operating table; 21-Robot arm; 21a-Adjustment arm; 21b-Tool arm; 30-AR device; 40-Surgical object; 50-Auxiliary equipment; 71-Binocular vision device; 711-Left camera; 712-Right camera. DETAILED DESCRIPTION

[0060] To make the objects, advantages, and features of the present invention more clearly apparent, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale. They are only used to conveniently and clearly assist in illustrating the purposes of the embodiments of the present invention. In addition, the structures shown in the drawings are often part of the actual structure. In particular, different drawings may need to illustrate different focuses and sometimes use different scales.

[0061] As used in the present invention, the singular forms "a", "an", and "the" include plural referents. The term "or" is generally used to include "and / or". The term "several" is generally used to include "at least one". The term "at least two" is generally used to include "two or more". In addition, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, the features specified as "first", "second", and "third" may explicitly or implicitly include one or at least two of such features. The term "proximal end" generally refers to the end closest to the operator, and the term "distal end" generally refers to the end closest to the patient, i.e., the end of the lesion. "One end" and "the other end" as well as "proximal end" and "distal end" generally refer to two corresponding parts, which include not only endpoints. The terms "mounted", "connected", and "connected" should be understood in a broad sense, for example, they can be fixedly connected, detachably connected, or integrated; they can be directly connected or indirectly connected through an intermediate medium; they can be internal communication between two elements or an interactive relationship between two elements. In addition, as used in the present invention, "one element is disposed on another element" generally only indicates that there is a connection, coupling, cooperation, or transmission relationship between the two elements, and the connection, coupling, cooperation, or transmission between the two elements may be direct or indirect through an intermediate element. It should not be understood as indicating or implying a spatial positional relationship between the two elements. That is, one element can be in any orientation, such as inside, outside, above, below, or to one side of another element, unless the content clearly indicates otherwise. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0062] The main purpose of the present invention is to provide an AR (Augmented Reality) device-assisted robotic arm positioning method, a readable storage medium, and a surgical robot to solve the problem of inaccurate preoperative positioning of existing surgical robots.

[0063] The following description is given with reference to the accompanying drawings.

[0064] Please refer to Figures 1 to 17 ,in, Figure 1 is a schematic diagram of an application scenario of a surgical robot according to an embodiment of the present invention; Figure 2 is a schematic diagram of a patient-side surgical operating table according to an embodiment of the present invention; Figure 3 is a schematic diagram of an adjustment arm joint of a robotic arm according to an embodiment of the present invention; Figure 4 Schematic diagram of the acquisition principle of a binocular vision device according to an embodiment of the present invention; Figure 5 1 is a schematic diagram of the principle of a binocular vision device according to an embodiment of the present invention; Figure 6is a flow chart of a method for positioning a robotic arm according to an embodiment of the present invention; Figure 7 is a schematic diagram of an application scenario of path planning according to an embodiment of the present invention; Figure 8 2 is a schematic diagram of the principle of virtual-real fusion coordinate registration according to an embodiment of the present invention, wherein the binocular vision device and the AR device are separately provided; Figure 9 This is a schematic diagram of a binocular vision device and an AR device that are separately configured according to an embodiment of the present invention; Figure 10 2 is a schematic diagram of the principle of virtual-real fusion coordinate registration according to an embodiment of the present invention, wherein a binocular vision device and an AR device are integrated; Figure 11 This is a schematic diagram of an integrated arrangement of a binocular vision device and an AR device according to an embodiment of the present invention; Figure 12 2 is a schematic diagram of a coordinate mapping relationship between a binocular vision device and an AR device according to an embodiment of the present invention; Figure 13 This is a flowchart of positioning path planning using an AR device according to an embodiment of the present invention; Figure 14 is a schematic diagram of a path planning method according to an embodiment of the present invention; Figure 15 This is a schematic diagram of a scenario in which an AR device prompts positioning path planning according to an embodiment of the present invention; Figure 16 is a schematic diagram of a real-time path planning method according to an embodiment of the present invention; Figure 17 FIG. 4 is a schematic diagram of an optimal path selection method according to an embodiment of the present invention.

[0065] An embodiment of the present invention provides a surgical robot. Figure 1 The following illustrates an exemplary embodiment of a surgical robot application scenario. The surgical robot of the present invention has no particular limitations on its application environment. The surgical system includes a doctor's console 10, a patient-side operating table 20, an AR device 30 (such as AR glasses), and a control device (not shown). The patient-side operating table 20 includes at least one robotic arm, on which surgical instruments or an endoscope are mounted. An operator (such as a doctor) controls the doctor's console 10 to drive the robotic arm, thereby manipulating the surgical instruments and performing the surgery. Preferably, the surgical robot also includes a bed and other auxiliary equipment 50 (such as a display trolley, sterile table, ventilator, or testing equipment). The control device, which is communicatively connected to the doctor's console 10 and the patient-side operating table 20, is used to control the various components of the system and exchange information between them. The control device, such as a processor, can be integrated or divided into multiple components and can be located in the same or distributed locations.

[0066] Figure 2A patient-side surgical operating table 20 is shown as an example, which includes four robotic arms 21, each of which includes an adjustment arm 21a and a tool arm 21b, wherein the tool arm 21b is arranged at the distal end of the adjustment arm 21a, that is, one end (distal end) of the tool arm 21b is used to connect with the surgical instrument (or endoscope), and the other end (proximal end) is used to connect with the adjustment arm 21a; the adjustment arm 21a can adopt different positioning methods according to different surgical procedures and patient signs, so that the tool arm 21b can accurately point to the lesion of the surgical object and enable the operator to complete the operation in the optimal operating space, reducing the probability of interference generated during the operation of the tool arm 21b. The adjustment arm 21a includes multiple joints, Figure 2 In the illustrated example, the adjustment arm 21a includes four joints, namely three rotation joints (the three rotation joints are the first rotation joint 1, the second rotation joint 2 and the third rotation joint 4) and one moving joint 3. Figure 3 As shown. Therefore, there are four joint positioning configurations for each adjustment arm 21a:

[0067] 1. The first rotational joint 1 rotates clockwise or counterclockwise by an angle θ1 from the zero position (referring to the default initial position) and remains within the limit; optionally, the rotation of the first rotational joint 1 can adjust the left and right rotation of the tool arm 21b;

[0068] 2. The second rotational joint 2 rotates clockwise / counterclockwise by an angle θ2 from the zero position and remains within the limit; optionally, the rotation of the second rotational joint 2 can adjust the up and down rotation of the tool arm 21b;

[0069] 3. The mobile joint 3 is linearly translated from the zero position by a distance s and remains within the limit; optionally, the translation of the mobile joint 3 can adjust the forward and backward translation of the tool arm 21b;

[0070] 4. The third rotational joint 4 rotates from the zero position in clockwise / counterclockwise directions by an angle of θ3 and remains within the limit; optionally, the rotation of the third rotational joint 4 can adjust the left and right rotation of the tool arm 21b.

[0071] For easier understanding, please refer to Figure 15 The diagram shows a scene where the AR device prompts the positioning path planning. Figure 15The diagram shows the joint adjustment sequence and adjustment amounts for adjustment arm 21a in an exemplary embodiment. ① indicates that the joint adjusted in the first step is the first rotational joint 1, with an adjustment amount of θ1; ② indicates that the joint adjusted in the second step is the third rotational joint 4, with an adjustment amount of θ3; and ③ indicates that the joint adjusted in the third step is the mobile joint 3, with an adjustment amount of s. In this example, the second rotational joint 2 remains in its initial position and is not adjusted. It will be understood that adjustments to the three rotational joints and one mobile joint on adjustment arm 21a will cause changes in the posture of tool arm 21b, thereby adjusting the posture of tool arm 21b to the most appropriate position.

[0072] Need to explain, Figure 2 and Figure 3 The patient-side surgical operating table 20 and its robotic arm 21 shown are only an example, not a limitation of the patient-side surgical operating table 20. Those skilled in the art may configure the number, structure, number of joints and joint form of the robotic arm 21 differently according to actual conditions, and the present invention is not limited to this. Based on the above-mentioned surgical robot, this embodiment also provides a surgical robot system, which includes the surgical robot and AR device 30 as described above, and the control device is communicatively connected to the AR device 30 to control the AR device 30 to prompt the positioning path planning of the adjustment arm. The control device controls the AR device 30 to prompt the joint that needs to be adjusted in the selected adjustment arm 21a according to the positioning path planning. For Figure 2 and Figure 3 In the illustrated example, the control device controls the AR device 30 to prompt at least one of the rotational joint and the movable joint of the selected adjustment arm 21 a to be adjusted according to the positioning path planning.

[0073] Furthermore, the surgical robot also includes a binocular vision device 71, such as Figure 4 As shown, the binocular vision device 71 is used to obtain parallax information of the surgical object 40, the pre-punching position on the surgical object and the robotic arm 21, and the control device is used to establish a virtual three-dimensional model of the surgical object 40, the pre-punching position and the robotic arm 21 based on the parallax information.

[0074] like Figure 4As shown, the binocular vision device 71 acquires image information via a binocular camera. Based on the principle of parallax, it captures two images of the object being measured from different angles to obtain two-dimensional information. It then establishes a correspondence between feature points and calculates positional deviations, thereby enabling three-dimensional reconstruction of the surgical object 40, the pre-drilled hole locations, and the robotic arm 21. Of course, in other embodiments, pre-operative CT, MRI, or other tomographic imaging techniques can also be used to obtain the virtual three-dimensional model. Based on this virtual three-dimensional model, the configuration of the adjustment arm 21a can be determined by inverse geometry using the surgical object and the pre-drilled hole locations.

[0075] Please refer to Figure 5 , which is a schematic diagram of the principle of the binocular vision device 71. The principle of binocular vision is as follows: In an exemplary embodiment, the binocular camera includes a left camera 711 and a right camera 712. The distance between the center of the left camera 711 and the center of the right camera 712 (similar to the pupil distance of the human eye) is b. A reference plane c parallel to the line connecting the binocular centers is set at a distance f in front of the binocular camera. The intersection point of the line connecting the left camera 711 and the point to be measured P with the reference plane c is A1, and the intersection point of the line connecting the right camera 712 and the point to be measured P with the reference plane c is A2. The left optical axis B1 of the left camera 711 extends perpendicular to the reference plane c through the left camera 711, and the right optical axis B2 of the right camera 712 extends perpendicular to the reference plane c through the right camera 712. The distance between point A1 and the left optical axis B1 is , the distance between point A2 and the right optical axis B2 is , establish a rectangular coordinate system with A1 as the origin, the left optical axis B1 as the z-axis, the x-axis and y-axis are both on the reference plane c, the x-axis is parallel to the line connecting the centers of the binoculars, and the y-axis is perpendicular to the line connecting the centers of the binoculars. Then the coordinates (x, y, z) of point P can be derived by the following formula:

[0076] ; ; ;

[0077] Based on the above surgical robot system, please refer to Figure 6 This embodiment provides a method for positioning a robotic arm, and the surgical robot system uses the method to control the AR device 30 to prompt the adjustment arm positioning path planning. The method includes:

[0078] Step S1: Obtain a virtual three-dimensional model of the surgical object 40, the pre-drilling position on the surgical object, and the robotic arm 21 of the patient-side surgical operating table 20; preferably, the virtual three-dimensional model of the surgical object 40, the pre-drilling position, and the robotic arm 21 is established based on the parallax information fed back by the binocular vision device 71.

[0079] Step S2: Based on the virtual three-dimensional model, the positioning configuration of the adjustment arm 21a of the robot arm 21 is obtained; those skilled in the art can obtain the positioning configuration of the robot arm 21 based on existing technologies, and further, obtain the positioning configuration of the adjustment arm 21a of the robot arm 21. For example, the positioning configuration of the adjustment arm 21a of the robot arm 21 can be obtained based on historical surgical data under the same surgical procedure and combined with the current virtual three-dimensional model. This step can be performed by the control device. For details, please refer to Figure 7 Through the positioning of the binocular vision device 71, the current position of the robot arm 21 and the modeling position in the virtual three-dimensional model are combined and calculated to obtain the positioning configuration.

[0080] Step S3: performing virtual-real fusion coordinate registration on the patient-side operating table coordinate system and the real body surface coordinate system of the surgical object 40 according to the current positioning configuration of the adjustment arm 21 a;

[0081] Step S4: Based on the real-time position of the AR device 30 and the position configuration of the adjustment arm 21a, a positioning path plan for the adjustment arm 21a is obtained. Specifically, the positioning path plan may be adjusted in real time due to differences in the real-time position of the AR device 30 and the position configuration of the adjustment arm 21a. Those skilled in the art can obtain the positioning path plan for the adjustment arm 21a based on the real-time position of the AR device 30 and the position configuration of the adjustment arm 21a according to existing techniques, and this will not be further described here.

[0082] Step S5: Prompt the positioning path planning through the AR device 30.

[0083] With such configuration, based on the virtual-real fusion coordinate registration, the obtained positioning path planning of the adjustment arm 21a is prompted through the AR device 30 according to the real-time position of the AR device 30, which can conveniently assist and guide the operator to perform the positioning operation of the adjustment arm 21a, thereby improving the accuracy and execution efficiency of preoperative positioning.

[0084] like Figure 8 and Figure 9 As shown, in an alternative embodiment, the binocular vision device 71 and the AR device 30 are separately provided, and step S3 specifically includes:

[0085] A coordinate mapping relationship is established between the binocular vision device coordinate system (X7 / Y7 / Z7) and the patient-side surgical operating table coordinate system (X2 / Y2 / Z2) and the AR device coordinate system (X3 / Y3 / Z3) in the world coordinate system (X0 / Y0 / Z0). Based on the mapping relationship between the real surface coordinate system (X4 / Y4 / Z4) of the surgical object 40 and the binocular vision device coordinate system (X7 / Y7 / Z7) (if calibrated before this step), a coordinate mapping relationship between the patient-side surgical operating table coordinate system (X2 / Y2 / Z2) and the real surface coordinate system (X4 / Y4 / Z4) of the surgical object 40 is established.

[0086] For further information, please refer to Figure 10 and Figure 11 In some embodiments, the binocular vision device 71 is integrated with the AR device 30, and the relative coordinate relationship between the binocular vision device coordinate system (X7 / Y7 / Z7) and the AR device coordinate system (X3 / Y3 / Z3) is fixed. In this case, in step S3, the coordinate mapping relationship between the binocular vision device coordinate system (X7 / Y7 / Z7) and the AR device coordinate system (X3 / Y3 / Z3) can be established based on the design configuration files of the binocular vision device 71 and the AR device 30. Figure 12 As shown, the relative coordinate relationship between the AR device 30 and the binocular vision device 71 is fixed. That is, the binocular vision device coordinate system (X7 / Y7 / Z7) can be mapped to the AR device coordinate system (X3 / Y3 / Z3) through the mechanical position, and the binocular vision device coordinate system (X7 / Y7 / Z7) and the world coordinate system (Z0 / X0 / Y0) can be mapped through the rotation matrix R and the translation vector t. The basic principles are as follows:

[0087]

[0088] Wherein, M1 is a transformation matrix. Those skilled in the art can understand the establishment of transformation matrices and mapping relationships between coordinate systems based on existing technologies, and will not be further described here.

[0089] Optionally, the robotic arm positioning method further includes:

[0090] Step S6: Obtain the actual position of the adjustment arm 21a after positioning according to the positioning path planning; step S6 is preferably performed after step S5.

[0091] Step S7: Obtain the planned position of the robotic arm 21 calculated according to the positioning path planning; step S7 is preferably performed in step S4, step S5 or after step S5.

[0092] Step S8: Calculate the deviation between the planned position and the actual position. If the deviation exceeds the preset standard, execute steps S3 to S5 again. If the deviation is within the preset standard, determine that the positioning of the robot arm 21 is completed.

[0093] Steps S6 to S8 are the positioning detection steps. The operator can adjust the positioning of the adjustment arm 21a according to the positioning path plan prompted by the AR device 30 in step S5. After the positioning is completed, the operator detects whether the deviation between the planned position and the actual position exceeds the preset standard. If so, it indicates that the positioning is not complete, and the process returns to step S3 and repeats. If the deviation does not exceed the preset standard, the positioning of the adjustment arm 21a is determined to be complete. Preferably, for example, the AR device 30 can prompt the operator whether the adjustment is complete or not. It is understood that those skilled in the art can set the preset standard to meet actual needs.

[0094] Preferably, please refer to Figure 13 , step S5 includes:

[0095] Step S51: The AR device 30 displays multiple planned positions of the adjustment arm 21a, and displays multiple positioning paths of the adjustment arm 21a for selection to reach the planned positions; after turning on the AR device 30, the AR device 30 displays the multiple planned positions of the adjustment arms 21a as virtual graphics, integrates them into the physical image of the adjustment arm 21a, and forms a positioning path for the operator to select.

[0096] Step S52: Prompt the joints of the selected adjustment arm 21a that need to be adjusted; after the operator selects the adjustment arm 21a to be adjusted, the AR device 30 will prompt the joints of the selected adjustment arm 21a that need to be adjusted.

[0097] Step S53: Prompt the adjustment amount of the joint that needs to be adjusted. The AR device 30 also prompts the adjustment amount of the joint that needs to be adjusted, such as displaying it in the form of data for the operator's reference. The adjustment amount of the joint here may include the position and posture of the joint adjustment, which may vary depending on the different forms of joint movement. For example, rotating the joint prompts the rotation direction and angle, and moving the joint prompts the displacement direction and distance, etc. Please refer to Figure 14 In one exemplary embodiment, the actual joint space motion position of the adjustment arm 21a can be converted between the actual adjustment arm 21a and the inverse kinematics modeling data. For example, for the first revolute joint 1, the second revolute joint 2, and the third revolute joint 4, the positioning is adjusted by angle, and the planned positions are mapped to the same plane. The adjustment angles θ and φ based on the current position are calculated based on the mechanical structure data. For the mobile joint 3, the movement path can be planned based on the displacement value S between the actual position and the modeled position.

[0098] Figure 15 A schematic diagram of a scene in which the AR device 30 displays the planned positions of multiple adjustment arms 21 a is shown, which roughly represents what the operator can observe on the AR device 30 .

[0099] Furthermore, after step S53, step S5 further includes:

[0100] Step S54: obtaining the adjustment difference value fed back during the adjustment process of the joint to be adjusted;

[0101] Step S55: If the adjustment difference exceeds the preset standard value, the step of prompting the adjustment amount of the joint to be adjusted is performed again until the adjustment difference is within the range of the preset standard value.

[0102] Steps S54 and S55 are joint adjustment detection steps. During the operator's adjustment of the positioning of the adjustment arm 21a, each joint of the adjustment arm 21a will generate an adjustment difference during the adjustment process. If the adjustment difference exceeds the preset target value range, it indicates that the positioning is not complete, and the process returns to step S53 and repeats, continuing to prompt the operator with the joint adjustment amount for the operator to continue adjustment. If the adjustment difference does not exceed the preset target value range, the joint adjustment is determined to be complete. It is understood that those skilled in the art can set the target value to meet actual needs.

[0103] Furthermore, after step S53, step S5 further includes:

[0104] Step S56: After detecting that all the adjustment arms 21a have been positioned, a prompt indicating that the current adjustment is completed is displayed and / or recorded.

[0105] Optional, please refer to Figure 16 and Figure 17 In one exemplary embodiment, the binocular vision device 71 is installed on the AR device 30, which may be wearable AR glasses. Thus, as the operator's perspective changes, the binocular vision device 71 can perform real-time position matching on the feature points of the robotic arm 21, thereby achieving real-time calibration by converting the coordinate position of the AR device 30 relative to the robotic arm 21. Figure 16 , the position change of the AR device 30 is shown, and its coordinate system moves from (X3 / Y3 / Z3) to (X31 / Y31 / Z31). At this time, the coordinate system (X31 / Y31 / Z31) and the coordinate system (X3 / Y3 / Z3) can be converted according to the translation vector t.

[0106] Preferably, in step S51, based on the real-time position of the AR device 30 and the relative positions between the plurality of adjustment arms 21a, the nearest adjustment arm 21a is recommended as the adjustment arm for the initial adjustment. Figure 17 As shown, during the actual adjustment of the adjustment arm 21a, the adjustment arm 21a closest to the AR device 30 can be recommended as the starting adjustment arm based on the relative position between the real-time position of the AR device 30 and each adjustment arm 21a. The recommended starting adjustment arm here refers to the adjustment arm 21a that is first recommended for the operator to adjust, which can be prompted on the AR device 30.

[0107] Furthermore, before prompting the operator in step S52 for the joints of the selected adjustment arm 21a that require adjustment, the control device recommends the joint with the highest collision probability as the starting joint for adjustment based on the positions of the joints of the selected adjustment arm 21a and adjacent joints. After the operator selects the adjustment arm 21a to be adjusted, the control device calculates the collision probability of each joint of the adjustment arm 21a based on the positional relationship between the joints of the selected adjustment arm 21a and the joints of other adjacent adjustment arms 21a. This determines the order in which the joints should be adjusted, prioritizing joints with the highest collision probability. This process is repeated to determine the optimal positioning path for all adjustment arms 21a throughout the patient-side surgical operating table 20.

[0108] This embodiment also provides a readable storage medium on which a program is stored. When the program is executed, the robotic arm collision prompt method as described above is implemented. The readable storage medium can be integrated into the surgical robot, such as integrated into the control device, or can be attached independently.

[0109] In summary, in the robotic arm positioning method, readable storage medium, and surgical robot system provided by the present invention, the robotic arm positioning method includes: obtaining a virtual three-dimensional model of the surgical object, the pre-drilled hole positions on the surgical object, and the robotic arm of the patient-side surgical operating table; obtaining the positioning configuration of the adjustment arm of the robotic arm based on the virtual three-dimensional model; performing virtual-real fusion coordinate registration on the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object based on the current positioning configuration of the adjustment arm; obtaining the positioning path planning of the adjustment arm based on the real-time position of the AR device and the positioning configuration of the adjustment arm; and prompting the positioning path planning through the AR device. With such a configuration, based on the virtual-real fusion coordinate registration, the obtained positioning path planning of the adjustment arm is prompted by the AR device based on the real-time position of the AR device, which can conveniently assist and guide the operator in performing the positioning operation of the adjustment arm, thereby improving the accuracy and execution efficiency of preoperative positioning.

[0110] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to in conjunction with each other. In addition, the different parts between the various embodiments can also be used in combination with each other, and the present invention is not limited to this. The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention. Any changes or modifications made by a person skilled in the art based on the above disclosure are within the scope of protection of the present invention.

Claims

1. A method for preoperative positioning of a robotic arm, characterized in that: include: Acquire a virtual three-dimensional model of the surgical object, pre-drilled hole locations on the surgical object, and a robotic arm of a patient-side surgical operating table; The virtual three-dimensional model of the surgical object, the pre-drilling position, and the robotic arm is established based on the parallax information fed back by a binocular vision device; the binocular vision device is installed on the AR device; Obtaining a positioning configuration of an adjustment arm of the robotic arm according to the virtual three-dimensional model; According to the current positioning configuration of the adjustment arm, the coordinate system of the patient's operating table is aligned with the real body surface coordinate system of the surgical object by virtual-real fusion coordinate registration; Obtaining a positioning path plan for the adjusting arm according to the real-time position of the AR device and the positioning configuration of the adjusting arm; as well as Prompt the positioning path planning through the AR device; The method for prompting the positioning path planning by using an AR device includes: The AR device displays a plurality of planned positions of the adjustment arm and displays a plurality of positioning paths of the adjustment arm for selection to reach the planned positions; Prompt the joint of the selected adjustment arm that needs to be adjusted; and Prompts the amount of adjustment required for the joint.

2. The robotic arm preoperative positioning method according to claim 1, characterized in that: The robotic arm positioning method further includes: Obtaining the actual position of the adjustment arm after positioning according to the positioning path planning; Obtaining a planned position of the adjustment arm calculated according to the positioning path planning; Calculate the deviation between the planned position and the actual position. If the deviation exceeds the preset standard, execute again: based on the current positioning configuration of the adjustment arm, perform virtual-real fusion coordinate alignment on the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object; based on the real-time position of the AR device, obtain the positioning path planning of the adjustment arm; and prompt the positioning path planning step through the AR device; if the deviation is within the preset standard, determine that the positioning of the adjustment arm is completed.

3. The robotic arm preoperative positioning method according to claim 1, characterized in that: According to the current positioning configuration of the adjustment arm, a method for performing virtual-real fusion coordinate registration between the patient-side operating table coordinate system and the real body surface coordinate system of the surgical object includes: Establishing coordinate mapping relationships between the binocular vision device coordinate system and the patient-side operating table coordinate system and the AR device coordinate system in the world coordinate system; According to the mapping relationship between the real body surface coordinate system of the surgical object and the coordinate system of the binocular vision device, a coordinate mapping relationship between the patient-side surgical operating table coordinate system and the real body surface coordinate system of the surgical object is established.

4. The method for preoperative positioning of a robotic arm according to claim 3, characterized in that: The relative coordinate relationship between the binocular vision device coordinate system and the AR device coordinate system is fixed.

5. The robotic arm preoperative positioning method according to claim 1, characterized in that: The method of prompting the positioning path planning by using an AR device further includes: Obtain the adjustment difference value fed back during the adjustment process of the joint that needs to be adjusted; If the adjustment difference exceeds the range of the preset target value, the step of prompting the adjustment amount of the joint to be adjusted is performed again until the adjustment difference is within the range of the preset target value.

6. The method for preoperative positioning of a robotic arm according to claim 1, characterized in that: The method of prompting the positioning path planning by using an AR device further includes: After detecting that the positioning of all the adjustment arms is completed, a prompt is given that the current adjustment is completed, and the operation time is displayed and / or recorded.

7. The method for preoperative positioning of a robotic arm according to claim 1, characterized in that: In the step where the AR device displays the planned positions of multiple adjustment arms and displays the positioning paths of multiple adjustment arms for selection, the nearest adjustment arm is recommended as the starting adjustment arm based on the real-time position of the AR device and the relative positions between the multiple adjustment arms.

8. The robotic arm preoperative positioning method according to claim 1, characterized in that: Before prompting the joints of the selected adjusting arm that need to be adjusted, the joint with the highest collision probability is recommended as the starting joint for adjustment based on the positions of the joints of the selected adjusting arm and the adjacent joints.

9. The method for preoperative positioning of a robotic arm according to claim 1, characterized in that: The current positioning configuration of the adjustment arm is obtained based on historical surgical data under the same surgical procedure and the current virtual three-dimensional model.

10. A readable storage medium having a program stored thereon, characterized in that: When the program is executed, the robotic arm preoperative positioning method according to any one of claims 1 to 9 is implemented.

11. A surgical robot system, characterized in that: The present invention comprises a patient-side surgical operating table, an AR device and a control device. The patient-side surgical operating table comprises a robotic arm, and the robotic arm comprises an adjustment arm. The control device uses the robotic arm preoperative positioning method according to any one of claims 1 to 9 to control the AR device to prompt the positioning path planning of the adjustment arm.

12. The surgical robot system according to claim 11, wherein: The surgical robot system also includes a binocular vision device, which is used to obtain parallax information of the surgical object, the pre-drilling position and the robotic arm, and the control device is used to establish a virtual three-dimensional model of the surgical object, the drilling position and the robotic arm based on the parallax information.

13. The surgical robot system according to claim 12, wherein: The binocular vision device is installed on an AR device, and the AR device is wearable AR glasses; the binocular vision device is integrated with the AR device.

14. The surgical robot system according to claim 11, wherein: The robotic arm also includes a tool arm, one end of which is used to connect to the surgical instrument, and the other end is used to connect to the adjustment arm. The adjustment arm includes multiple joints. The control device controls the AR device to prompt the joint that needs to be adjusted in the selected adjustment arm according to the positioning path planning.

15. The surgical robot system according to claim 14, wherein: The multiple joints of the adjustment arm include at least three rotational joints and one movable joint. The control device controls the AR device to prompt at least one of the rotational joints and movable joints of the selected adjustment arm that needs to be adjusted according to the positioning path planning.

Images