Surgical robot system and method

Abstract

The present application relates to a surgical robot system and method, comprising: a surgical device equipped with a robotic arm; a navigation system for determining a planned motion path for the robotic arm in a real surgical space based on the robotic arm's starting position and target surgical position, the planned motion path including the motion path and information on the posture change of the robotic arm along the motion path; and an augmented reality device, the augmented reality device being in communication with the navigation system and configured to obtain the planned motion path and display the posture change process of a virtual robotic arm model on the augmented reality device based on the planned motion path, the posture change process including the virtual motion path and virtual posture of the virtual robotic arm model to demonstrate the relative positional relationship with objects in the real surgical space. The user can intuitively understand whether a collision will occur or whether the robotic arm will reach a singularity position when controlling the robotic arm's movement using the current planned motion path, and can avoid such situations before controlling the robotic arm's movement.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a surgical robot system and method. Background Art

[0002] During surgery, surgical robots are responsible for automating the process of accurately aligning the tool at the end of their robotic arm with the patient's surgical site. However, current robotic arm path planning solutions often require multiple adjustments during surgery due to the tool hitting obstacles or the arm pulling the tool reaching singular points, resulting in inefficient surgery. Summary of the Invention

[0003] Based on this, it is necessary to address the above technical problems and provide a surgical robot system and method that does not require multiple positioning operations during surgery, thereby improving the efficiency of surgical operations.

[0004] In a first aspect, a surgical robot system is provided, comprising:

[0005] A surgical device equipped with a robotic arm;

[0006] a navigation system for determining a planned motion path of the robotic arm in a real surgical space based on the starting position and target surgical position of the robotic arm, the planned motion path including the motion path and information on position changes of the robotic arm along the motion path; and

[0007] An augmented reality device is communicatively connected to the navigation system and is used to obtain a planned motion path and display a posture change process of a virtual robotic arm model on the augmented reality device according to the planned motion path. The posture change process of the virtual robotic arm model includes a virtual motion path and a virtual posture of the virtual robotic arm model.

[0008] In one embodiment, the augmented reality device is further configured to respond to an adjustment operation performed on a posture change process of the virtual robotic arm model to form a revised planned motion path;

[0009] The posture change process of the virtual robotic arm model is displayed on the augmented reality device according to the corrected planned motion path.

[0010] In one embodiment, the augmented reality device is used to adjust the posture change process of the virtual robotic arm model in response to the operation to form a corrected planned motion path, including:

[0011] Display the posture adjustment interface;

[0012] In response to the adjustment action performed on the posture change process of the virtual robotic arm model in the posture adjustment interface, a corrected planned motion path is formed.

[0013] In one embodiment, the adjustment action performed on the posture change process of the virtual robotic arm model includes at least one of a starting posture adjustment action, an on-path robotic arm posture adjustment action, and a target surgical posture adjustment action.

[0014] In one embodiment, the augmented reality device is configured to send a virtual motion path and a virtual posture to a target object, which is a navigation system or a surgical device, in response to a path confirmation action performed after displaying a posture change process of a virtual robotic arm model;

[0015] Target objects are also used to:

[0016] Calculate the final planned motion path of the robotic arm in the real surgical space based on the current starting position of the robotic arm in the real surgical space, the virtual motion path, and the conversion relationship between the virtual position and the display space coordinate system of the augmented reality device and the real surgical space;

[0017] In the real surgical space, the robotic arm moves to the target surgical position according to the final planned motion path.

[0018] In one embodiment, the real surgical space includes a surgical coordinate system, and the augmented reality device is used to:

[0019] Defining a display space coordinate system based on a display interface and sending the display space coordinate system to the navigation system;

[0020] Acquiring coordinate information of an identification pattern on the surgical device to be identified in the display interface and sending the information to the navigation system;

[0021] The navigation system is also used to:

[0022] Defining the surgical coordinate system based on the optical target on the entity to be operated on, and acquiring coordinate information of the optical target on the entity to be operated on and coordinate information of the optical target of the base target on the surgical device in the surgical coordinate system;

[0023] Based on the coordinate information of the optical target of the base target on the surgical device in the surgical coordinate system and the coordinate information of the identification pattern to be identified in the display interface, feature matching of corresponding point coordinates is performed to determine the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system.

[0024] In one embodiment,

[0025] The end of the robotic arm is provided with a medical instrument, the real surgical space further includes a robotic arm motion coordinate system, the surgical device includes a medical instrument, the optical target on the surgical device further includes a tool target, and the tool target is provided at the end of the robotic arm, and the navigation system is further used for:

[0026] Establishing a motion coordinate system of the robotic arm with the center point of at least four points on one end surface of the medical device as the origin;

[0027] Obtaining coordinates of at least four points on the end surface in the robotic arm motion coordinate system;

[0028] Acquiring the coordinates of the tool target in the surgical coordinate system;

[0029] The coordinates of at least four points on the end surface are matched with the coordinates of the tool target to determine the coordinate conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system.

[0030] In one embodiment, the navigation system is further configured to:

[0031] Based on the starting position and target surgical position of the robotic arm, the planned motion path in the robotic arm motion coordinate system is determined;

[0032] Augmented reality devices are used to:

[0033] According to the coordinate conversion relationship between the virtual surgical space and the surgical coordinate system, the coordinate conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system, the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system, and the planned motion path under the robotic arm motion coordinate system, the posture change process of the virtual robotic arm model is displayed on the augmented reality device. The posture change process of the virtual robotic arm model includes the virtual motion path and virtual posture of the virtual robotic arm model.

[0034] In one embodiment, the navigation system is further configured to:

[0035] Determine the coordinate transformation relationship between the robotic arm motion coordinate system and the display space coordinate system based on the coordinate transformation relationship between the virtual surgical space and the surgical coordinate system, the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system, and the coordinate transformation relationship between the surgical coordinate system and the display space coordinate system;

[0036] Based on the current starting position of the manipulator in the manipulator motion coordinate system, the virtual motion path and virtual position in the display space coordinate system, and the coordinate transformation relationship between the manipulator motion coordinate system and the display space coordinate system, the final planned motion path of the manipulator in the manipulator motion coordinate system is obtained;

[0037] Control the robotic arm to move to the target surgical position according to the final planned motion path in the robotic arm motion coordinate system.

[0038] In a second aspect, the present application further provides a control method, which is applied to the above-mentioned surgical robot system, and the method comprises:

[0039] Obtaining a planned motion path in a real surgical space determined by a navigation system; wherein the planned motion path includes the motion path and information about position changes of a robotic arm along the motion path, wherein the robotic arm is mounted on the surgical device;

[0040] The posture change process of the virtual robotic arm model is displayed on the augmented reality device according to the planned motion path. The posture change process of the virtual robotic arm model includes the virtual motion path and virtual posture of the virtual robotic arm model.

[0041] In one embodiment, the method further includes:

[0042] In response to the adjustment operation performed on the posture change process of the virtual robot arm model, a revised planned motion path is formed;

[0043] The posture change process of the virtual robotic arm model is displayed on the augmented reality device according to the corrected planned motion path.

[0044] In one embodiment, in response to an adjustment operation performed on a posture change process of a virtual robotic arm model, forming a revised planned motion path includes:

[0045] Display the posture adjustment interface;

[0046] In response to the adjustment action performed on the posture change process of the virtual robotic arm model in the posture adjustment interface, a corrected planned motion path is formed.

[0047] In one embodiment, the adjustment action performed on the posture change process of the virtual robotic arm model includes at least one of a starting posture adjustment action, an on-path robotic arm posture adjustment action, and a target surgical posture adjustment action.

[0048] In one embodiment, in response to an adjustment operation performed on a posture change process of a virtual robotic arm model, forming a revised planned motion path includes:

[0049] Marking the virtual robotic arm model during the posture change process of the virtual robotic arm model;

[0050] In response to the pose adjustment operation on the marked area of the virtual robotic arm model, a corrected planned motion path is formed.

[0051] In one embodiment, the method further includes:

[0052] In response to a path confirmation action performed after displaying the posture change process of the virtual robotic arm model, sending the virtual motion path and the virtual posture to the target object, so that the target object determines a final planned motion path based on the virtual motion path and the virtual posture;

[0053] The final planned motion path is used to instruct the robotic arm to move to the target surgical position according to the final planned motion path, and the target object is the navigation system or surgical device.

[0054] The above-mentioned surgical robot system and method have at least the following beneficial effects:

[0055] The surgical robot system includes a surgical device, a navigation system, and an augmented reality device. The surgical device is equipped with a robotic arm, and the navigation system can determine the planned motion path of the robotic arm in a real surgical space based on the robotic arm's starting position and target surgical position. The augmented reality device is connected to the navigation system and can display the posture change process of a virtual robotic arm model on itself based on the robotic arm's motion path and posture change information planned by the navigation system, thereby demonstrating the virtual motion path and virtual posture of the virtual robotic arm model. At the same time, because the augmented reality device can simultaneously display objects in the real surgical space, the user can use the augmented reality device to visualize the relative positional relationship between the virtual robotic arm model and objects in the real surgical space along the virtual motion path before surgery. This allows the user to intuitively understand whether the planned motion path calculated by the current navigation system will cause collisions with surrounding objects in the real surgical space, or whether the robotic arm will be unable to continue moving due to reaching a singularity point. Based on the specific situation displayed, the user can further adjust the robotic arm's starting position, planned motion path, etc., so as to control the robotic arm to the user's desired target osteotomy position in one go.

[0056] This system, by utilizing the virtual-reality combined display capability of the augmented reality device, previews the motion trajectory of the virtual robotic arm model in the display space of the augmented reality device, helping users to intuitively judge whether a robotic arm collision will occur or the robotic arm movement will reach a singularity point before surgery, and make preoperative adjustments to prevent the robotic arm from colliding during the automatic positioning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 is a schematic structural diagram of a surgical robot system in one embodiment;

[0059] Figure 2 is a schematic diagram of a posture adjustment interface in one embodiment;

[0060] Figure 3is a schematic diagram of an operation for adjusting the posture of a virtual robotic arm model displayed on a virtual motion path in one embodiment;

[0061] Figure 4 A schematic diagram showing the virtual motion trajectory before and after adjustment in a spatial coordinate system when a virtual robotic arm model collides with an obstacle in one embodiment;

[0062] Figure 5 A schematic diagram showing the virtual motion trajectory before and after adjustment in a spatial coordinate system and a schematic diagram showing an alarm message when a virtual robotic arm model reaches a singular point on a virtual motion trajectory in an embodiment;

[0063] Figure 6 FIG1 is a schematic diagram of adjusting the position of a robotic arm in a motion coordinate system based on an inflection point determined in the display space coordinate system when a virtual motion trajectory in the display space coordinate system collides with an obstacle in one embodiment;

[0064] Figure 7 is a schematic diagram of the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system in one embodiment;

[0065] Figure 8 Schematic diagram of the coordinate conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system in one embodiment;

[0066] Figure 9 Schematic diagram of coordinate transformation relationships of various spatial coordinate systems in one embodiment;

[0067] Figure 10 Schematic diagram of a multi-axis structure of a robotic arm in one embodiment;

[0068] Figure 11 A schematic diagram showing, in one embodiment, a virtual motion trajectory of a robotic arm based on a virtual surgical model in a virtual posture to reach a target osteotomy surface of the virtual surgical model in a display space coordinate system;

[0069] Figure 12 1 is a flow chart of a control method in one embodiment;

[0070] Figure 13 1 is a flow chart of a control method in one embodiment;

[0071] Figure 14 1 is a flow chart of a control method in another embodiment;

[0072] Figure 15 A schematic flow chart of a control method in yet another embodiment;

[0073] Figure 16is a structural block diagram of a control device in one embodiment;

[0074] Figure 17 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0075] To facilitate understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The accompanying drawings provide embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of the present application more thorough and comprehensive.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0077] It should be noted that when an element is considered to be “connected to” another element, it can be directly connected to the other element or connected to the other element through an intervening element.

[0078] As used herein, the singular forms "a," "an," and "the" may also include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include," "comprising," "having," and the like specify the presence of stated features, integers, steps, operations, components, parts, or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof. Furthermore, the term "and / or" as used in this specification includes any and all combinations of the relevant listed items.

[0079] During surgery, the orthopedic surgical robot is responsible for automating the process of accurately aligning the osteotomy guide with the osteotomy surface of the patient's bones. During surgery, the target osteotomy surface is determined on the skeletal entity based on the mapping relationship between the virtual surgical model obtained through intraoperative registration and the patient's skeletal entity on the operating table. The robot uses the starting position and starting posture of the robotic arm, as well as the position of the target osteotomy surface, provided by an optical positioning system (e.g., an optical camera), to construct a movement path and posture change method for the robotic arm from its starting position and starting posture to the target posture and target position, and then drives the robotic arm to move autonomously along the movement path and posture change method.

[0080] However, when the robotic arm automatically positions itself along a predetermined moving path, the robotic arm joint, the arm body, or the osteotomy guide may collide with obstacles in the actual surgical scenario, or may reach the singularity point of the robotic arm. Once the robotic arm collides with an obstacle or is at the singularity point of the robotic arm, the doctor needs to stop the robotic arm, manually operate the robotic arm to another position, and then repeat the automatic positioning process of the robotic arm osteotomy surface, resulting in low surgical efficiency. In addition, a secondary collision or the occurrence of the robotic arm's singularity point may occur.

[0081] However, solutions that rely on optical positioning systems (such as optical cameras) for automatic positioning and movement of the robotic arm do not provide real-time obstacle avoidance detection during the movement of the robotic arm, making it impossible for the robotic arm to select a suitable path and posture change method.

[0082] In order to solve the above problems, in one embodiment, Figure 1 As shown, a surgical robot system is provided, which includes: a surgical device equipped with a robotic arm 12; a navigation system 61, which is used to determine the planned motion path of the robotic arm 12 in the real surgical space based on the starting posture and target surgical posture of the robotic arm 12, wherein the planned motion path includes the motion path and posture change information of the robotic arm 12 moving along the motion path; and an augmented reality device 91, which is communicatively connected to the navigation system 61, and is used to obtain the planned motion path and display the posture change process of the virtual robotic arm model on the augmented reality device 91 according to the planned motion path, wherein the posture change process of the virtual robotic arm model includes the virtual motion path and virtual posture of the virtual robotic arm model.

[0083] Among them, the end of the robot arm 12 can be mechanically connected to some medical devices. For example, in an orthopedic surgery scenario, the medical device can be Figure 1 The osteotomy guide 31 shown, etc. The navigation system 61 can achieve positioning and alignment. For example, it can align the virtual robotic arm model with the physical robotic arm 12 on the navigation system 61, and it can also align the virtual bone model with the physical bone to be operated on (for example, the femur 71 and tibia 72 shown in the figure), that is, it can determine the conversion relationship between the virtual surgical space and the real surgical space. The navigation system can also determine the planned motion path of the robotic arm 12 based on the starting position and target surgical position of the robotic arm 12. In addition to the motion path, the planned motion path also includes information on the posture change of the robotic arm when moving on the motion path.

[0084] The surgical device is a device that can drive the robot arm 12 to move autonomously. For example, it can be as follows Figure 1As shown, a robotic arm 12 is mechanically connected to the upper surface of the surgical device 11. The robotic arm 12 may have multiple joints and may perform multi-axis motion so as to drive medical devices such as the osteotomy guide 31 to accurately reach the target surgical position. The surgical device 11 can also be provided with multiple targets (for example, the tool target 21, femoral target 22, tibial target 23 and base target 24 as shown in the figure), and the navigation system (which can be equipped with an optical positioning system (for example, an optical tracking system such as an optical camera 51)) obtains the coordinates of the base, robotic arm 12 and osteotomy guide 31 on the surgical device 11 to determine the conversion relationship between multiple coordinate systems. The registration process can be a process of establishing a mapping relationship between the virtual robotic arm model and the robotic arm entity 12, for example, a mapping relationship between the coordinates of the two is established. Among them, the virtual robotic arm model can be established in advance or obtained by laser scanning. The starting posture and target surgical posture of the robotic arm 12 can be determined based on the surgical plan. Taking into account that the medical devices used in different surgical scenarios are different, the robotic arm 12 and the medical device can be detachably connected, and the user can select and install the medical device according to their needs.

[0085] Among them, the augmented reality device 91 can be a device such as AR (Augmented Reality) glasses or AR helmets that can present a virtual and real picture to the user. A wireless connection can be adopted between the augmented reality device 91 and the navigation system 61 and the surgical device 11 to improve the convenience of the surgical device during movement and avoid stumbling due to wiring during movement. Posture change information refers to the changes in the position and posture of each joint of the robotic arm during the process of the robotic arm moving from the starting position to the target osteotomy position. Objects in the real surgical space include but are not limited to the above-mentioned surgical device, the operating table 81, the robotic arm 12, the patient 73 corresponding to the skeletal entity to be operated on, and the doctor 92.

[0086] The real surgical space refers to the space where the robotic arm that performs the actual surgical operation and the patient being treated are located. The real surgical space may include a surgical coordinate system and a robotic arm motion coordinate system. In the case where a medical device is installed at the end of the robotic arm, the movement of the medical device can be based on an independent robotic arm motion coordinate system. The surgical coordinate system of the skeletal entity can be two independent coordinate systems, but can also be based on the position calibration of the same spatial coordinate system, which depends on the definition of the coordinate system in the real surgical space. Those skilled in the art should understand that the real surgical space referred to here refers to a scene with real objects, in which the objects can be in the same coordinate system or in different coordinate systems, but different coordinate systems can be converted into coordinates, and ultimately all coordinates can be converted into object coordinates in the coordinate system where the robotic arm is located, so that the surgical device can drive the movement of the robotic arm 12 based on this coordinate.

[0087] Specifically, the navigation system 61 determines a planned motion path based on the starting position and target surgical position of the robotic arm 12 on the surgical device 11. Then, based on a communication connection with the augmented reality device 91, the augmented reality device 91 obtains the planned motion path obtained by the navigation system 61. Before surgery, the augmented reality device 91 determines and displays the virtual position changes of the virtual robotic arm model corresponding to the robotic arm 12 along the virtual motion path in its display space based on the motion path and position change information, thereby demonstrating the relative positional relationship between the virtual robotic arm model and objects in the real surgical space along the virtual motion path. The user 92 can intuitively understand whether the robotic arm 12 will collide with objects in the real surgical space if the originally obtained planned motion path is used to control the motion of the robotic arm 12 in the real surgical space, or whether the robotic arm 12 will be unable to continue moving due to reaching a singularity point. Based on the specific circumstances displayed, the user 92 can further adjust the starting position of the robotic arm 12, the planned motion path, and other factors, so as to be able to control the robotic arm 12 to the target osteotomy position desired by the user 92 in one go.

[0088] By utilizing the virtual-reality combined display capability of the augmented reality device 91, the motion trajectory of the virtual robotic arm model in the display space coordinate system is previewed, helping the user 92 to intuitively judge whether a collision event of the robotic arm 12 will occur or the robotic arm 12 will reach a singularity point before the operation, and make preoperative adjustments to prevent the robotic arm 12 from colliding during the automatic positioning process.

[0089] In one embodiment, the virtual surgical model can be obtained based on preoperative scans of the skeletal entities to be operated on (e.g., 71 and 72). The virtual robotic arm model can also be obtained by the navigation system based on preoperative photography or scanning of the physical robotic arm 12.

[0090] In one embodiment, the augmented reality device 91 is further configured to respond to adjustments made to the posture change process of the virtual robotic arm model, forming a revised planned motion path; and displaying the posture change process of the virtual robotic arm model on the augmented reality device 91 based on the revised planned motion path. Adjustments to the posture change process of the virtual robotic arm model can be made by the user adjusting the starting posture and target surgical posture of the robotic arm in the real surgical space, or by the user adjusting the posture of the displayed virtual robotic arm model within the visual range of the augmented reality device 91 using the interface of the augmented reality device 91. Regardless of whether it is based on the adjustment of the robot arm posture in the real surgical space or the adjustment of the posture of the virtual robot arm model on the virtual motion path, the augmented reality device 91 can re-plan a virtual motion path based on the adjustment operation and determine the posture change of the virtual robot arm model on the virtual motion path, which is called the revised planned motion path. The planned motion path refers to the planned path in the display space coordinate system of the augmented reality device 91. According to the planned motion path, the augmented reality device 91 displays the new posture change of the virtual robot arm model on the new virtual motion path to show whether the robot arm 12 will collide with objects in the real surgical space or the robot arm 12 itself will move to a singularity point according to the revised planned motion path after the adjustment operation.

[0091] In one embodiment, Figure 2 As shown, the augmented reality device 91 is used to respond to the posture change process of the virtual robotic arm model and adjust the operation to form a corrected planned motion path. The process includes:

[0092] Display posture adjustment interface 94;

[0093] In response to the adjustment action performed on the posture change process of the virtual robotic arm model in the posture adjustment interface 94, a corrected planned motion path is formed. The virtual robotic arm model can be pre-established based on the positioning scanning function of the optical tracking system carried by the navigation system. It can also be constructed based on pre-scanning of other scanning devices, and those skilled in the art should understand that it can be known in advance. The above-mentioned virtual robotic arm model and virtual surgical model both belong to the virtual surgical space, and their coordinates can be determined based on the virtual surgical coordinate system. However, it should be understood that if the emphasis is on the virtual robotic arm model displayed on the augmented reality device 91, it refers to the model display achieved by converting the virtual robotic arm model from the virtual surgical space to the display space coordinate system of the augmented reality device 91.

[0094] During operation, the doctor 92 can use his finger 93 or a stylus to trigger the up, down, left, and right translation arrows and the rotation arrows on the left side of the posture adjustment interface 94. Through gesture recognition and other technologies, the doctor can respond to the adjustment operation and change the posture of the robotic arm accordingly. The corresponding command button can also be triggered in the column on the right side of the schematic diagram to change the adjustment distance of the starting position of the robotic arm 12 and the adjustment amplitude of the angle, etc. Of course, the examples here give the specific situations of two interactive implementation interfaces, and other display icons can also be used. Interfaces that can respond to the posture adjustment action of the robotic arm are all within the scope of protection of this application.

[0095] In one embodiment, the adjustment action performed on the posture change process of the virtual robotic arm model includes at least one of a starting posture adjustment action, an on-path robotic arm posture adjustment action, and a target surgical posture adjustment action.

[0096] Specifically, the posture adjustment interface may display at least one posture adjustment area of the robot arm's starting posture, the robot arm's posture on the path, and the target surgical posture; the robot arm's posture adjustment area on the path (used to implement the robot arm's posture adjustment action on the path) may display control icons for adjusting the robot arm's posture at any point or multiple points on the motion path. For those situations where the robot arm reaches a singular point during the movement, the singular point problem can be avoided by adjusting the robot arm's posture at that point on the motion path. The target surgical posture adjustment area (used to implement the target surgical posture adjustment action) may display icons for the target osteotomy position and the osteotomy guide plate osteotomy posture. The robot arm's starting posture adjustment area (used to implement the starting posture adjustment action) may display icons for adjusting the robot arm's starting position and posture, so as to obtain a revised planned motion path and re-rehearse the robot arm's motion process based on the revised planned motion path until the surgical requirements are met, thereby avoiding collisions during the robot arm's autonomous movement or the robot arm joint reaching a singular point.

[0097] In one embodiment, Figure 3 As shown, the augmented reality device is also used to mark the virtual robotic arm model during the posture change process of the virtual robotic arm model; and respond to the posture adjustment operation of the marked area of the virtual robotic arm model to form a corrected planned motion path; and then display the posture change process of the virtual robotic arm model on the augmented reality device according to the corrected planned motion path.

[0098] by Figure 3The example shown is used to illustrate that the augmented reality device can acquire and recognize the doctor's gesture grabbing action, and adjust the posture of the robotic arm based on the recognition result. For example, as shown in the figure, the doctor can grab the robotic arm in the marked area (virtual rectangular frame part 95) with his hand, and after grabbing, adjust the up, down, left, and right translation and rotation of the robotic arm in conjunction with the hand movement. Of course, the virtual rectangular frame used as an example here is just one of the ways. You can also directly grab the virtual robotic arm model for posture adjustment. The augmented reality device can adjust the posture of the robotic arm based on the user's posture adjustment operation and the recognition of the interactive action, and optimize the virtual motion path and virtual posture based on the posture of the adjusted robotic arm.

[0099] Of course, the process of optimizing the virtual motion path and virtual posture can also be to first determine the starting posture and target surgical posture that the robotic arm needs to adjust in the real surgical space based on the posture adjustment operation, and then re-enter the step of determining the planned motion path and posture change information based on the starting posture and target surgical posture of the robotic arm. The augmented reality device 91 generates a new virtual motion path and virtual posture based on the re-determined motion path and posture change information and performs a preview.

[0100] The preview refers to the augmented reality device 91 displaying the changes in the position of the virtual robotic arm model along the virtual motion path. This can be done by displaying the virtual positions of the virtual robotic arm model and the virtual osteotomy guide at intervals along the motion path, as shown in the accompanying figure, or by presenting the changes in the virtual position in the form of an animation. If the preview shows that the virtual robotic arm model will not collide or reach a singularity point, a coordinate system transformation is performed based on the corrected planned motion path to obtain the planned motion path required for the robotic arm 12 to reach the target surgical position in the real surgical space.

[0101] like Figure 4 As shown, the doctor can use the virtual motion path 103 based on the starting position 101 and the target surgical position 106 of the robotic arm 12 and the virtual position of the virtual robotic arm model 34 displayed at intervals on the virtual motion path 103 displayed by the augmented reality device 91 to visually determine that the movement of the robotic arm based on the current starting position and the target surgical position will cause a collision with the obstacle 105.

[0102] At this point, the robotic arm posture adjustment solution provided in the above embodiment can be used to move the starting position of the robotic arm 12 to an appropriate position 102. By determining the virtual motion path 104 based on the new starting position and the virtual posture of the virtual robotic arm model displayed in intervals within the path 104, the robotic arm will not collide with obstacles when moving based on the adjusted starting position. By performing a pre-operative rehearsal, the safety of the motion path is ensured before the actual robotic arm 12 automatically positions and moves.

[0103] In addition, if Figure 5As shown, the virtual motion path planned based on the starting position and target surgical position of the robot arm 12 shows the situation where the robot arm 12 reaches the singular point position during the movement, as described below:

[0104] like Figure 5 As shown, the doctor can use the augmented reality device 91 to display the posture changes of the virtual robotic arm model as it moves on the virtual motion path 103, thereby seeing the singular point position posture 36 of the virtual robotic arm model on the path 103, and an alarm prompt box 107 pops up in the display interface of the augmented reality device 91.

[0105] At this point, the robot arm's initial position can be adjusted to an appropriate position 102 by responding to the position adjustment action described in the above embodiment. By examining the virtual motion path 104 based on the new robot arm's initial position 102 and the virtual robot arm model 34 displayed in between along the path 104, it can be determined that the robot arm 12 will not generate another alarm when moving based on the adjusted initial and final positions. Similarly, the safety of the motion path is ensured before the actual robot arm 12 automatically positions and moves.

[0106] In one embodiment, the augmented reality device 91 is further configured to:

[0107] Selecting and recording the coordinate positions of at least two inflection points of the virtual motion path after the path adjustment in the display space coordinate system; wherein the inflection points can be selected using a gesture recognition function;

[0108] The coordinate positions of at least two inflection points are converted to the robot arm motion coordinate system through the coordinate conversion relationship of each spatial coordinate system (the determination of the coordinate conversion relationship can be based on the description in other embodiments);

[0109] Based on the coordinates of the above-mentioned inflection points in the robot arm motion coordinate system, the planned motion path of the robot arm 12 in the robot arm motion coordinate system is planned.

[0110] by Figure 6 For example, the doctor can use the virtual robotic arm model 34 displayed in the interval of the virtual motion path 103 based on the robotic arm's starting position 101 and the target surgical position 106 displayed by the augmented reality device 91 to judge with the naked eye that the movement of the robotic arm based on the current initial and final positions will cause a collision with the obstacle 105.

[0111] At this point, the gesture recognition module of the augmented reality device can select and record the coordinate positions of the inflection points E and F after the path adjustment in the display space coordinate system, and convert them to the robot arm motion coordinate system through the coordinate transformation relationship of each space coordinate system, thereby obtaining the coordinates of the inflection points E and F in the robot arm motion coordinate system. Based on these coordinates, the virtual motion path 107 is adjusted and calculated and displayed. By judging the new virtual motion path 107 and the virtual robot arm model displayed in the interval of path 107, it can be determined that the robot will not collide with obstacles based on the adjusted motion path 107. The safety of the motion path is ensured before the real robot arm 12 automatically positions and moves.

[0112] In one embodiment, the augmented reality device 91 is used to send the virtual motion path and virtual posture to the target object in response to the path confirmation action performed after the posture change process of the virtual robotic arm model is displayed, and the target object is a navigation system or a surgical device; the target object is also used to: calculate the final planned motion path of the robotic arm in the real surgical space based on the current starting posture of the robotic arm 12 in the real surgical space, the virtual motion path, the virtual posture and the conversion relationship between the display space coordinate system of the augmented reality device and the real surgical space; and be used for the robotic arm 12 to move to the target surgical posture according to the final planned motion path in the real surgical space.

[0113] Among them, the path confirmation action can be triggered by the user based on the virtual posture of the virtual robotic arm model and the virtual osteotomy guide on the virtual motion path displayed by the augmented reality device 91, and judging that the robotic arm 12 follows the path planning without collision and the robotic arm 12 joint reaching the singular point position. The path confirmation action can be triggered by the user based on the confirmation command icon on the display interface of the augmented reality device 91, or based on the operation on the display screen 62 on the navigation system 61, or by inputting information on the keyboard 63 on the navigation system 61, or by other triggering methods. The real surgical space and the robotic arm motion coordinate system can be a coordinate system. For example, a surgical coordinate system can be constructed with a point on a stationary object in the operating room as the coordinate origin, and the coordinates of the robotic arm can also be determined based on this coordinate system.

[0114] When the user determines that the virtual posture of the virtual robotic arm model on the virtual motion path displayed by the augmented reality device 91 will not collide with objects in the real surgical space and that the robotic arm itself will not reach a singularity, the user can perform a path confirmation action. At this time, the augmented reality device 91 can send the current virtual motion path and virtual posture (i.e., the above-mentioned revised planned motion path) to the navigation system or surgical device based on the path confirmation action, so that the navigation system or surgical device can convert the revised planned motion path in the displayed space coordinate system into the final planned motion path of the robotic arm in the real surgical space based on the conversion relationship between multiple coordinate systems. The robotic arm 12 moves to the target surgical posture according to the final planned motion path (if the target surgical posture is adjusted in the middle, the target surgical posture here refers to the target surgical posture after the last adjustment). During this process, the robotic arm 12 will not collide with surrounding objects and will not move to a singularity.

[0115] In one embodiment, the real surgical space includes a surgical coordinate system, and the augmented reality device 91 is used to: define the display space coordinate system based on the display interface and send it to the navigation system 61; and obtain the coordinate information of the optical target at the end of the robotic arm to be identified in the display interface and send it to the navigation system 61. The navigation system 61 is also used to: obtain the coordinate information of the optical targets (femur target 22 and tibial target 23, the femur target 22 and tibial target 23 can also be called patient targets) on the entity to be operated on (for example: the patient's bones) in the surgical coordinate system and the coordinate information of the base target 24 on the surgical device 11; and based on the coordinate information of the optical targets (femur target 22 and tibial target 23) on the entity to be operated on (for example: the patient's bones) in the surgical coordinate system, align the virtual surgical model and the entity to be operated on (the patient's bones), and align and determine the coordinate transformation relationship between the virtual surgical space and the surgical coordinate system. The virtual surgical space is a preoperative medical image. The space in which it is located; obtain the coordinate information of the tool target 21 on the robotic arm 12 of the surgical device 11 in the surgical coordinate system, and establish a robotic arm motion coordinate system with the center point on the osteotomy groove surface 32 on the osteotomy guide 31 at the end of the robotic arm 12 as the origin, and establish a conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system; finally, based on the coordinate information of the optical target (base target 24) on the surgical device 11 in the surgical coordinate system and the center point coordinate information of the identification pattern 242 to be identified in the display interface of the augmented reality device 91, the corresponding coordinates are matched to determine the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system.

[0116] The navigation system 61 may include an optical tracking system that is communicatively connected to the surgical device 11 and the augmented reality device 91. The optical tracking system is used to define a surgical coordinate system and obtain coordinate information of optical targets (such as the femoral target 22 and tibial target 23 shown in the figure) on the skeletal entity to be operated on in the surgical coordinate system, as well as coordinate information of the tool target 21 on the robotic arm 12 of the surgical device 11 in the surgical coordinate system. The system then establishes a robotic arm motion coordinate system with the center point of the osteotomy groove surface 32 on the osteotomy guide 31 at the end of the robotic arm 12 as the origin, and obtains the conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system. The optical tracking system is used to define a surgical coordinate system and obtain coordinate information of the base target 24 on the surgical device 61 in the surgical coordinate system. The system then calculates the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system, combining the display space coordinate system obtained from the augmented reality device 91 and the coordinate information of the center point of the identification pattern 242 to be identified in its display interface. Furthermore, the conversion relationship between the display space coordinate system and the robotic arm motion coordinate system is indirectly established.

[0117] The optical tracking system may include an optical camera 51, an auxiliary display 52, and other devices. Coordinate measurement can be performed. When it includes the optical camera 51, it can self-position and construct an optical positioning system coordinate system, which corresponds to the surgical coordinate system.

[0118] In one embodiment, a medical device is provided at the end of the robotic arm 12, and the real surgical space also includes a robotic arm motion coordinate system. A tool target 21 is provided on the surgical device 11, and the tool target 21 is set at the end of the robotic arm 12. The navigation system 61 is also used to: establish the robotic arm motion coordinate system of the robotic arm 12 with the center point of the osteotomy groove surface 32 of the osteotomy guide tool 31 at the end of the robotic arm 31 as the origin. Specifically, the coordinates of the four points A1B1C1D1 (or at least four points) on the osteotomy groove surface 32 corresponding to the pointed target 241 (probe) in the surgical coordinate system are obtained and converted into the tool target 21 coordinate system of the optical camera 51 of the navigation system 61 in the real surgical space; based on the coordinate positions of the four points A1B1C1D1 on the osteotomy groove surface 32 obtained by the pointed target 241 and the coordinate position of the tool target 21 of the optical camera 51 in the real surgical space, the coordinate conversion relationship between the surgical coordinate system (tool target 21 coordinate system) and the robotic arm motion coordinate system (osteotomy groove surface 32 coordinate system) is determined.

[0119] By using this method of optical target positioning and matching the coordinates of the pointed target (probe), the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system is determined. No major changes to the existing navigation system 61 are required, only software improvements are required. The addition of an augmented reality device 91 can be used to preview the robotic arm motion process, which is low-cost and effective, facilitating large-scale promotion and application.

[0120] In one embodiment, Figure 1 As shown, the process of using the optical tracking system to define a surgical coordinate system can be to obtain the coordinates of the base target 24 of the surgical device 11 in the coordinate system of the optical positioning system (for example, the optical camera 51), and use this point as the coordinate origin to construct the surgical coordinate system.

[0121] In one embodiment, the navigation system 61 is also used to align the virtual surgical model (such as the bone model of the preoperative medical image) and the bone entity to be operated on based on the coordinate information of the optical targets (femur target 22 and tibial target 23) in the surgical coordinate system.

[0122] The augmented reality device 91 is used to define a display space coordinate system based on the display interface, and based on the coordinate information of the optical target (base target 24) on the surgical device 11 in the surgical coordinate system, establish a display space coordinate system with the center point of the identification pattern 242 to be identified in the display interface as the origin.

[0123] In one embodiment, a base target 24 may be provided on the base of the surgical device 11 to facilitate positioning by the optical tracking system. Specifically, the optical camera 51 of the optical tracking system identifies the base target 24 to construct a surgical coordinate system.

[0124] Furthermore, a surgical coordinate system (base target 24 coordinate system) and a display space coordinate system (recognition pattern 242 coordinate system) are constructed.

[0125] The above-mentioned process of constructing the coordinate transformation relationship between the surgical coordinate system and the display space coordinate system may be to establish an image coordinate system with the center point of the recognition image 242 in the display field of the augmented reality device 91 as the origin, and Figure 7 As shown, the optical positioning system (optical camera 51 ) is used to obtain the coordinates of at least four points, such as ABCD, of the tip target 241 corresponding to the identification image 242 , and convert them into the surgical coordinate system (base target 24 coordinate system).

[0126] The coordinates of the corresponding points obtained in the first two steps can be used to obtain the coordinate transformation relationship between the image coordinate system and the surgical coordinate system using a matching algorithm.

[0127] Then, the image recognition algorithm is used to obtain the coordinates of the four points ABCD in the display space coordinate system;

[0128] Then, the coordinates of the corresponding points obtained by the coordinates of the four points ABCD in the image coordinate system and the coordinates of the four points ABCD in the display space coordinate system can be used to obtain the coordinate conversion relationship between the image coordinate system and the display space coordinate system using a matching algorithm.

[0129] Furthermore, based on the coordinate conversion relationship between the image coordinate system and the surgical coordinate system and the coordinate conversion relationship between the image coordinate system and the display space coordinate system, the coordinate conversion relationship between the surgical coordinate system (base target 24 coordinate system) and the display space coordinate system can be obtained.

[0130] It should be noted that, in the case where the real surgical space includes two coordinate systems, the surgical coordinate system (the base target 24 coordinate system) and the robotic arm motion coordinate system (the coordinate system where the tool target 21 is located), the motion control of the robotic arm 12 in the real surgical space in the above embodiments can be understood as being based on the planned motion path of the robotic arm 12 in the robotic arm motion coordinate system. The acquisition of the planned motion path in the robotic arm motion coordinate system can be determined based on the conversion relationship between several coordinate systems.

[0131] In one embodiment, the navigation system is further configured to determine a planned motion path within the robotic arm motion coordinate system based on the robotic arm's starting position and target surgical position. The augmented reality device 91 is configured to display, on the augmented reality device 91, a posture change process of the virtual robotic arm model based on the coordinate transformation relationship between the virtual surgical space and the surgical coordinate system, the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system, the coordinate transformation relationship between the surgical coordinate system and the display space coordinate system, and the planned motion path within the robotic arm motion coordinate system. The posture change process of the virtual robotic arm model includes the virtual motion path and virtual posture of the virtual robotic arm model.

[0132] Based on the coordinate transformation relationship between the virtual surgical space and the surgical coordinate system, and the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system, the coordinate transformation relationship between the virtual surgical space and the robotic arm motion coordinate system can be determined. This coordinate transformation relationship can then be used to obtain a virtual motion path and virtual posture based on the planned motion path. The coordinate transformation relationship between the virtual surgical space and the display space coordinate system can then be further used (using the surgical coordinate system as an intermediate transformation medium to achieve the transformation between the two coordinate systems) to transform the virtual motion path and virtual posture into the display space coordinate system of the augmented reality device 91 for display.

[0133] Real objects in the operating room can also be converted to the display space coordinate system for display based on the conversion relationship between their coordinate system and the display space coordinate system. Through this virtual-real fusion, the virtual position changes of the virtual robotic arm model along the virtual motion path are displayed on the augmented reality device 91, showing the relative position relationship between the virtual robotic arm and the real objects in the operating room.

[0134] In one embodiment, the navigation system is also used to: determine the coordinate transformation relationship between the robotic arm motion coordinate system and the display space coordinate system based on the coordinate transformation relationship between the virtual surgical space and the surgical coordinate system, the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system, and the coordinate transformation relationship between the surgical coordinate system and the display space coordinate system; obtain the final planned motion path of the robotic arm in the robotic arm motion coordinate system based on the current starting position of the robotic arm 12 in the robotic arm motion coordinate system, the virtual motion path and virtual posture in the display space coordinate system, and the coordinate transformation relationship between the robotic arm motion coordinate system and the display space coordinate system; control the robotic arm to move to the target surgical position according to the final planned motion path in the robotic arm motion coordinate system.

[0135] If the user determines that the virtual robotic arm's posture changes displayed on the augmented reality device 91 will not cause a robotic arm collision or movement to a singular point, an inverse transformation can be used to convert the virtual motion path and virtual posture in the display space coordinate system into a motion path and posture in the robotic arm's motion coordinate system. The resulting motion path and posture after conversion becomes the final planned motion path. The robotic arm 12 moves to the target surgical posture based on the final planned motion path and performs subsequent surgical operations. The robotic arm 12 does not collide or become unable to move during its movement.

[0136] In orthopedic surgery applications, in order to further improve the accuracy of positioning navigation, considering that it is not easy for the optical tracking system to obtain the specific position coordinates of each part of the osteotomy guide 31 at a fixed position, in one embodiment, the optical tracking system is also used to:

[0137] A coordinate system of the mechanical motion space of the robotic arm 12 is established with the center point of the osteotomy groove surface 32 on the osteotomy guide plate 31 as the origin;

[0138] Obtaining coordinates of at least four points on the osteotomy groove surface 32 in the coordinate system of the mechanical motion space;

[0139] Obtaining coordinates of at least four points on another pointed target 241 on the tool target 21 on the end of the robotic arm of the surgical device in a coordinate system of the surgical coordinate system;

[0140] The points on the osteotomy groove surface 32 are feature matched with the points on the tool target 21; the matching process is to match the corresponding marked points.

[0141] Then, based on the feature matching results and the coordinates of points on the osteotomy groove surface 32 and the tool target 21, the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system is determined. By achieving this coordinate transformation relationship, all robotic arm movements performed in the surgical coordinate system in the aforementioned embodiments can be transformed into the robotic arm motion coordinate system. Furthermore, the virtual poses of the virtual robotic arm model and virtual osteotomy guide along the virtual motion path displayed by the aforementioned augmented reality device can be established based on the coordinate transformation relationship between the robotic arm motion coordinate system and the display space coordinate system.

[0142] In one embodiment, Figure 8 As shown, a tool target 21 may be provided at the end of the robotic arm, and the optical tracking system may obtain the established surgical coordinate system based on the position of the tool target 21 to determine the coordinates of the tool target 21 in the surgical coordinate system.

[0143] The calculation process for determining the coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system can be:

[0144] The optical tracking system can be used to establish a coordinate system with the center point of one of the osteotomy groove surfaces 32 of the osteotomy guide plate 31 at the end of the robot arm as the origin, and the coordinate system can be obtained as follows: Figure 8 The coordinates of at least four points shown, for example, four points A1B1C1D1;

[0145] Then obtain the corresponding osteotomy groove surface 32 on the osteotomy guide plate 31. Figure 8 The coordinates of the four points A1B1C1D1 of another pointed target 241 are shown in the coordinate system of an optical positioning system (e.g., an optical camera) and converted to the surgical coordinate system (the coordinate system where the tool target 21 is located);

[0146] The coordinates of the corresponding points obtained in the first two steps can be used to obtain the coordinate conversion relationship between the robot arm motion coordinate system where the osteotomy groove surface 32 at the end of the robot arm 12 is located and the surgical coordinate system where the tool target 21 is located using a matching algorithm.

[0147] like Figure 9 As shown, the coordinate transformation relationship of each spatial coordinate system can be determined as shown in the figure. Specifically, the transformation relationship calculation process of each spatial coordinate system can be:

[0148] The coordinate transformation relationship between the virtual surgical coordinate system V and the robotic arm motion coordinate system R can be calculated by the following formula, which is used to transform the virtual target osteotomy surface information into the robotic arm motion coordinate system:

[0149] RT V =CT V *RT C

[0150] Among them, the transformation relationship CT between the virtual surgical coordinate system V and the surgical coordinate system C is V It can be obtained through the coordinate conversion relationship between the point cloud data of the virtual surgical model surface reconstructed in the CT (Computed Tomography) image coordinate system and the characteristic point cloud data on the surface of the patient's bone to be operated on recorded by the optical tracking system.

[0151] Transformation relationship RT between surgical coordinate system C and robotic arm motion coordinate system R C This can be achieved through the process described in the above embodiment.

[0152] In addition, the coordinate transformation relationship from the display space coordinate system D to the robotic arm motion coordinate system R can be calculated by the following formula, which is used for the augmented reality device to display the robotic arm posture change and the robotic arm posture adjustment:

[0153] RT D =DT C *RT C

[0154] Among them, the transformation relationship CT between the spatial coordinate system D and the surgical coordinate system C is shown V It can be obtained through the calculation process in the above embodiment.

[0155] In one embodiment, the process of determining the virtual motion trajectory may be:

[0156] According to the coordinate transformation relationship of the above-mentioned spatial coordinate systems, the osteotomy surface grooves of the osteotomy guide plate at the end of the robotic arm (used to guide the doctor to Figure 1 a conversion matrix between the oscillating saw 41 positioned to the target osteotomy surface) and the corresponding virtual target osteotomy surface;

[0157] According to the transformation matrix, the starting position of the robot arm 12 and the target surgical position, the robot inverse kinematics is used to calculate the motion parameters of each joint of the robot arm 12 when the robot arm moves from the starting position to the target osteotomy position;

[0158] A virtual motion trajectory is generated based on the motion parameters of each of the 12 joints of the robotic arm.

[0159] In one embodiment, the robot arm 12 can be a 5-axis robot arm 12, such as Figure 10 As shown, of course, it should be noted that the control system of the present application is also applicable to robotic arms 12 with other numbers of axes.

[0160] The osteotomy guide plate 31 is mounted on the end of the robotic arm 12 via the interface 33 . The osteotomy guide plate 31 can be rotated by controlling the rotation of the axis of the robotic arm 12 to drive the osteotomy guide plate 31 to align with the target osteotomy surface for osteotomy guidance.

[0161] In order to further improve the accuracy of the robot positioning and navigation, a robot motion coordinate system can be established based on the osteotomy guide, which is different from the surgical coordinate system where the bone entity is located. Based on the above coordinate system transformation, based on the coordinate conversion relationship between the robot motion coordinate system and the display space coordinate system, the virtual posture of the virtual robot model and the virtual osteotomy guide on the virtual path is displayed in the display space coordinate system. The generation process of the virtual path and virtual posture is more accurate because it is determined based on the coordinates in the robot motion coordinate system. When the robot motion coordinate system is established based on the guide groove on the osteotomy guide surface, the motion trajectory of the robot and the osteotomy guide previewed in the display space coordinate system more accurately reflects the movement of the osteotomy guide at the end of the robot arm in the real surgical space.

[0162] The above systems, such as Figure 11 As shown, based on the virtual motion path of the robotic arm and the virtual posture within the path displayed by the augmented reality device 91, the doctor can determine in advance whether the robotic arm 12 will collide with an interference object or reach the singularity point of the robotic arm 12 when performing osteotomy positioning based on the current initial and final postures, thereby improving the safety of surgical osteotomy positioning. In addition, the doctor can be guided to non-contactly operate the robotic arm 12 to the appropriate initial position and motion path before performing the automatic osteotomy positioning operation, avoiding repeated automatic positioning and improving the efficiency of surgical osteotomy positioning. In addition, the method of guiding the doctor to operate the robotic arm 12 non-contact saves manpower, improves the intelligence level of the surgery, and reduces the risk of contaminating the sterile field.

[0163] In a second aspect, the present application also provides a control method, which is applied to the above-mentioned surgical robot system, such as Figure 12 As shown, the method includes:

[0164] S200: Acquiring a planned motion path in a real surgical space determined by a navigation system; wherein the planned motion path includes the motion path and information on position changes of a robotic arm along the motion path, the robotic arm being mounted on a surgical device;

[0165] S400: Displaying a posture change process of the virtual robotic arm model on the augmented reality device according to the planned motion path, where the posture change process of the virtual robotic arm model includes a virtual motion path and a virtual posture of the virtual robotic arm model.

[0166] In one embodiment, Figure 13 As shown, the above control method also includes:

[0167] S600: Responding to the adjustment operation performed on the posture change process of the virtual robotic arm model to form a corrected planned motion path;

[0168] S800: Displaying the posture change process of the virtual robotic arm model on the augmented reality device according to the corrected planned motion path.

[0169] Regarding the definitions of various terms and their implementation processes in this method embodiment, please refer to the descriptions in the above-mentioned system embodiment and will not be elaborated here. It should be noted that the execution subject of this method can be the subject described in the above-mentioned system embodiment, or it can be implemented by a separate controller. If implemented by a separate controller, the controller can communicate with the above-mentioned surgical device and augmented reality device, obtain the data generated by them, and control the movement of the surgical device to drive the robotic arm.

[0170] In one embodiment, in response to the adjustment operation performed on the posture change process of the virtual robotic arm model, a revised planned motion path is formed, such as Figure 14 Shown, including:

[0171] S620: Displaying the posture adjustment interface;

[0172] S640: In response to the adjustment action performed on the posture change process of the virtual robotic arm model in the posture adjustment interface, a corrected planned motion path is formed.

[0173] In one embodiment, the adjustment action performed on the posture change process of the virtual robotic arm model includes at least one of a starting posture adjustment action, an on-path robotic arm posture adjustment action, and a target surgical posture adjustment action.

[0174] In one embodiment, Figure 14 As shown, in response to the adjustment operation performed on the posture change process of the virtual robotic arm model, a revised planned motion path is formed, including:

[0175] S660: marking the virtual robotic arm model during the posture change process of the virtual robotic arm model;

[0176] S680: Responding to the posture adjustment operation on the marked area of the virtual robotic arm model, forming a corrected planned motion path.

[0177] In one embodiment, Figure 15 As shown, the above method also includes:

[0178] S900: In response to a path confirmation action performed after displaying a posture change process of the virtual robotic arm model, sending a virtual motion path and a virtual posture to a target object, so that the target object determines a final planned motion path based on the virtual motion path and the virtual posture;

[0179] The final planned motion path is used to instruct the robotic arm to move to the target surgical position according to the final planned motion path, and the target object is the navigation system or surgical device.

[0180] In one embodiment, before the step of determining the planned motion path and posture change information based on the starting posture and target surgical posture of the robotic arm, the following steps may be performed:

[0181] Acquire a CT image; the CT image data includes a skeletal entity to be operated on;

[0182] Performing bone segmentation on the CT image;

[0183] Perform three-dimensional skeleton reconstruction based on the skeleton segmentation results;

[0184] Marking anatomical landmarks on the CT image;

[0185] The prosthesis is positioned based on the anatomical landmarks to obtain surgical plan information such as the target osteotomy surface.

[0186] During surgery, the surgical device mounted on the robotic arm and the navigation system equipped with an optical tracking system are placed in a suitable location next to the patient bed 81. As described in the above embodiments, optical targets are installed on the physical skeleton, the robotic arm, and the surgical device to facilitate optical tracking, locate key locations, and establish a coordinate system. Position data of the optical targets on the physical skeleton is then collected. Based on the acquired feature points of the physical skeleton in real space and the feature points of the virtual surgical model in the virtual surgical space, a feature matching algorithm is used to match the real surgical space with the virtual surgical space, establishing a coordinate transformation relationship between the two spaces.

[0187] Then, the intraoperative plan is adjusted, and the display scheme of the virtual posture of the virtual robotic arm model and the virtual osteotomy guide under the virtual motion path based on the augmented reality device in the above embodiment is executed. Based on the displayed content, it is determined whether it is necessary to adjust the starting posture of the robotic arm or the target surgical posture, etc. If it is determined that there will be no collision or the robotic arm reaches a singular point during movement, the robotic arm is driven to automatically move according to the newly generated planned motion path and posture change information to perform the osteotomy operation. The doctor then installs the prosthesis and completes the operation. During the intraoperative process, there is no need to stop the robotic arm multiple times for posture adjustment, which can greatly improve operational efficiency.

[0188] Regarding the implementation process of the solution in the above method embodiment, please refer to the description in the above system embodiment, and no further details will be given here.

[0189] Furthermore, it should be emphasized that the control method provided in the embodiments of the present application also includes the various steps performed by the surgical device, augmented reality device, and optical tracking system in the aforementioned system embodiments, and achieves the corresponding beneficial effects. Of course, this step can also be performed by a separate controller, as described in the above embodiments, and will not be further elaborated here.

[0190] It should be noted that, in the above embodiment, the confirmation of the coordinate conversion relationship of the spatial coordinate system refers to the confirmation of the coordinate conversion relationship between various spaces.

[0191] It should be understood that, although the various steps in the flowcharts are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0192] In a third aspect, the present application also provides a control device, which is applied to the above-mentioned surgical robot system, such as Figure 16 As shown, the device includes:

[0193] A path planning module 200 is configured to obtain a planned motion path in a real surgical space determined by a navigation system; wherein the planned motion path includes the motion path and information on position changes of the robotic arm along the motion path, the robotic arm being mounted on the surgical device;

[0194] The preview module 400 is used to display the posture change process of the virtual robotic arm model on the augmented reality device according to the planned motion path. The posture change process of the virtual robotic arm model includes the virtual motion path and virtual posture of the virtual robotic arm model.

[0195] For the specific definition of the control device, please refer to the definition of the control method above, which will not be repeated here. The various modules in the above-mentioned control device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0196] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 17 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a control method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0197] Those skilled in the art will understand that Figure 17 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present invention and does not limit the computer device to which the present invention is applied. Specific computer devices may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. The computer device may be at least one of the aforementioned surgical devices, augmented reality devices, and optical tracking systems. It may also be an independent controller.

[0198] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, any step of the above control method is implemented, and corresponding beneficial effects are achieved.

[0199] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, part or all of the steps of the above-mentioned control method are implemented, and corresponding beneficial effects are achieved.

[0200] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0201] Throughout this specification, references to terms such as "some embodiments," "other embodiments," and "desired embodiments" indicate that a particular feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Although these terms are used interchangeably throughout this specification, they do not necessarily refer to the same embodiment or example.

[0202] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0203] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A surgical robot system, characterized in that: include: A surgical device equipped with a robotic arm; a navigation system for determining a planned motion path of the robotic arm in a real surgical space based on the starting position and target surgical position of the robotic arm, the planned motion path comprising a motion path and information on position changes of the robotic arm along the motion path; as well as an augmented reality device, the augmented reality device being communicatively connected to the navigation system and configured to obtain the planned motion path and display a posture change process of the virtual robotic arm model on the augmented reality device according to the planned motion path, wherein the posture change process of the virtual robotic arm model includes a virtual motion path and a virtual posture of the virtual robotic arm model; The augmented reality device is also used to respond to the adjustment operation on the posture change process of the virtual robotic arm model to form a corrected planned motion path. The adjustment operation includes adjusting the starting posture and target surgical posture of the robotic arm in the real surgical space, or adjusting the posture of the virtual robotic arm model displayed by the augmented reality device.

2. The surgical robot system according to claim 1, wherein: The augmented reality device is further used to display the posture change process of the virtual robotic arm model on the augmented reality device according to the corrected planned motion path.

3. The surgical robot system according to claim 1, wherein: The augmented reality device is used to adjust the posture change process of the virtual robotic arm model in response to the operation to form a corrected planned motion path, including: Display the posture adjustment interface; In response to the adjustment action performed on the posture change process of the virtual robotic arm model on the posture adjustment interface, a corrected planned motion path is formed.

4. The surgical robot system according to claim 3, wherein: The adjustment action performed on the posture change process of the virtual robotic arm model includes at least one of an initial posture adjustment action, an on-path robotic arm posture adjustment action, and a target surgical posture adjustment action.

5. The surgical robot system according to any one of claims 1 to 4, characterized in that: The augmented reality device is configured to send the virtual motion path and the virtual posture to a target object in response to a path confirmation action performed after displaying the posture change process of the virtual robotic arm model, wherein the target object is the navigation system or the surgical device; The target object is also used to: Calculating a final planned motion path of the robotic arm in the real surgical space based on the current starting position of the robotic arm in the real surgical space, the virtual motion path, the virtual position and the conversion relationship between the display space coordinate system of the augmented reality device and the real surgical space; In the real surgical space, the robotic arm moves to the target surgical posture according to the final planned motion path.

6. The surgical robot system according to claim 1, wherein: The real surgical space includes a surgical coordinate system, and the augmented reality device is used to: Defining a display space coordinate system based on a display interface and sending the display space coordinate system to the navigation system; Acquiring coordinate information of an identification pattern on the surgical device to be identified in the display interface and sending the information to the navigation system; The navigation system is also used to: Defining the surgical coordinate system based on the optical target on the entity to be operated on, and acquiring coordinate information of the optical target on the entity to be operated on and coordinate information of the optical target of the base target on the surgical device in the surgical coordinate system; Based on the coordinate information of the optical target of the base target on the surgical device in the surgical coordinate system and the coordinate information of the identification pattern to be identified in the display interface, feature matching of corresponding point coordinates is performed to determine the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system.

7. The surgical robot system according to claim 6, characterized in that: The end of the robotic arm is provided with a medical instrument, the real surgical space further includes a robotic arm motion coordinate system, the surgical device includes a medical instrument, the optical target on the surgical device further includes a tool target, and the tool target is provided at the end of the robotic arm, and the navigation system is further used for: Establishing a motion coordinate system of the robotic arm with the center point of at least four points on one end surface of the medical device as the origin; Obtaining coordinates of at least four points on the end surface in the robotic arm motion coordinate system; Acquiring the coordinates of the tool target in the surgical coordinate system; The coordinates of at least four points on the end surface are matched with the coordinates of the tool target to determine the coordinate conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system.

8. The surgical robot system according to claim 7, characterized in that: The navigation system is also used to: Determining a planned motion path of the robotic arm in a motion coordinate system based on the starting position and target surgical position of the robotic arm; The augmented reality device is used for: According to the coordinate conversion relationship between the virtual surgical space and the surgical coordinate system, the coordinate conversion relationship between the surgical coordinate system and the robotic arm motion coordinate system, the coordinate conversion relationship between the surgical coordinate system and the display space coordinate system, and the planned motion path under the robotic arm motion coordinate system, the posture change process of the virtual robotic arm model is displayed on the augmented reality device. The posture change process of the virtual robotic arm model includes the virtual motion path and virtual posture of the virtual robotic arm model.

9. The surgical robot system according to claim 8, characterized in that: The navigation system is also used to: Determining a coordinate transformation relationship between the robotic arm motion coordinate system and the display space coordinate system based on a coordinate transformation relationship between the virtual surgical space and the surgical coordinate system, a coordinate transformation relationship between the surgical coordinate system and the robotic arm motion coordinate system, and a coordinate transformation relationship between the surgical coordinate system and the display space coordinate system; Obtaining a final planned motion path of the robotic arm in the robotic arm motion coordinate system based on the current starting pose of the robotic arm in the robotic arm motion coordinate system, the virtual motion path and the virtual pose in the display space coordinate system, and a coordinate transformation relationship between the robotic arm motion coordinate system and the display space coordinate system; The robotic arm is controlled to move to the target surgical posture according to the final planned motion path in the robotic arm motion coordinate system.

10. A control method, characterized in that: The surgical robot system according to any one of claims 1 to 9, wherein the method comprises: Acquiring a planned motion path in a real surgical space determined by a navigation system; wherein the planned motion path includes the motion path and position change information of a robotic arm moving along the motion path, the robotic arm being mounted on a surgical device; Displaying a posture change process of the virtual robotic arm model on the augmented reality device according to the planned motion path, wherein the posture change process of the virtual robotic arm model includes a virtual motion path and a virtual posture of the virtual robotic arm model; In response to an adjustment operation performed on the posture change process of the virtual robotic arm model, a revised planned motion path is formed, wherein the adjustment operation includes adjusting the starting posture and target surgical posture of the robotic arm in the real surgical space, or adjusting the posture of the virtual robotic arm model displayed by the augmented reality device.

11. The method according to claim 10, characterized in that The method further comprises: The posture change process of the virtual robotic arm model is displayed on the augmented reality device according to the corrected planned motion path.

12. The method according to claim 10, characterized in that In response to the adjustment operation performed on the posture change process of the virtual robotic arm model, a revised planned motion path is formed, including: Display the posture adjustment interface; In response to the adjustment action performed on the posture change process of the virtual robotic arm model on the posture adjustment interface, a corrected planned motion path is formed.

13. The method according to claim 12, characterized in that The adjustment action performed on the posture change process of the virtual robotic arm model includes at least one of an initial posture adjustment action, an on-path robotic arm posture adjustment action, and a target surgical posture adjustment action.

14. The method according to claim 10, characterized in that The response adjusts the posture change process of the virtual robotic arm model to form a corrected planned motion path, including: Marking the virtual robotic arm model during the posture change process of the virtual robotic arm model; In response to the posture adjustment operation on the marked area of the virtual robotic arm model, a revised planned motion path is formed.

15. The method according to any one of claims 10 to 14, characterized in that The method further comprises: In response to a path confirmation action performed after displaying the posture change process of the virtual robotic arm model, sending the virtual motion path and the virtual posture to a target object, so that the target object determines a final planned motion path based on the virtual motion path and the virtual posture; The final planned motion path is used to instruct the robotic arm to move to a target surgical posture according to the final planned motion path, and the target object is the navigation system or the surgical device.

Images