Soft tissue cooperative operation control method and system based on humanoid robot

Abstract

This invention provides a method and system for soft tissue collaborative surgery control based on a humanoid robot. It relates to the fields of medical robotics, computer vision, and remote teleoperation. The method includes: displaying, via a display terminal, image stream information acquired in real time by the humanoid robot for the surgical operation area, wherein the surgical operation area includes the area where a first instrument held by a first robotic hand of the humanoid robot is located; in response to detecting a grasping operation representing soft tissue grasped by the first instrument, obtaining the tissue deformation rate of the grasped soft tissue; generating, based on the tissue deformation rate of the grasped soft tissue, a prompt message for indicating the tissue stretching state on the display terminal; and sending the prompt message to the display terminal for display.

Description

Technical Field

[0001] This invention relates to the fields of medical robots, computer vision, and remote teleoperation technology, and more specifically, to a soft tissue collaborative surgical control method and system based on a humanoid robot. Background Technology

[0002] With the rapid development of humanoid robot technology, its dual arms, dexterous hands, and humanoid torso structure give it a natural advantage in adapting to human working environments. In high-risk scenarios such as disaster relief and battlefield first aid, using humanoid robots to replace medical personnel on-site and perform emergency surgical procedures such as hemostasis and wound cleaning through remote operation is an important application direction of current robotics technology.

[0003] In realizing the concept of this invention, the inventors discovered at least the following problems in the related technology: During remote operation, the operator can only observe a two-dimensional or three-dimensional video stream through a screen. Regarding the condition of soft tissue, the operator can only make visual estimations based on experience, making it difficult to guarantee the accuracy of the estimation results. Summary of the Invention

[0004] In view of this, the present invention provides a method and system for soft tissue collaborative surgery control based on a humanoid robot.

[0005] One aspect of the present invention provides a soft tissue collaborative surgical control method based on a humanoid robot, comprising: displaying image stream information acquired in real time by the humanoid robot for a surgical operation area via a display terminal, wherein the surgical operation area includes the area where a first instrument held by a first robotic hand of the humanoid robot is located; in response to detecting a grasping operation characterizing soft tissue grasped by the first instrument, obtaining the tissue deformation rate of the grasped soft tissue; generating a prompt message for indicating the tissue stretching state on the display terminal based on the tissue deformation rate of the grasped soft tissue; and sending the prompt message to the display terminal for display.

[0006] Another aspect of the present invention provides a soft tissue collaborative surgery control system based on a humanoid robot, comprising: an onboard core control unit for implementing the soft tissue collaborative surgery control method based on a humanoid robot of the present invention; a main control end including a handle and a display terminal, the handle being held by the operator; and a humanoid robot end including a joint motion controller and an image acquisition module, the image acquisition module being used to acquire image stream information in real time for the surgical operation area; wherein, the display terminal is used to display the image stream information acquired by the image acquisition module and processed by the onboard core control unit, and the joint motion controller is used to control the movement state of the target robot hand on the humanoid robot end according to the operator's movement state of the handle processed by the onboard core control unit.

[0007] Another aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the humanoid robot-based soft tissue collaborative surgery control method of the present invention.

[0008] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the humanoid robot-based soft tissue collaborative surgical control method of the present invention.

[0009] According to an embodiment of the present invention, by employing a technical means of responding to the detection of a grasping operation characterizing the soft tissue grasped by the first instrument, obtaining the tissue deformation rate of the grasped soft tissue; generating a prompt message for indicating the tissue stretching state on a display terminal based on the tissue deformation rate of the grasped soft tissue; and sending the prompt message to the display terminal for display, the operator can intuitively obtain the accurate state by directly outputting the prompt message indicating the soft tissue stretching state through the display terminal, without the need for manual estimation. This is beneficial to improving the accuracy of stretching state judgment and enhancing the reliability and safety of surgical operations. Attached Figure Description

[0010] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0011] Figure 1 The illustration schematically depicts an exemplary system architecture of a humanoid robot-based soft tissue collaborative surgical control system according to an embodiment of the present invention;

[0012] Figure 2 A flowchart illustrating a soft tissue collaborative surgical control method based on a humanoid robot according to an embodiment of the present invention is shown schematically.

[0013] Figure 3 This schematically illustrates an overall flowchart of the surgical procedure according to an embodiment of the present invention;

[0014] Figure 4 A block diagram of a humanoid robot-based soft tissue collaborative surgical control device according to an embodiment of the present invention is shown schematically.

[0015] Figure 5 A block diagram of an electronic device suitable for implementing a humanoid robot-based soft tissue collaborative surgical control method is shown schematically according to an embodiment of the present invention. Detailed Implementation

[0016] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0017] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0018] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0019] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0020] In the embodiments of this invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security.

[0021] In the embodiments of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0022] In remote surgical scenarios, operators typically use general-purpose spatial input devices to control robots. However, directly applying such general-purpose solutions to complex soft tissue surgeries presents the following significant challenges.

[0023] Subjectivity in judging soft tissue condition: In remote operation, the operator can only observe a two-dimensional or three-dimensional video stream on a screen. Whether the soft tissue has been stretched or reached a suitable tension level for cutting can only be estimated visually based on experience. This vague visual judgment can easily lead to insufficient traction (unable to cut) or excessive traction (causing tissue tearing).

[0024] The contradiction between macro- and micro motion control: General-purpose humanoid robots are typically designed for large-scale operations, while surgical procedures such as vascular dissection require millimeter-level precision. Direct motion mapping amplifies even the smallest tremors in the operator's hand, making it difficult to perform delicate operations.

[0025] Trajectory conflict in dual-arm coordination: When performing a coordinated operation of "left-hand traction and right-hand cutting," the operator needs to plan the spatial trajectory of both robotic arms simultaneously. Without auxiliary guidance, it is highly likely that the instruments will exceed the safe operating space, accidentally damaging surrounding healthy tissue.

[0026] Therefore, quantifying tissue state through visual algorithms and endowing humanoid robots with refined motion control capabilities are key to achieving highly reliable remote surgery.

[0027] Based on this, embodiments of the present invention provide a soft tissue collaborative surgery control method and system based on a humanoid robot.

[0028] According to embodiments of the present invention, a humanoid robot-based soft tissue collaborative surgical control system employs a dual control strategy of "visual deformation analysis + kinematic virtual constraints," constructing a purely visual front-end perception layer and a purely kinematic back-end control layer. Regarding the front-end perception layer, computer vision algorithms can be used to track the surface texture features of soft tissue in real time, and based on the calculated degree of tissue stretching deformation, it is converted into a visual color halo superimposed on the video stream to assist the operator in judging the tension state. Regarding the back-end control layer, heterogeneous mapping algorithms and virtual gripper technology can be embedded in the humanoid robot controller. The left arm is responsible for position-based intelligent locking, and the right arm is responsible for fine scaling based on electronic fences, ensuring the safety of the surgical trajectory from a kinematic perspective. The process of calculating the degree of tissue stretching deformation and converting it into a halo superimposed on the video stream can be implemented by the back-end control layer.

[0029] Figure 1 An exemplary system architecture of a humanoid robot-based soft tissue collaborative surgical control system according to an embodiment of the present invention is illustrated.

[0030] like Figure 1As shown, the control system 100 includes a main control terminal 110, an airborne core control unit 120, and a humanoid robot terminal 130. The main control terminal 110 and the airborne core control unit 120, as well as the airborne core control unit 120 and the humanoid robot terminal 130, are connected via communication links. The main control terminal 110 includes a handle 111 and a display terminal 112. The humanoid robot terminal 130 includes a joint motion controller 131 and an image acquisition module 132.

[0031] The main control terminal 110 serves as the interaction medium for the operator to control the humanoid robot terminal 130. It uses a universal spatial positioning handle 111 as the input device, which is held by the operator. The universal spatial positioning handle 111 can output 6-DOF pose signals, including position and orientation, as well as button status. The display terminal 112 can be a head-mounted display or a high-resolution display screen, used to display a low-latency stereoscopic video stream overlaid with augmented reality UI (User Interface) information.

[0032] The humanoid robot end effector 130 has a body structure that can be a humanoid robot with a redundant configuration of 7 degrees of freedom and two arms. Its vision system, which serves as the image acquisition module 132, can employ a head-mounted binocular stereo camera to acquire real-time image stream information of the surgical area, providing high-definition RGB (Red, Green, Blue) images and depth point cloud data of the surgical area. Its end effector includes a first robotic hand and a second robotic hand. The first robotic hand can hold a non-traumatic gripper for traction. The second robotic hand can hold an energy device, such as an electric hook, for cutting.

[0033] The airborne core control unit 120 can be deployed on the robot's onboard computer, or on other devices or servers that have communication connections with the main control terminal 110 and the humanoid robot terminal 130. The airborne core control unit 120 is used to run the real-time control system, and it can be internally designed with a heterogeneous motion mapping module 121, a virtual fixture and motion constraint module 122, an tissue deformation visual analysis module 123, and an AR enhanced rendering engine 124.

[0034] On one hand, the operator manipulates the handle 111, and the operator's actions cause the handle 111 to generate pose commands. The heterogeneous motion mapping module 121 receives the pose commands from the handle 111 and, combined with motion scaling and filtering algorithms, converts the operator's main hand displacement into joint angles of the robot hand. The virtual gripper and motion constraint module 122 can perform 3D environment reconstruction based on the 3D environmental data represented by the image stream information acquired by the image acquisition module 132, and generate an electronic fence that restricts the range of motion of the device. Thus, based on the heterogeneous motion mapping module 121 and the virtual gripper and motion constraint module 122, the corrected joint angles can be generated and sent to the joint motion controller 131 to control the motion state of the target robot hand on the humanoid robot end 130.

[0035] On the other hand, the tissue deformation visual analysis module 123 can receive the original image stream acquired by the image acquisition module 132, and calculate the tissue deformation rate of the stretched soft tissue based on optical flow or feature point matching. Combined with the AR enhanced rendering engine 124, the original image stream is rendered to obtain a rendered video stream with AR UI overlaid, which is then sent to the display terminal 112 for display.

[0036] The aforementioned soft tissue collaborative surgical control system based on humanoid robots utilizes a general-purpose dual-arm humanoid robot and employs visual deformation analysis and kinematic constraint strategies to facilitate the collaborative control of safe traction and precise cutting of biological soft tissue.

[0037] The following describes, in conjunction with specific embodiments, how... Figure 1 The design of the soft tissue collaborative surgery control method based on humanoid robots in the airborne core control unit 120 is further described in detail.

[0038] Figure 2 A flowchart illustrating a soft tissue collaborative surgical control method based on a humanoid robot according to an embodiment of the present invention is shown.

[0039] like Figure 2 As shown, the method includes operations S201 to S204.

[0040] In operation S201, the image stream information collected in real time by the humanoid robot for the surgical operation area is displayed on the display terminal. The surgical operation area includes the area where the first instrument held by the first robotic hand of the humanoid robot is located.

[0041] According to an embodiment of the present invention, the first robotic hand can be any robotic hand of a humanoid robot, adaptively determined according to the operator's usage habits. Generally, the first robotic hand can be the robot's left arm.

[0042] In operation S202, in response to detecting a grasping operation characterizing the soft tissue grasped by the first instrument, the tissue deformation rate of the grasped soft tissue is obtained.

[0043] According to an embodiment of the present invention, based on the establishment of a tension triangle, a pure visual tension analysis algorithm can be used to perform deformation calculation through feature tracking to obtain the tissue deformation rate of the captured soft tissue.

[0044] For example, in a scenario where the operator controls the left arm to grasp and lift tissue, a tension triangle can be established first. Then, the system can automatically select several feature points on the soft tissue surface near the grasping point for feature tracking. Subsequently, as the left arm lifts, the system can calculate the rate of change of distance between the feature points in real time, i.e., tensile strain, thereby calculating the tissue deformation rate of the grasped soft tissue.

[0045] In operation S203, based on the tissue deformation rate of the captured soft tissue, a prompt message is generated to indicate the tissue stretching status on the display terminal.

[0046] According to embodiments of the present invention, the prompt message can be a pop-up prompt, such as using only text. It can also be a color prompt, such as using different colors for rendering. No limitation is made here.

[0047] When operating S204, a prompt message is sent to the display terminal for display.

[0048] like Figure 1 As shown, based on the organization deformation visual analysis module 123 and AR enhanced rendering engine 124 in the airborne core control unit 120, the above operations S203 to S204 can be performed.

[0049] Through the above embodiments of the present invention, the operator can intuitively obtain accurate information about the soft tissue stretching status by directly outputting prompts on the display terminal, without the need for manual estimation. This helps to improve the accuracy of stretching status judgment and enhance the reliability and safety of surgical operations.

[0050] According to an embodiment of the present invention, the above operation S203 may include: generating a first prompt message in response to determining that the tissue deformation rate is within a first deformation range, wherein the first deformation range indicates that the grasped soft tissue is in a safe stretching state; generating a second prompt message in response to determining that the tissue deformation rate is within a second deformation range, wherein the second deformation range indicates that the grasped soft tissue is in an optimal stretching state; and generating a third prompt message in response to determining that the tissue deformation rate is within a third deformation range, wherein the third deformation range indicates that the grasped soft tissue is in an overstretching state.

[0051] According to embodiments of the present invention, the first, second, and third prompt messages can be text-based pop-up prompts. For example, the first prompt message can be a "Normal" pop-up, the second prompt message can be a "Safe" pop-up, and the third prompt message can be an "Alarm" pop-up.

[0052] According to an embodiment of the present invention, the prompt information may further include halo information, such as using green to represent the first prompt information, yellow to represent the second prompt information, and red to represent the third prompt information. In this scenario, the above operation S204 may include: determining the position of the end of the first instrument in the image stream based on the image stream information; sending the halo information to the position of the end of the instrument; rendering the image stream information to obtain a rendered video stream; and sending the rendered video stream to a display terminal for display.

[0053] Accordingly, the following color mapping logic can be designed to achieve the above rendering and display: Let the first deformation range be a tissue deformation rate of less than 10%, representing a safe zone; the display terminal will show a green halo around the gripper. Let the second deformation range be a tissue deformation rate between 10% and 20%, representing the optimal tension zone; the display terminal will show a yellow halo around the gripper, indicating to the operator that the tissue has been straightened and is suitable for cutting. Let the third deformation range be a tissue deformation rate greater than 20%, representing a danger zone; the display terminal will show a red and flashing halo around the gripper, indicating possible overstretching.

[0054] It should be noted that the actual values ​​of the first deformation range, the second deformation range, and the third deformation range can be preset according to the tissue type, and are not limited here.

[0055] Through the above embodiments of the present invention, an intuitive visual perception closed loop is constructed. Specifically, through innovative "tissue deformation visual analysis" technology, the difficult-to-quantify state of soft tissue tension is transformed into intuitive color changes. Operators do not need to rely on any contact sensors; they can accurately determine whether the tissue is stretched simply by observing the screen, effectively solving the problem of blind spots in remote operation.

[0056] According to an embodiment of the present invention, the above-described soft tissue collaborative surgery control method based on a humanoid robot may further include: in response to detecting that the prompt information is target information, sending a locking command to the humanoid robot; and performing locking operations on each joint of the first robot hand based on the locking command.

[0057] When the prompt is a text-based pop-up, the target information can be "Safe". When the prompt is a halo effect, the target information can be yellow. Specific implementation methods for the prompt and target information are not limited here. The first robotic arm controlled by the lock command can be a robotic arm currently in a working state that does not need to move for a short period of time.

[0058] Corresponding to the scenario described above where the operator controls the left arm to grasp and lift tissue, the first robotic hand is the robot's left arm. For example, when the operator observes the halo turn yellow, they press the handle button to generate a locking command. The system receives the command and immediately locks all joints of the left arm, entering "position holding mode." At this point, the operator can release the left handle, and the robot's left arm will act like a rigid support, stably maintaining the tissue's position.

[0059] Through the above embodiments of the present invention, the ergonomics of dual-arm coordination is optimized. Specifically, by designing a "smart locking" function, the operator is freed from the heavy burden of simultaneous dual-arm control, allowing them to focus on the delicate operations that require the use of hands, significantly reducing the cognitive load and fatigue of remote surgery.

[0060] According to an embodiment of the present invention, the humanoid robot further includes a second robotic hand that holds a second instrument. Corresponding to... Figure 1 The functional design of the virtual fixture and motion constraint module 122, the above-mentioned soft tissue collaborative surgery control method based on humanoid robot may also include: fusing a three-dimensional reconstruction model pre-constructed for the surgical operation object into the image stream information, wherein the three-dimensional reconstruction model is pre-set with an electronic fence for limiting the surgical operation range; in response to detecting that the movement path of the second instrument reaches the edge of the electronic fence, filtering out the displacement component perpendicular to the electronic fence in the movement path.

[0061] Specifically, the system can first identify the anatomical structures to be cut in the current surgical operation area, such as the cystic duct. Then, using 3D visual reconstruction technology, a pre-constructed three-dimensional model of the cystic duct, such as the cystic duct, can be fused into the image stream used to display the current surgical operation area through feature point matching. Based on the pre-determined virtual "funnel-shaped" safety channel in the three-dimensional cystic duct model, an electronic fence for the surgical operation area can be generated, thereby enabling motion constraints based on virtual clamps.

[0062] Motion constraints can be specifically manifested as follows: if the operator's command attempts to deviate the electric hook from the safety passage maintained by the electronic fence, the control algorithm will automatically ignore the displacement component perpendicular to the passage direction. The electric hook on the screen will always remain attached to the safety path, limiting misoperation at the software level and preventing the instrument from scratching surrounding tissues.

[0063] According to an embodiment of the present invention, corresponding to such Figure 1 The heterogeneous mapping strategy of the heterogeneous motion mapping module 121 is functionally designed. The above-mentioned soft tissue collaborative surgery control method based on humanoid robot may further include: in response to detecting that the operator's handle has moved a first distance, scaling the first distance according to a preset scaling ratio to obtain a second distance; sending the second distance to the humanoid robot to control the target robot hand of the humanoid robot to move the second distance, wherein the posture of the operator's handle is mapped to the posture of the target robot hand, and the target robot hand is at least one of the following: a first robot hand and a second robot hand.

[0064] For example, a heterogeneous mapping strategy can be manifested as enabling an ultra-large motion scaling ratio of 10:1. That is, if the operator moves the handle 10 centimeters, the robot's end effector only moves 1 centimeter. This scaling greatly improves the precision of the operation and filters out minute hand tremors.

[0065] It should be noted that 10:1 is only an exemplary embodiment. In actual implementation, an appropriate scaling ratio can be set according to actual business needs.

[0066] The above embodiments of the present invention improve the accuracy and safety of operation. Specifically, a safe electronic fence is constructed at the software level through "virtual gripper" and "motion scaling" technologies. Even if the operator's hand shakes or makes a mistake, the robot's movement trajectory will be restricted within a preset safe passage, achieving intelligent assisted risk avoidance.

[0067] According to embodiments of the present invention, to address the instability in control caused by communication delays or differences in movement speed, a virtual hand can be rendered and displayed on a display terminal, and compared with the real-time motion information of the target robotic hand included in the image stream information to construct a control strategy and visual deviation feedback. Based on this, the aforementioned soft tissue collaborative surgery control method based on a humanoid robot can further include: in response to detecting that the distance between the target robotic hand's position on the display terminal and the virtual hand's position on the display terminal meets a preset condition, keeping the virtual hand and the target robotic hand overlapping on the display terminal, and rendering and displaying a fourth prompt message on the display terminal; in response to detecting that the distance between the target robotic hand's position on the display terminal and the virtual hand's position on the display terminal does not meet the preset condition, rendering and displaying a fifth prompt message on the display terminal.

[0068] The aforementioned preset conditions may include: the distance is less than a preset threshold. The fourth prompt may include a green indicator light indicating that the robot can keep up with the operator's speed. The fifth prompt may include a red indicator light indicating that the robot cannot keep up with the operator's speed.

[0069] The control strategy can be represented as follows: the system renders a semi-transparent virtual "hand" model on the screen, which responds to the operator's hand movements in real time. The physical "robot hand" model then displays the actual position of the robot hand.

[0070] Visual deviation feedback can manifest as follows: when the robotic hand can keep up with the operator's speed, the "virtual hand" and the "robotic hand" overlap, and the indicator light is green. When the operator moves too fast, causing the distance between them to increase, the "virtual hand" turns red (or the indicator light turns red), and a "Wait to follow" message is displayed through the UI.

[0071] Through the above embodiments of the present invention, the operator can be guided to actively adjust the speed of movement through visual separation, thus ensuring the consistency of human-machine movement.

[0072] According to the humanoid robot-based soft tissue collaborative surgery control method and system of the present invention, the surgical operation process can be divided into four key stages, each of which adopts a different control strategy.

[0073] Figure 3 The diagram illustrates the overall flow chart of the surgical procedure according to an embodiment of the present invention.

[0074] like Figure 3 As shown, the method includes operations S301 to S311. Operation S302 is the first stage, used to achieve macroscopic approach based on handle guidance. Operations S303 to S306 are the second stage, in which the task of the first robotic hand is performed to achieve soft tissue traction and deformation visualization. Operations S307 to S308 are the third stage, in which the first robotic hand is locked, and the operator can focus on controlling the second robotic hand, such as an electric hook, to achieve fine cutting based on a virtual gripper. Operations S309 to S310 are the fourth stage, in which collaborative operation and looping are achieved.

[0075] During S301 operation, the system is initialized.

[0076] When operating the S302, the operator's handle and the robot's hand are brought into macroscopic proximity on the display terminal through virtual hand vision guidance.

[0077] When operating S303, the first robotic arm is controlled to grasp soft tissue and establish a tension triangle.

[0078] In operation S304, visual deformation analysis is performed to calculate the soft tissue deformation rate.

[0079] In operation S305, determine whether the halo has turned yellow. If yes, proceed to operation S306; otherwise, return to operation S303.

[0080] During this stage, if the halo is green, the soft tissue continues to be stretched based on operation S303. If the halo is red, the soft tissue is relaxed based on operation S303.

[0081] When operating S306, lock the first robotic arm and enter "position holding mode".

[0082] Operating S307 controls the second robotic arm to perform precision cutting tasks.

[0083] In operation S308, determine whether the segment cutting has been completed. If yes, proceed to operation S309; ​​otherwise, return to operation S307.

[0084] In operation S309, determine whether all cutting has been completed. If not, proceed to operation S310; if yes, proceed to operation S311.

[0085] When operating the S310, unlock the first robotic arm, and resume follow mode, the system automatically clears the current deformation calculation baseline and prepares for the next grasping operation.

[0086] The operation of S311 has ended.

[0087] Based on the above embodiments, this invention significantly reduces hardware costs. Specifically, this invention is entirely based on computer vision algorithms and general motion control algorithms, eliminating the need to install expensive force sensors or modify hardware at the robot's end effector. This allows the system to be directly applied to existing general-purpose humanoid robot platforms, greatly reducing deployment costs and maintenance complexity.

[0088] Figure 4 A block diagram of a humanoid robot-based soft tissue collaborative surgical control device according to an embodiment of the present invention is shown schematically.

[0089] like Figure 4 As shown, the soft tissue collaborative surgery control device 400 based on a humanoid robot includes a display module 410, a tissue deformation rate acquisition module 420, a prompt information generation module 430, and a prompt information display module 440.

[0090] Display module 410 is used to display image stream information collected in real time by the humanoid robot for the surgical operation area through a display terminal, wherein the surgical operation area includes the area where the first instrument held by the first robotic hand of the humanoid robot is located.

[0091] The tissue deformation rate acquisition module 420 is used to obtain the tissue deformation rate of the grasped soft tissue in response to the detection of a grasping operation that characterizes the soft tissue grasped by the first instrument.

[0092] The prompt message generation module 430 is used to generate prompt messages for displaying the tissue stretching status on the display terminal based on the tissue deformation rate of the captured soft tissue.

[0093] The prompt message display module 440 is used to send prompt messages to the display terminal for display.

[0094] According to an embodiment of the present invention, the prompt information generation module includes a first prompt information generation unit, a second prompt information generation unit, and a third prompt information generation unit.

[0095] The first prompt information generation unit is used to generate a first prompt information in response to determining that the tissue deformation rate is within a first deformation range, wherein the first deformation range represents that the captured soft tissue is in a safe stretching state.

[0096] The second prompt information generation unit is used to generate a second prompt information in response to determining that the tissue deformation rate is within a second deformation range, wherein the second deformation range indicates that the captured soft tissue is in an optimal stretching state.

[0097] The third prompt information generation unit is used to generate a third prompt information in response to determining that the tissue deformation rate is within the third deformation range, wherein the third deformation range indicates that the captured soft tissue is in an overstretched state.

[0098] According to an embodiment of the present invention, the prompt information includes halo information. The prompt information display module includes a device end-effector position determination unit, a rendering unit, and a display unit.

[0099] The instrument end position determination unit is used to determine the instrument end position of the first instrument in the image stream based on the image stream information.

[0100] The rendering unit is used to send halo information to the end position of the instrument, render the image stream information, and obtain the rendered video stream.

[0101] The display unit is used to send the rendered video stream to the display terminal for display.

[0102] According to an embodiment of the present invention, the soft tissue collaborative surgery control device based on a humanoid robot further includes a locking command generation module and a locking module.

[0103] The lock command generation module is used to send a lock command to the humanoid robot in response to the detection that the prompt information is the target information.

[0104] The locking module is used to perform locking operations on each joint of the first robotic hand based on locking commands.

[0105] According to an embodiment of the present invention, the humanoid robot further includes a second robotic hand that holds a second instrument. The soft tissue collaborative surgical control device based on the humanoid robot also includes a fusion module and a filtering module.

[0106] The fusion module is used to fuse a pre-built 3D reconstruction model of the surgical object into the image stream information. The 3D reconstruction model is pre-set with an electronic fence to limit the scope of the surgical operation.

[0107] The filtering module is used to filter out the displacement component perpendicular to the electronic fence in the movement path in response to the detection that the movement path of the second device has reached the edge of the electronic fence.

[0108] According to an embodiment of the present invention, the soft tissue collaborative surgery control device based on a humanoid robot further includes a scaling module and a control module.

[0109] The scaling module is used to scale the first distance according to a preset scaling ratio in response to detecting that the operator's handle has moved a first distance, so as to obtain a second distance.

[0110] A control module is used to send a second distance to the humanoid robot to control the target robot hand of the humanoid robot to move a second distance, wherein the posture of the operator's handle is mapped to the posture of the target robot hand, and the target robot hand is at least one of the following: a first robot hand and a second robot hand.

[0111] According to an embodiment of the present invention, a virtual hand is also rendered and displayed on the display terminal, and the image stream information includes real-time motion information of the target robot hand, with a correspondence between the virtual hand and the target robot hand. The soft tissue collaborative surgery control device based on a humanoid robot also includes a fourth prompting module and a fifth prompting module.

[0112] The fourth prompt module is used to respond to the detection that the distance between the target robot hand and the virtual hand in the display terminal meets the preset conditions, keep the virtual hand and the target robot hand overlapping in the display terminal, and render and display the fourth prompt information in the display terminal.

[0113] The fifth prompt module is used to render and display a fifth prompt message on the display terminal in response to the detection that the distance between the target robot hand's position on the display terminal and the virtual hand's position on the display terminal does not meet the preset conditions.

[0114] Any one or more of the modules or units according to embodiments of the present invention, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules or units according to embodiments of the present invention can be implemented by dividing them into multiple modules. Any one or more of the modules or units according to embodiments of the present invention can be at least partially implemented as hardware circuitry, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, and firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules or units according to embodiments of the present invention can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0115] For example, any plurality of the display module 410, tissue deformation rate acquisition module 420, prompt information generation module 430, and prompt information display module 440 can be combined into one module / unit, or any one of these modules / units can be split into multiple modules / units. Alternatively, at least part of the functionality of one or more of these modules / units can be combined with at least part of the functionality of other modules / units and implemented in one module / unit. According to embodiments of the present invention, at least one of the display module 410, tissue deformation rate acquisition module 420, prompt information generation module 430, and prompt information display module 440 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the display module 410, the tissue deformation rate acquisition module 420, the prompt information generation module 430, and the prompt information display module 440 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0116] Figure 5 A block diagram of an electronic device suitable for implementing a humanoid robot-based soft tissue collaborative surgical control method is shown schematically according to an embodiment of the present invention. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0117] like Figure 5As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0118] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0119] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The system 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A driver 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the driver 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0120] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of the present invention. According to embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0122] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] For example, according to embodiments of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0124] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the soft tissue collaborative surgery control method based on a humanoid robot provided in the embodiments of the present invention.

[0125] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0127] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not expressly stated in the present invention. In particular, the features described in the various embodiments and / or claims of this invention can be combined and / or combined in various ways without departing from the spirit and teachings of this invention. All such combinations and / or combinations fall within the scope of this invention.

[0129] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for soft tissue collaborative surgery control based on a humanoid robot, comprising: The display terminal shows the real-time image stream information captured by the humanoid robot in the surgical operation area, wherein the surgical operation area includes the area where the first instrument held by the first robotic hand of the humanoid robot is located; In response to the detection of a grasping operation characterizing soft tissue grasped by the first instrument, the tissue deformation rate of the grasped soft tissue is obtained; Based on the tissue deformation rate of the captured soft tissue, a prompt message is generated to indicate the tissue stretching status on the display terminal; The prompt message is sent to the display terminal for display.

2. The method according to claim 1, wherein, The step of generating a prompt message for indicating the tissue stretching state on the display terminal based on the tissue deformation rate of the captured soft tissue includes: In response to determining that the tissue deformation rate is within a first deformation range, a first prompt message is generated, wherein the first deformation range indicates that the grasped soft tissue is in a safe stretching state; In response to determining that the tissue deformation rate is within a second deformation range, a second prompt message is generated, wherein the second deformation range indicates that the grasped soft tissue is in an optimal stretching state; In response to determining that the tissue deformation rate is within a third deformation range, a third prompt message is generated, wherein the third deformation range indicates that the grasped soft tissue is in an overstretched state.

3. The method according to claim 1, wherein, The prompt information includes halo information; sending the prompt information to the display terminal for display includes: Based on the image stream information, determine the position of the end of the first instrument in the image stream; The halo information is sent to the end of the device, and the image stream information is rendered to obtain a rendered video stream. The rendered video stream is sent to the display terminal for display.

4. The method according to claim 1, further comprising: In response to detecting that the prompt information is target information, a locking command is sent to the humanoid robot; Based on the locking command, locking operations are performed on each joint of the first robotic hand.

5. The method according to any one of claims 1 to 4, wherein, The humanoid robot further includes a second robotic hand, which holds a second instrument; the method further includes: A pre-constructed three-dimensional reconstruction model for the surgical procedure is integrated into the image stream information, wherein the three-dimensional reconstruction model is pre-set with an electronic fence to limit the surgical operation range; In response to detecting that the movement path of the second device reaches the edge of the electronic fence, the displacement component perpendicular to the electronic fence in the movement path is filtered out.

6. The method according to claim 5, further comprising: In response to detecting that the operator's handle has moved a first distance, the first distance is scaled according to a preset scaling ratio to obtain a second distance; The second distance is sent to the humanoid robot to control the target robot hand of the humanoid robot to move the second distance, wherein the posture of the operator handle is mapped to the posture of the target robot hand, and the target robot hand is at least one of the following: a first robot hand and a second robot hand.

7. The method according to claim 6, wherein, A virtual hand is also rendered and displayed on the display terminal, and the image stream information includes real-time motion information of the target robot hand. The virtual hand and the target robot hand have a corresponding relationship. The method further includes: In response to detecting that the distance between the position of the target robot hand in the display terminal and the position of the virtual hand in the display terminal meets a preset condition, the virtual hand and the target robot hand are kept overlapping in the display terminal, and a fourth prompt message is rendered and displayed on the display terminal; In response to the detection that the distance between the position of the target robot hand in the display terminal and the position of the virtual hand in the display terminal does not meet the preset condition, a fifth prompt message is rendered and displayed on the display terminal.

8. A soft tissue collaborative surgical control system based on a humanoid robot, comprising: An airborne core control unit for implementing the method of any one of claims 1 to 7. The main control unit includes a handle and a display terminal, wherein the handle is held by the operator; The humanoid robot end includes a joint motion controller and an image acquisition module, wherein the image acquisition module is used to acquire image stream information in real time for the surgical operation area; The display terminal is used to display image stream information acquired by the image acquisition module and processed by the airborne core control unit, and the joint motion controller is used to control the motion state of the target robot hand at the humanoid robot end based on the operator's action state on the handle after processing by the airborne core control unit.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.

Images