Surgical robot shared control method and system

Abstract

The present disclosure provides a surgical robot shared control method and system, which can be applied to the field of medical instrument control. The method comprises: in response to the task system collecting an execution statement of a task controller, inputting the execution statement into a large language model to obtain a simulation action sequence of a surgical robot; sending the simulation action sequence to a twin surgery system, so that a twin surgical robot in the twin surgery system performs a surgical task on a virtual surgical organ in a virtual surgery scene according to the simulation action sequence to obtain organ simulation state information; adjusting the control weight of the task system and the surgical robot according to the organ simulation state information; updating the simulation action sequence according to the control weight to obtain an updated simulation action sequence, which is used for the surgical robot to perform a surgical task on a surgical organ in a real surgery scene. The present disclosure promotes the intelligentization and human-computer cooperation of the surgical robot, and improves the accuracy, safety and efficiency of the surgery.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of medical instrument control, and particularly relates to a surgical robot shared control method and system. BACKGROUND

[0002] With the continuous development of artificial intelligence and robot technology, most existing organ surgeries are performed by surgeons who flexibly change surgical plans according to surgical conditions to achieve personalized surgery, or by surgeons who use robots for fine motion control to complete precise organ surgery.

[0003] However, due to too many uncertain factors in the surgical process, only surgical operation by surgeons has the problem of low surgical precision. But if a surgical robot is used for surgical operation, although the surgical precision is high, the surgical robot is relatively difficult to handle random unexpected situations during organ surgery, resulting in weak operation reliability of the surgical robot. SUMMARY

[0004] In view of the above problems, the present disclosure provides a surgical robot shared control method and system.

[0005] According to a first aspect of the present disclosure, a surgical robot shared control method is provided, comprising:

[0006] In response to the task system collecting an execution statement of a task controller, inputting the execution statement into a large language model to obtain a simulated action sequence of the surgical robot, the large language model being trained using execution statement samples and action sequence samples of the surgical robot;

[0007] sending the simulated action sequence to a twin surgery system, so that a twin surgical robot in the twin surgery system performs a surgical task on a virtual surgical organ under a virtual surgery scene according to the simulated action sequence to obtain organ simulation state information, the organ simulation state information representing organ biological state parameters obtained by the surgical robot performing a surgical task on a surgical organ under a real surgery scene according to the simulated action sequence;

[0008] adjusting a control weight of the task system and the surgical robot according to the organ simulation state information;

[0009] updating the simulated action sequence according to the control weight to obtain an updated simulated action sequence, the updated simulated action sequence being used for the surgical robot to perform a surgical task on a surgical organ under a real surgery scene.

[0010] According to an embodiment of the present disclosure, in response to the task system collecting the execution statement of the task controller, the execution statement is input into the large language model to obtain the simulation action sequence of the surgical robot, which includes: in response to the task system collecting the execution statement of the task controller, the execution statement is subjected to vectorization processing through an embedding layer of the large language model to obtain an execution vector; the execution vector is matched with an execution statement node in a surgical knowledge graph to obtain the simulation action sequence of the surgical robot, the surgical knowledge graph being obtained by inputting an execution statement sample, a surgical scene information sample and an action sequence sample into a reasoning engine, and the execution statement node representing the execution statement sample.

[0011] According to an embodiment of the present disclosure, the real surgical scene information includes surgical organ information, surgical robot part information, surgical instrument information and surgical method information, and the virtual surgical scene is obtained by modeling based on the real surgical scene information.

[0012] According to an embodiment of the present disclosure, the control weight of the task system and the surgical robot is adjusted according to the organ simulation state information, which includes: comparing the organ simulation state information with real execution state information of the surgical robot to obtain a comparison result, the real execution state information being obtained by the twin surgical robot according to a planned path to perform a surgical task, and the planned path being obtained by a control system of the surgical robot according to the execution statement to plan a path; and adjusting the control weight of the task system and the surgical robot according to the comparison result.

[0013] According to an embodiment of the present disclosure, the control weight of the task system and the surgical robot is adjusted according to the comparison result, which includes: adjusting the control weight of the task system and the surgical robot according to the comparison result and real-time task information of the surgical task.

[0014] According to an embodiment of the present disclosure, the control system of the surgical robot plans a path according to the execution statement by at least one of a feedforward control algorithm, a feedback control algorithm, an adaptive control algorithm and a sliding mode control algorithm to obtain a planned path corresponding to the execution statement.

[0015] According to an embodiment of the present disclosure, the simulation action sequence is updated according to the control weight to obtain an updated simulation action sequence, which includes: in a case where the control weight is greater than a first preset threshold, the simulation action sequence is updated according to an auxiliary statement of the task controller collected by the task system to obtain the updated simulation action sequence; and in a case where the control weight is less than a second preset threshold, the planned path is determined as the updated simulation action sequence.

[0016] According to an embodiment of the present disclosure, the above method further includes: in a case where a deviation between the simulation action sequence and the planned path satisfies a deviation threshold, generating a feedback instruction according to the organ simulation state information; and sending the feedback instruction to the task system.

[0017] The second aspect of the present disclosure provides a surgical robot shared control system, comprising: a task system configured to collect an execution statement of a task controller; a shared control system configured to input the execution statement into a large language model to obtain a simulated action sequence of a surgical robot, and send the simulated action sequence to a twin surgery system, wherein the large language model is trained using an execution statement sample and an action sequence sample of the surgical robot; the twin surgery system comprises a twin surgical robot corresponding to the surgical robot, and the twin surgical robot performs a surgical task on a virtual surgical organ in a virtual surgery scene according to the simulated action sequence to obtain organ simulation state information, wherein the organ simulation state information represents an organ biological state parameter obtained by the surgical robot performing the surgical task on the surgical organ in a real surgery scene according to the simulated action sequence; the shared control system is configured to adjust a control weight of the task system and the surgical robot according to the organ simulation state information, update the simulated action sequence according to the control weight to obtain an updated simulated action sequence, and use the updated simulated action sequence for the surgical robot to perform the surgical task in the real surgery scene.

[0018] According to an embodiment of the present disclosure, the surgical robot comprises a control system; and adjusting the control weight of the task system and the twin surgery system according to the organ simulation state information comprises: comparing the organ simulation state information with real execution state information of the surgical robot to obtain a comparison result, wherein the real execution state information is obtained by a twin surgical robot in the twin surgery system performing a surgical task according to a planned path, and the planned path is obtained by a control system of the surgical robot performing path planning according to the execution statement; and adjusting the control weight of the task system and the surgical robot according to the comparison result.

[0019] According to the surgical robot shared control method and system provided by the present disclosure, the execution statement of the task controller is input into a large language model to obtain a simulated action sequence of a surgical robot in response to the task system collecting the execution statement of the task controller, and the large language model is trained using an execution statement sample and an action sequence sample of the surgical robot. The execution statement of the task controller is parsed into the simulated action sequence of the surgical robot by the large language model.

[0020] The simulated action sequence is sent to a twin surgery system, so that a twin surgical robot in the twin surgery system performs a surgical task on a virtual surgical organ in a virtual surgery scene according to the simulated action sequence to obtain organ simulation state information, and the execution statement of the task controller is simulated.

[0021] According to the organ simulation state information, the control weight of the task system and the surgical robot is adjusted to ensure the smooth execution of the surgical task; the simulation action sequence is updated according to the control weight to obtain an updated simulation action sequence, the updated simulation action sequence is used for the surgical robot to execute the surgical task on the surgical organ in the real surgical scene, the collaborative cooperation between the task controller and the surgical robot is realized, the personalization of the surgical scheme is realized, and the operation reliability of the surgical robot is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 An application scenario diagram of a surgical robot shared control method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 2A A flowchart of a surgical robot shared control method according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 2B A schematic diagram of obtaining a simulation action sequence of a surgical robot according to an execution statement of a task controller according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 3 A schematic diagram of constructing a surgical knowledge graph according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 4 A schematic diagram of adjusting the control weight of a task system and a surgical robot according to a comparison result according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 5A An interaction schematic diagram between a task system and a twin surgical system according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 5B A schematic diagram of a virtual surgical scene and an updated ideal trajectory according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 5C A schematic diagram of updating a simulation action sequence according to a control weight according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 6 A schematic diagram of a surgical robot shared control system according to an embodiment of the present disclosure is schematically shown; and

[0032] Figure 7 A block diagram of an electronic device suitable for implementing a surgical robot shared control method according to an embodiment of the present disclosure is schematically shown. Detailed Implementation

[0033] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0037] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0038] In the scenario of making an automated decision by using personal information, the method, device and system provided by the embodiment of the present disclosure provide a corresponding operation portal for the user to select to agree or reject the automated decision result; if the user selects to reject, the expert decision process is entered. The expression "automated decision" herein refers to an activity of automatically analyzing and evaluating a person's behavior habit, interest and hobby, or economic, health and credit status, etc. by a computer program, and making a decision. The expression "expert decision" herein refers to an activity of making a decision by a person who is engaged in a certain field of work, has special experience, knowledge and skills, and reaches a certain professional level.

[0039] With the continuous development of artificial intelligence and robot technology, most of the existing organ surgeries are performed by doctors who flexibly change the surgical plan according to the surgical situation to realize personalized surgery, or by doctors who use robots to perform fine motion control to complete precise organ surgery.

[0040] However, due to too many uncertain factors in the surgical process, only the surgical operation of the doctor, there is a problem of low surgical precision. But if the surgical robot performs the surgical operation, although the surgical precision is high, the surgical robot is difficult to handle the random sudden situation in the organ surgery process, resulting in weak running reliability of the surgical robot.

[0041] Figure 1 An application scenario diagram of a surgical robot shared control method according to an embodiment of the present disclosure is schematically shown.

[0042] As shown in Figure 1 The application scenario 100 according to the embodiment can include a task system 101, a twin surgery system 102, a surgical robot 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the task system 101, the twin surgery system 102, the surgical robot 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] The user can use at least one of the task system 101 and the twin surgery system 102 to interact with the surgical robot 103 and the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the task system 101 and the twin surgery system 102.

[0044] The task system 101 and the twin surgery system 102 can be various electronic devices with a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer and a desktop computer, etc.

[0045] The task system 101 can be used to collect the execution statement of the task controller.

[0046] The surgical robot 103 can be used to perform a surgical task on a surgical organ.

[0047] The server 105 can be a shared control system configured to input an execution statement into a large language model to obtain a simulated action sequence of the surgical robot, and send the simulated action sequence to the twin surgery system, where the large language model is trained using execution statement samples and action sequence samples of the surgical robot.

[0048] The twin surgery system 102 includes a twin surgical robot corresponding to the surgical robot 103, and the twin surgical robot performs a surgical task on a virtual surgical organ in a virtual surgery scene according to the simulated action sequence to obtain organ simulation state information, which represents organ biological state parameters obtained by the surgical robot performing a surgical task on a surgical organ in a real surgery scene according to the simulated action sequence.

[0049] The shared control system is further configured to adjust a control weight of the task system 101 and the surgical robot 103 according to the organ simulation state information, update the simulated action sequence according to the control weight to obtain an updated simulated action sequence, and use the updated simulated action sequence for the surgical robot 103 to perform a surgical task in a real surgery scene.

[0050] It should be noted that the surgical robot shared control method provided by the embodiments of the present disclosure can generally be executed by the server 105 (shared control system). Correspondingly, the surgical robot shared control system provided by the embodiments of the present disclosure can generally be arranged in the server 105. The surgical robot shared control method provided by the embodiments of the present disclosure can also be executed by a server or server cluster different from the server 105 and capable of communicating with the task system 101, the twin surgery system 102, the surgical robot 103, and / or the server 105. Correspondingly, the surgical robot shared control system provided by the embodiments of the present disclosure can also be arranged in a server or server cluster different from the server 105 and capable of communicating with the task system 101, the twin surgery system 102, the surgical robot 103, and / or the server 105.

[0051] It should be understood that Figure 1 The number of task systems, twin surgery systems, surgical robots, networks, and servers in the above description is only illustrative. According to the needs of implementation, there can be any number of task systems, twin surgery systems, surgical robots, networks, and servers.

[0052] Therefore, the embodiments of the present disclosure provide a surgical robot shared control method.

[0053] Figure 2A A flowchart of a surgical robot shared control method according to an embodiment of the present disclosure is schematically shown.

[0054] As Figure 2A shown, the surgical robot shared control method of this embodiment includes operation S210 to operation S240.

[0055] In operation S210, in response to the task system collecting the execution statement of the task controller, the execution statement is input into the large language model to obtain a simulation action sequence of the surgical robot.

[0056] According to an embodiment of the present disclosure, before the task system collects the execution statement of the task controller, the consent or authorization of the task controller can be obtained. For example, before operation S210, a request for collecting the execution statement of the task controller can be sent to the task controller. In the case where the task controller agrees or authorizes the collection of the execution statement of the task controller, operation S210 is performed.

[0057] According to an embodiment of the present disclosure, the task system is also used for training of the large language model.

[0058] According to an embodiment of the present disclosure, the voice collection device of the task system collects the voice information of the task controller and analyzes the voice information to obtain the execution statement.

[0059] According to an embodiment of the present disclosure, in response to the input operation of the task controller, the input text is obtained and the input text is subjected to semantic analysis to obtain the execution statement. For example, the execution statement can be “suture the liver” or “cut the liver”, etc.

[0060] According to an embodiment of the present disclosure, generally, the surgical robot can plan a path according to the spatial positions of the start and end preset according to the surgical task, determine the operation steps of the surgery, and complete the surgical task.

[0061] According to an embodiment of the present disclosure, the large language model is trained by using the execution statement sample and the action sequence sample of the surgical robot.

[0062] According to an embodiment of the present disclosure, the simulation action sequence can be composed of multiple actions of the surgical robot arranged based on a time sequence relationship. The time sequence relationship can be the temporal relationship. The action of the surgical robot can be moving, grabbing, placing, etc.

[0063] For example, the simulation action sequence of the mechanical hand of the surgical robot can be {“move 5mm”, “lower 1mm”, “grab”}. In the time relationship dimension, the mechanical hand first moves 5mm, then lowers 1mm, and then performs “grabbing”.

[0064] Figure 2B A schematic diagram of obtaining a simulation action sequence of a surgical robot according to an execution statement of a task controller according to an embodiment of the present disclosure is schematically shown.

[0065] As shown in Figure 2B the execution statement of the task controller is input into the large language model, and the large language model generates a simulated action sequence.

[0066] In operation S220, the simulated action sequence is sent to the twin surgery system, so that the twin surgery robot in the twin surgery system performs a surgical task on a virtual surgical organ according to the simulated action sequence in a virtual surgical scene, and obtains organ simulation state information.

[0067] According to an embodiment of the present disclosure, the twin surgery system can be based on digital twin technology, and a digital model of a real surgery scene is updated synchronously with an actual system.

[0068] According to an embodiment of the present disclosure, the organ simulation state information represents an organ biological state parameter obtained by the surgical robot performing a surgical task on the surgical organ according to the simulated action sequence in the real surgery scene. For example, the organ simulation state information can be a liver state, a liver color, a liver bleeding amount, a liver weight, etc.

[0069] In operation S230, the control weight of the task system and the surgical robot is adjusted according to the organ simulation state information.

[0070] For example, the organ simulation state information can be the bleeding amount of the liver. When the bleeding amount of the liver exceeds the bleeding threshold, i.e., the surgical robot does not achieve the preset organ biological state parameter by performing a surgical task on the surgical organ according to the simulated action sequence in the real surgery scene, the control weight of the task system needs to be increased to make the surgical robot reliably perform the surgical task. Conversely, when the bleeding amount of the liver is lower than the bleeding threshold, the control weight of the surgical robot is increased.

[0071] In operation S240, the simulated action sequence is updated according to the control weight, and an updated simulated action sequence is obtained.

[0072] According to an embodiment of the present disclosure, the updated simulated action sequence is used for the surgical robot to perform a surgical task on the surgical organ in the real surgery scene.

[0073] According to an embodiment of the present disclosure, taking the amount of bleeding of the liver as an example, when the amount of bleeding of the liver is higher than the bleeding threshold, the control weight of the task system is increased, and the simulation action sequence is updated for the first time, the twin surgery robot performs a surgical task on the virtual surgical organ according to the simulation action sequence updated for the first time, and obtains the amount of bleeding of the liver as a first amount of bleeding; if the first amount of bleeding is still higher than the bleeding threshold, the simulation action sequence is updated for the second time, the twin surgery robot performs a surgical task on the virtual surgical organ according to the simulation action sequence updated for the second time, and obtains the amount of bleeding of the liver as a second amount of bleeding; the simulation action sequence is iteratively verified in the virtual surgical scene according to the amount of bleeding of the liver, until the amount of bleeding of the liver is lower than the bleeding threshold, and the simulation action sequence updated for the Nth time is obtained, at this time, the simulation action sequence for the Nth time is directly used to perform a surgical task on the surgical organ in a real surgical scene.

[0074] For example, the doctor issues an execution statement of "stitching the liver", the large language model decomposes the execution statement of "stitching the liver" into a plurality of execution actions and an execution order simulation action sequence that can be executed by the twin surgery robot, the twin surgery robot performs a stitching task on the virtual liver in the virtual surgical scene according to the simulation action sequence, and can obtain organ simulation state information such as liver state and amount of bleeding, according to the simulation state information of the liver organ, the control weights of the doctor and the surgical robot are adjusted to intuitively optimize and fine-tune the action sequence of the surgical robot, to obtain an updated simulation action sequence, and the updated simulation action sequence is used for the surgical robot to perform a surgical task on the surgical organ in a real surgical scene.

[0075] According to an embodiment of the present disclosure, in response to the task system collecting an execution statement of a task controller, the execution statement is input into a large language model to obtain a simulation action sequence of a surgical robot, the large language model being trained by using an execution statement sample and an action sequence sample of the surgical robot. The execution statement of the task controller is parsed into the simulation action sequence of the surgical robot by the large language model.

[0076] The simulation action sequence is sent to a twin surgery system, so that a twin surgery robot in the twin surgery system performs a surgical task on a virtual surgical organ according to the simulation action sequence in a virtual surgical scene, to obtain organ simulation state information, and to realize simulation operation on the execution statement of the task controller.

[0077] According to the organ simulation state information, the control weight of the task system and the surgical robot is adjusted to ensure the smooth execution of the surgical task; the simulation action sequence is updated according to the control weight to obtain an updated simulation action sequence, and the updated simulation action sequence is used for the surgical robot to execute the surgical task on the surgical organ in a real surgical scene, realizing the cooperation between the task controller and the surgical robot, realizing the personalization of the surgical scheme, and improving the operation reliability of the surgical robot.

[0078] According to an embodiment of the present disclosure, through fine motion control of the surgical robot, surgical errors can be significantly reduced, and operation accuracy can be improved; operation safety is enhanced: with the help of simulation optimization of the twin reality environment, operation risk can be greatly reduced; personalized operation is realized: relying on natural language interaction of the large language model, the task controller can flexibly instruct the surgical robot to realize a personalized operation scheme; operation efficiency is improved: the advantages of man and machine are complementary, the surgical robot is precise and efficient, the task controller controls the overall situation, and the operation time is shortened.

[0079] According to an embodiment of the present disclosure, in response to the task system collecting an execution statement of the task controller, the execution statement is input into the large language model to obtain a simulation action sequence of the surgical robot, which includes:

[0080] In response to the task system collecting an execution statement of the task controller, the execution statement is vectorized by the embedding layer of the large language model to obtain an execution vector;

[0081] The execution vector is matched with an execution statement node in the surgical knowledge graph to obtain a simulation action sequence of the surgical robot, and the surgical knowledge graph is obtained by inputting execution statement samples, surgical scene information samples and action sequence samples into a reasoning engine, and the execution statement node represents the execution statement sample.

[0082] According to an embodiment of the present disclosure, the execution statement sample and the action sequence sample can be obtained from a surgical medical field corpus. The execution statement sample and the action sequence sample are used to train the large language model.

[0083] According to an embodiment of the present disclosure, human-machine cooperation can be promoted: through digital twin technology and knowledge graph support, seamless cooperation between the task controller and the robot is realized.

[0084] According to an embodiment of the present disclosure, the embedding layer of the large language model can perform semantic analysis on the execution statement, for example, extracting keywords in the execution statement, embedding different vectors in the execution statement to analyze semantics.

[0085] According to an embodiment of the present disclosure, the surgical scene information sample can include a surgical task type, a surgical method of the surgical task, a surgical instrument, a surgical organ, etc.

[0086] According to an embodiment of the present disclosure, the inference engine can infer and deduce new conclusions or knowledge from the input execution sentence samples, action sequence samples, and surgical scene information samples and the like data by using pre-defined rules or patterns, so as to obtain some hidden nodes contained in the surgical knowledge graph, thereby continuously expanding and improving the surgical knowledge graph.

[0087] The large language model can first be fine-tuned by using the execution sentence samples and action sequence samples obtained from the surgical medical field corpus of surgical organs and the like to enhance its understanding ability of surgical terms and step descriptions, and then the constructed surgical knowledge graph is integrated into the fine-tuned large language model to realize precise mapping between the execution sentence of the task controller and the concept entity of the surgical knowledge graph. Not only can it receive the execution sentence of the task controller and parse the executable action sequence of the surgical robot, but also can feed back the running state of the surgical robot to the task controller in the form of natural language.

[0088] Figure 3 An illustrative diagram of constructing a surgical knowledge graph according to an embodiment of the present disclosure is shown.

[0089] As shown in Figure 3 After the execution sentence samples, surgical scene information samples, and action sequence samples and the like surgical related information are preprocessed such as information cleaning and standardized annotation, a unified medical knowledge dataset is formed, a data-associated surgical knowledge graph is constructed based on the medical knowledge dataset, the surgical knowledge graph and the field expert experience rules are input into the inference engine, so as to obtain the hidden nodes of the surgical knowledge graph, realize the expansion and improvement of the surgical knowledge graph, and finally integrate the surgical knowledge graph into the large language model to realize precise mapping between natural language and the concept entity of the surgical knowledge graph, provide knowledge support for semantic understanding and action generation.

[0090] For example, the execution sentence sample can be action information such as the action "suction". The surgical scene information sample can be collected from medical literature, clinical guidelines or databases, etc., such as structured information of surgical instruments, unstructured information of surgical organs, surgical method information, etc. The inference engine expands the breadth and depth of the surgical knowledge graph, and provides knowledge support for the large language model and the surgical robot.

[0091] According to an embodiment of the present disclosure, by vectorizing the execution sentence by the large language model and matching it with the execution sentence node in the surgical knowledge graph, the corresponding simulated action sequence can be determined, and it is ensured that the generated simulated action sequence is highly consistent with the intention of the task controller, improving the accuracy of the simulated action sequence and improving the response speed and efficiency.

[0092] According to an embodiment of the present disclosure, the real surgery scene information includes a surgery organ, part information of a surgery robot, surgery instrument information, and surgery mode information, and the virtual surgery scene is obtained by modeling based on the real surgery scene information.

[0093] The surgery organ can be an organ such as a liver, an eye, or a lower leg.

[0094] The part information of the surgery robot can be a mechanical hand or a mechanical wrist joint of the surgery robot.

[0095] The surgery instrument information can be a scalpel, surgical scissors, or a blood vessel clamp.

[0096] The surgery mode information can be open surgery, laparoscopic surgery, or endoscopic surgery.

[0097] The virtual surgery scene can be a virtual surgery scene highly close to a real surgery scene, which is constructed by using computer graphics and virtual reality technology, and real-time dynamic change information of a real surgery scene such as surgery instrument movement and tissue deformation is captured by fusion of various sensors such as a laparoscope and a nuclear magnetic instrument, the change of the real surgery scene is synchronized in real time, and the virtual surgery scene is updated to establish a fine virtual surgery scene.

[0098] For example, the surgery organ is modeled according to three-dimensional information and internal structure information of the surgery organ, and fine structure information of the surgery organ can be displayed, which facilitates subsequent execution of a surgery task.

[0099] For example, the surgery robot is modeled according to three-dimensional information, relative position information, and connection relationship of each joint of the surgery robot, and a twin surgery robot is obtained, and movement of the twin surgery robot executing a simulation action sequence can be clearly displayed.

[0100] For example, a surgery instrument is modeled according to surgery instrument information, and a twin surgery instrument is obtained, so that the twin surgery instrument is selected according to a surgery task to realize simulation close to a real surgery scene.

[0101] According to an embodiment of the present disclosure, the virtual surgery scene can be modeled according to relative positions between objects in a real surgery scene, so that a twin surgery robot can accurately execute a surgery task on a virtual surgery organ according to a simulation action sequence. For example, the simulation action sequence is obtained by modeling based on relative positions between objects such as a surgery bed and a bandage to realize object avoidance, and safety of a surgery task is improved.

[0102] According to an embodiment of the present disclosure, a twin surgery robot executes a task in a virtual surgery scene according to an execution statement of a task controller, the task controller observes and optimizes running effects of an action sequence of the twin surgery robot, and the action sequence is interactively optimized and fine-tuned to generate a final simulation action sequence, which is applied to a real surgery scene.

[0103] According to the embodiment of the present disclosure, by constructing a virtual surgery scene close to the real surgery scene, the task controller can reduce the risk in the real surgery scene and ensure the safety of the patient by simulating the surgery process in the virtual surgery scene.

[0104] According to the embodiment of the present disclosure, adjusting the control weights of the task system and the twin surgery system according to the organ simulation state information includes:

[0105] Comparing the organ simulation state information with the real execution state information of the surgery robot to obtain a comparison result, the real execution state information being obtained by the twin surgery robot according to a planned path, the planned path being obtained by a control system of the surgery robot according to an execution statement;

[0106] Adjusting the control weights of the task system and the surgery robot according to the comparison result.

[0107] According to the embodiment of the present disclosure, the planned path can be a desired motion trajectory or action sequence obtained by a path planning algorithm based on the execution statement, which can guide the surgery robot to complete the surgery task.

[0108] The path planning algorithm can be an optimization algorithm such as gradient descent, dynamic programming, genetic algorithm, or a machine learning algorithm such as reinforcement learning and deep learning, to achieve intelligent planning. For example, the planned path can be the shortest path that meets the obstacle avoidance condition, the path with the minimum time consumption, etc.

[0109] Since the virtual surgery scene is modeled based on real surgery scene information, the twin surgery robot can accurately execute the surgery task according to the planned path in the virtual surgery scene, obtain the real execution state information of the surgery, improve the update accuracy of the simulation action sequence, and thus improve the reliability of the surgery robot.

[0110] According to the embodiment of the present disclosure, according to the comparison result of the organ simulation state information and the real execution state information of the surgery robot, possible deviations or errors in the virtual surgery process can be found in time, and the control weights can be adjusted according to the comparison result, so as to ensure that the surgery robot executes the surgery task according to a more accurate and safer path, reduces the surgery risk, improves the surgery success rate, and realizes seamless switching of human-machine cooperation.

[0111] According to the embodiment of the present disclosure, adjusting the control weights of the task system and the surgery robot according to the comparison result includes adjusting the control weights of the task system and the surgery robot according to the comparison result and real-time task information of the surgery task.

[0112] According to an embodiment of the present disclosure, the real-time task information of the surgical task can be a surgical task feature, a stage progress, or an environment complexity, etc. The surgical task feature can be a characteristic of a surgical manner. The stage progress can be a stage of the execution of the surgical task. The environment complexity can be a surgical humidity, a temperature, a sterilization condition, etc.

[0113] Figure 4 A schematic diagram of adjusting the control weight of the task system and the surgical robot according to the comparison result is schematically shown according to an embodiment of the present disclosure.

[0114] As Figure 4 shown, the real execution state information of the surgical robot is synchronized to the virtual surgery scene in real time, and the comparison result is obtained by comparing with the current organ simulation state information. For example, if there is a difference between the organ simulation state information and the real execution state information of the surgical robot, the control weight a(t) of the task system is increased, based on the state difference, combined with the surgical knowledge graph and the large language model, the instruction is automatically generated and fed back to the task controller, the task controller optimizes and adjusts the simulation action sequence in the virtual surgery scene according to the instruction, until the updated simulation action sequence can accurately meet the real expected surgical effect, and the updated simulation action sequence is directly used in the real surgery scene. For example, the real-time task information of the surgical task can be the stage progress. For example, the surgical task has a high requirement on the stage progress, and needs to be completed within a fixed time period, so the control weight 1-a(t) of the surgical robot can be increased. For example, the sterilization state of the surgical task only requires to be maintained for three hours, so the control weight a(t) of the task system and the control weight 1-a(t) of the surgical robot need to be reasonably adjusted in real time.

[0115] If there is no deviation between the organ simulation state information and the real execution state information of the surgical robot, the control weight 1-a(t) of the surgical robot is increased, and the surgical robot executes the simulation action sequence on the surgical organ.

[0116] Meanwhile, the control system of the surgical robot can have a parameter setting of confidence, which is used to represent the degree of trust of the task controller to the surgical robot. The control weight 1-a(t) of the surgical robot can be adjusted by referring to the confidence.

[0117] The confidence of the control system of the surgical robot can be determined according to the soft tissue state information (the soft tissue state information can be one of the organ simulation state information) and other information. For example, according to the soft tissue state information of the organ obtained after the surgical robot executes the simulation action sequence on the surgical organ, the confidence is generated, and based on the confidence, the control system will configure the motion parameters such as the pose change matrix, the acceleration, the contact force and the speed of the surgical robot, to realize the update of the simulation action sequence.

[0118] According to an embodiment of the present disclosure, the weight coefficients of the task system and the surgical robot in control are adjusted in real time according to the characteristics of the surgical task, the progress of the stage, or the complexity of the environment, and the roles and participation of the task system and the surgical robot are flexibly adjusted to realize seamless switching of human-machine cooperation, ensure smooth progress of the surgery, and ensure closer cooperation between humans and machines, realize complementary advantages, improve the adaptability, success rate, and safety of the surgery, and provide more accurate and effective treatment for patients.

[0119] According to an embodiment of the present disclosure, the control system of the surgical robot is path planning according to the execution statement by at least one of a feedforward control algorithm, a feedback control algorithm, an adaptive control algorithm, and a sliding mode control algorithm, to obtain a planning path corresponding to the execution statement.

[0120] The feedforward control algorithm can be a control strategy for adjusting the control output of the control system of the surgical robot based on the expected change of the input or disturbance before the surgical task starts. For example, the feedforward control algorithm generates a desired trajectory according to the surgical plan.

[0121] However, the feedforward control algorithm is usually affected by disturbances of the surgical robot, external noise, etc., and cannot directly obtain the desired trajectory. A feedback control algorithm is usually used to assist in adjusting the initial planning path obtained by the feedforward control algorithm.

[0122] The feedback control algorithm can be a control strategy for adjusting the input of the control system based on the deviation between the control output of the control system of the surgical robot and the desired output. By continuously monitoring the output of the control system and comparing it with the output of the desired trajectory, the control signal is adjusted to reduce the deviation. For example, the feedback control algorithm detects external disturbances in real time and compensates for them.

[0123] Considering unexpected situations during the execution of the surgical task, for example, an abnormal protrusion appears on the surface of the surgical organ, an adaptive control algorithm can be used to adjust the planning path in real time according to the changes in the external environment.

[0124] The adaptive control algorithm can be a control strategy that can adjust the control parameters of the control system in real time according to the changes in the input, output, and external environment of the control system of the surgical robot to achieve the optimal performance of the control system. For example, the adaptive control algorithm can identify and adaptively adjust the parameter changes of the control system online.

[0125] The sliding mode control algorithm can be a control strategy that constantly changes the current state of the control system of the surgical robot with a purpose, forcing the control system to move according to the predetermined "sliding mode" state trajectory. For example, the control system state is forced to converge to the desired trajectory by switching the control law.

[0126] According to an embodiment of the present disclosure, the algorithm of the control system of the surgical robot can also be a robust optimization control algorithm. The robust optimization control algorithm can be a method for processing uncertainty factors in an optimization problem, and seeks a solution that is not sensitive to parameter perturbations in an uncertain environment.

[0127] According to an embodiment of the present disclosure, through the fusion of various sensors such as laparoscopes and nuclear magnetic devices, the dynamic change information of the real surgical scene such as the movement of the surgical instrument and the deformation of the tissue captured in real time can be used as feedback to guide the control algorithm to correct the control system of the surgical robot, so as to ensure that the surgical robot can accurately and stably execute the optimized action sequence.

[0128] According to an embodiment of the present disclosure, through the path planning according to the execution statement by the control algorithm, the external interference and system uncertainty are detected and compensated in real time, so as to ensure the accuracy, stability and reliability of the surgical robot in the real surgical scene.

[0129] According to an embodiment of the present disclosure, the simulated action sequence is updated according to the control weight, and the updated simulated action sequence comprises:

[0130] In the case where the control weight is greater than the first preset threshold, the simulated action sequence is updated according to the auxiliary statement of the task controller collected by the task system, and the updated simulated action sequence is obtained;

[0131] In the case where the control weight is less than the second preset threshold, the planning path is determined as the updated simulated action sequence.

[0132] The greater the control weight is, the greater the weight coefficient of the task controller in the control weight is.

[0133] According to an embodiment of the present disclosure, the control weight can be a control coefficient of the task system. For example, the first preset threshold can be 95%, if the control coefficient of the task system is greater than 95%, the decision of the task controller is dominant, and the Nth updated simulated action sequence is obtained through the auxiliary statement of the task controller. On the contrary, the second preset threshold can be 5%, if the control coefficient of the task system is less than 5%, the planning path of the surgical robot is dominant, and the planning path is determined as the updated simulated action sequence.

[0134] The auxiliary statement can be a statement that can optimize the action sequence. For example, it can be "reduce the suture interval", "increase the knot strength", etc.

[0135] According to the embodiment of the present disclosure, when the task system is dominant, the task controller can intuitively fine-tune the action sequence of the surgical robot in the virtual surgery scene through execution statements such as "reduce suture interval" and "increase knot strength". The large language model and the surgery knowledge graph will update the simulated action sequence according to the execution statement and iteratively verify it in the virtual surgery scene. When the surgical robot is dominant, the updated simulated action sequence will be directly executed.

[0136] According to the embodiment of the present disclosure, when the control weight is greater than the first preset threshold, the simulated action sequence is updated by collecting the auxiliary statement of the task controller, and the simulated action sequence is optimized and iterated to accurately meet the real expected surgery effect; when the control weight is less than the second preset threshold, the surgical robot executes according to the updated simulated action sequence as the planning path, without human intervention, ensuring the stability and reliability of the simulated action sequence, thereby improving the efficiency and quality of task completion. According to the different control weights, the way of updating the simulated action sequence is flexibly selected, and the safety and efficiency of the task system are improved.

[0137] According to the embodiment of the present disclosure, the above method further comprises:

[0138] In the case that the deviation between the simulated action sequence and the planning path meets the deviation threshold, a feedback instruction is generated according to the organ simulation state information;

[0139] The feedback instruction is sent to the task system.

[0140] According to the embodiment of the present disclosure, the feedback instruction can be an instruction for adjusting or correcting the simulated action sequence. For example, an instruction for changing the action speed, adjusting the action direction, etc.

[0141] According to the embodiment of the present disclosure, in the case that the deviation between the simulated action sequence and the planning path meets the deviation threshold, the organ simulation state information that can be obtained represents the organ biological state parameter that exceeds the preset parameter, and the feedback instruction needs to be generated in time to remind the task controller to assist the surgery task. For example, the feedback instruction can be a warning information, such as "high bleeding volume, increase knot strength".

[0142] After the task system receives the feedback instruction, the task controller can refer to the warning information and issue the auxiliary statement of the task controller.

[0143] According to the organ simulation state information, the control weight of the task system can be increased, so as to update the simulated action sequence according to the adjusted control weight and the auxiliary statement, and obtain the updated simulated action sequence.

[0144] According to an embodiment of the present disclosure, when there is a deviation between the simulated action sequence and the planned path, based on the organ simulation state information, combined with the knowledge graph and the large language model, feedback instructions such as "increase suture tension" or "avoid protruding tissue" are automatically generated to the task system. The task controller optimizes and adjusts the action sequence in the virtual surgery scene according to the feedback instructions until it can accurately conform to the real expected surgery effect. Through the closed-loop process, the real-time feedback of the real surgery scene can continuously guide and improve the action optimization sequence in the virtual surgery scene, so that it constantly approaches the expected surgery effect, ensuring the high quality of the action sequence and improving the precision and stability of the surgery operation.

[0145] According to an embodiment of the present disclosure, through the combination of the large language model, the surgery knowledge graph and the twin surgery system, the execution statements of the task controller can be efficiently, accurately and intelligently processed, and the simulated action sequence of the surgery robot that is highly consistent with the intention of the task controller can be generated, promoting the medical surgery robot system to realize true intelligence and human-machine cooperation, fully utilizing the advantages of both parties in experience judgment and precise control, thereby improving the accuracy, efficiency and safety of the surgery process and providing patients with high-quality and personalized intelligent surgery experience.

[0146] Figure 5A An interaction schematic diagram between the task system and the twin surgery system according to an embodiment of the present disclosure is schematically shown.

[0147] As shown in Figure 5A , the execution statement collected by the task system is input into the large language model to obtain the simulated action sequence of the surgery robot. The twin surgery system performs a preview according to the simulated action sequence to obtain organ simulation state information that can represent the execution of the surgery task by the surgery robot in the real surgery scene. The organ simulation state information is fed back to the task system. The task controller can issue an auxiliary statement according to the organ simulation state information to update the simulated action sequence. At the same time, the control weight of the task system and the surgery robot is adjusted according to the organ simulation state information, so as to output the updated simulated action sequence.

[0148] The task controller will optimize and adjust the simulated action sequence in the virtual surgery scene until the optimized and adjusted simulated action sequence can accurately conform to the real expected surgery effect. The large language model will map the simulated action sequence that conforms to the real expected surgery effect into a simulated action sequence executable by the surgery robot and update it to the real surgery scene, forming a closed-loop human-machine optimization cycle, and finally generating a high-quality action sequence that meets the surgery demand. The surgery process is carried out through human-machine cooperation, interaction between the virtual surgery scene and the real surgery scene, and the process cycle of closed-loop optimization until the surgery is completed.

[0149] Figure 5BA schematic diagram of a virtual surgery scene and an updated ideal trajectory according to an embodiment of the present disclosure is shown schematically.

[0150] As shown in Figure 5B The spatial position relationship between each object in the virtual surgery scene is displayed.

[0151] The large language model can parse the execution statement of the task controller into the simulation action sequence of the surgical robot according to the semantic mapping between the natural language and the surgery knowledge graph, while supporting the fusion of multi-modal input. During the execution of the surgery, the large language model can feed back the running state of the surgical robot to the task controller to realize closed-loop interactive optimization, and can automatically supplement the surgery knowledge graph to continuously improve the semantic understanding ability of the large language model.

[0152] The twin surgery system can pre-play the updated simulation action sequence to obtain the updated ideal trajectory, thereby ensuring the safety of the surgical robot in performing the surgical task according to the updated simulation action sequence in the real surgery scene.

[0153] Figure 5C A schematic diagram of updating the simulation action sequence according to the control weight according to an embodiment of the present disclosure is shown schematically.

[0154] As shown in Figure 5C First, the construction of the surgery knowledge graph is performed, wherein the surgery knowledge graph can provide knowledge support for the semantic understanding of the large language model, and provide guidance for the actions of the virtual surgery scene, which is simulated and constructed according to the real surgery scene information. The twin surgery system can realize the pre-play of the twin surgical robot in the virtual surgery scene.

[0155] According to an embodiment of the present disclosure, the large language model can parse the execution statement of the task controller to obtain semantic information of the execution statement; the semantic information is vectorized to obtain an execution vector. The execution vector is matched with the execution statement node in the surgery knowledge graph to realize semantic mapping, thereby obtaining the simulation action sequence of the surgical robot.

[0156] The large language model can realize semantic analysis of various forms of execution statements such as voice information, text information, and nuclear magnetic organ images. For example, the multi-modal execution statement can be voice information collected from the task controller, input text information, and nuclear magnetic organ images input by the task controller. The large language model performs semantic analysis and semantic mapping on the multi-modal execution statement to obtain the final execution statement of the task controller.

[0157] Then, the execution statement of the task controller is input into the large language model to obtain the simulation action sequence of the surgical robot, and the simulation action sequence is sent to the twin surgery system. As shown inFigure 5C The simulation action sequence is input into the twin surgery system, so that the twin surgery robot performs a surgical task on a virtual surgical organ according to the simulation action sequence in a virtual surgical scene, thereby outputting organ simulation state information. The organ simulation state information is input into the large language model to obtain a feedback instruction. The task system receives the feedback instruction and displays the feedback instruction to the task controller, thereby achieving state feedback. The task controller can refer to the feedback instruction and issue an auxiliary statement. The simulation action sequence is optimized (updated) according to the auxiliary statement to obtain an updated simulation action sequence.

[0158] According to the embodiments of the present disclosure, the control weights of the task system and the surgical robot are adjusted according to the organ simulation state information, thereby achieving precise updating of the simulation action sequence.

[0159] In a real surgical scene, the control system of the surgical robot can optimize the action sequence according to a robust control algorithm such as a feedforward control algorithm, a feedback control algorithm, an adaptive control algorithm, a sliding mode control algorithm, and a robust optimization control algorithm. Meanwhile, the twin surgery robot also has a robust control algorithm such as a feedforward control algorithm, a feedback control algorithm, an adaptive control algorithm, a sliding mode control algorithm, and a robust optimization control algorithm, thereby achieving precise updating of the simulation action sequence.

[0160] Through fusion of various sensors such as laparoscopes and nuclear magnetic devices, dynamic change information of a real surgical scene such as surgical instrument movement and tissue deformation can be captured in real time, which can also be fed back to the twin surgery system and compared with the current simulation action sequence in a virtual surgical scene, and a feedback instruction is generated again based on the difference in comparison to the task system. The task controller can issue an auxiliary statement according to the feedback instruction to optimize and adjust the action sequence in a virtual surgical scene until the updated simulation action sequence can accurately meet the expected surgical effect in reality, thereby forming a closed-loop process and finally generating a high-quality surgical action sequence that meets the surgical needs.

[0161] For example, in the case of being dominated by the surgical robot, if the organ biological state parameter represented by the organ simulation state information exceeds the preset biological state threshold, the task controller can be directly switched to dominate. For example, in the case of being dominated by the task controller, the environment of the surgery changes, and the surgery time needs to be shortened, the surgical robot can be directly switched to dominate.

[0162] For example, the action sequence of the surgical robot includes two action sequences that need to be updated, such as “grasping” and “suture”. After the “grasping” action sequence is updated, the action sequence of the surgical robot needs to be reconstructed again, and the “suture” action sequence is updated based on the reconstructed action sequence of the surgical robot.

[0163] Based on the above surgical robot shared control method, the disclosure also provides a surgical robot shared control system. In the following, the surgical robot shared control system according to the embodiment of the disclosure will be described in detail. Figure 6 The system is described in detail.

[0164] Figure 6 The schematic diagram of the surgical robot shared control system according to the embodiment of the disclosure is schematically shown.

[0165] As Figure 6 shown, the surgical robot shared control system 600 of this embodiment includes a task system 610, a shared control system 620, and a twin surgery system 630.

[0166] The task system 610 is used to collect the execution statements of the task controller.

[0167] The shared control system 620 is used to input the execution statements into a large language model to obtain a simulated action sequence of the surgical robot, and send the simulated action sequence to the twin surgery system, wherein the large language model is trained by using execution statement samples and action sequence samples of the surgical robot.

[0168] The twin surgery system 630 includes a twin surgery robot corresponding to the surgical robot, and the twin surgery robot performs a surgical task on a virtual surgical organ in a virtual surgical scene according to the simulated action sequence to obtain organ simulation state information, wherein the organ simulation state information represents organ biological state parameters obtained by the surgical robot performing a surgical task on a surgical organ in a real surgical scene according to the simulated action sequence.

[0169] The shared control system 620 is used to adjust the control weight of the task system and the surgical robot according to the organ simulation state information, update the simulated action sequence according to the control weight to obtain an updated simulated action sequence, and use the updated simulated action sequence for the surgical robot to perform a surgical task in a real surgical scene.

[0170] According to the embodiment of the disclosure, the surgical robot includes a control system;

[0171] Adjusting the control weight of the task system and the twin surgery system according to the organ simulation state information includes:

[0172] Comparing the organ simulation state information with real execution state information of the surgical robot to obtain a comparison result, wherein the real execution state information is obtained by the twin surgery robot in the twin surgery system performing a surgical task according to a planned path, and the planned path is obtained by the control system of the surgical robot performing path planning according to the execution statement;

[0173] Adjusting the control weight of the task system and the surgical robot according to the comparison result.

[0174] It should be noted that the surgical robot shared control system part in the embodiments of the present disclosure corresponds to the surgical robot shared control method part in the embodiments of the present disclosure, and the description of the surgical robot shared control system part is specifically referred to the surgical robot shared control method part, which will not be repeated here.

[0175] According to embodiments of the present disclosure, any of the task system 610, the shared control system 620 and the twin surgery system 630 can be combined in one system, or any of them can be split into multiple systems. Alternatively, at least part of the function of one or more of these systems can be combined with at least part of the function of other systems, and implemented in one system. According to embodiments of the present disclosure, at least one of the task system 610, the shared control system 620 and the twin surgery system 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware and firmware implementation or in a suitable combination of any of them. Alternatively, at least one of the task system 610, the shared control system 620 and the twin surgery system 630 can be at least partially implemented as a computer program module which can perform corresponding functions when it is run.

[0176] Figure 7 The block diagram of an electronic device suitable for implementing the surgical robot shared control method according to embodiments of the present disclosure is schematically shown.

[0177] As shown in Figure 7 The electronic device 700 according to embodiments of the present disclosure includes a processor 701 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0178] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via the bus 704. The processor 701 performs various operations of the method processes according to the embodiments of the disclosure by executing the programs in the ROM 702 and / or the RAM 703. It is noted that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method processes according to the embodiments of the disclosure by executing the programs stored in the one or more memories.

[0179] According to an embodiment of the disclosure, the electronic device 700 can further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 can further include one or more of the following components connected to the I / O interface 705: an input part 706 including a keyboard, a mouse, etc.; an output part 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 708 including a hard disk, etc.; and a communication part 709 including a network interface card such as a LAN card, a modem, etc. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read therefrom is installed in the storage part 708 as necessary.

[0180] According to an embodiment of the disclosure, the program code for executing the computer programs provided by the embodiments of the disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. The programming language includes, but is not limited to, a programming language such as Java, C++, python, "C" language, or a similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0183] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A shared control system for surgical robots, characterized in that, The system includes: The task system is used to collect the execution statements of the task controller; A shared control system is used to input the execution statement into a large language model to obtain a simulated action sequence of the surgical robot, and send the simulated action sequence to a twin surgery system. The large language model is trained using execution statement samples and action sequence samples of the surgical robot. The twin surgery system includes a twin surgical robot corresponding to the surgical robot. The twin surgical robot performs surgical tasks on a virtual surgical organ according to the simulated action sequence in a virtual surgical scenario to obtain organ simulation state information. The organ simulation state information represents the organ biological state parameters obtained by the surgical robot performing the surgical task on the surgical organ according to the simulated action sequence in a real surgical scenario. The shared control system is used to adjust the control weights of the task system and the surgical robot according to the simulated organ state information; update the simulated action sequence according to the control weights to obtain an updated simulated action sequence; the updated simulated action sequence is used by the surgical robot to perform the surgical task in the real surgical scenario; The step of adjusting the control weights of the task system and the twin surgery system based on the organ simulation state information includes: The simulated organ state information is compared with the actual execution state information of the surgical robot to obtain a comparison result. The actual execution state information is obtained by the twin surgical robot executing the surgical task according to the planned path. The planned path is obtained by the control system of the surgical robot performing path planning according to the execution statement. Adjust the control weights of the task system and the surgical robot based on the comparison results; The step of adjusting the control weights of the task system and the surgical robot based on the comparison result includes: The control weights of the task system and the surgical robot are adjusted based on the comparison results and the real-time task information of the surgical task. The step of updating the simulated action sequence according to the control weights to obtain the updated simulated action sequence includes: When the control weight is greater than the first preset threshold, the simulated action sequence is updated according to the auxiliary statements collected by the task controller by the task system to obtain the updated simulated action sequence. If the control weight is less than the second preset threshold, the planned path is determined as the updated simulated action sequence.

2. The system according to claim 1, characterized in that, The step of inputting the executed statement into a large language model to obtain the simulated action sequence of the surgical robot includes: The execution statement is vectorized through the embedding layer of the large language model to obtain the execution vector; The execution vector is matched with the execution statement nodes in the surgical knowledge graph to obtain the simulated action sequence of the surgical robot. The surgical knowledge graph is obtained by inputting the execution statement samples, surgical scene information samples and action sequence samples into the inference engine. The execution statement nodes represent the execution statement samples.

3. The system according to claim 2, characterized in that, The real surgical scene information includes the surgical organ, the location information of the surgical robot, the surgical instrument information, and the surgical method information. The virtual surgical scene is obtained by modeling based on the real surgical scene information.

4. The system according to claim 1, characterized in that, The control system of the surgical robot performs path planning based on the executed statement using at least one of the following algorithms: feedforward control algorithm, feedback control algorithm, adaptive control algorithm, and sliding mode control algorithm, to obtain the planned path corresponding to the executed statement.

5. The system according to claim 1, characterized in that, The system also includes: If the deviation between the simulated action sequence and the planned path meets the deviation threshold, a feedback instruction is generated based on the organ simulation state information. The feedback instruction is sent to the task system.

Images