A surgical robot interaction system

Abstract

This invention discloses a surgical robot interaction system. The system is configured on a surgical robot, which includes a target screen. The system comprises a candidate interaction area display module and an interaction command execution module. The candidate interaction area display module displays at least one candidate interaction area on the target screen. The interaction command execution module, in response to detecting a gaze falling on the target screen, executes an interaction command corresponding to the target interaction area within the at least one candidate interaction area. This invention addresses the problems of high operational load and potential interference with surgical smoothness in surgical robot interaction.

Description

Technical Field

[0001] The present invention relates to the field of surgical technology, and in particular to a surgical robot interaction system. Background Technology

[0002] Surgical robots can employ a master-slave control architecture to achieve high-precision minimally invasive surgical procedures.

[0003] However, the current interaction modes used by surgical robots mainly rely on devices such as foot pedals, touch screens, and physical buttons. The limited control functions of foot pedals mean that complex interactions require frequent switching of foot pedals, resulting in a high operational load for surgical robot interactions. On the other hand, interaction via touch screens and physical buttons requires interrupting the current surgical action, which can easily disrupt the smoothness of the surgery. These issues urgently need to be addressed. Summary of the Invention

[0004] This invention provides a surgical robot interaction system that solves the problems of high operational load and easy interference with the smoothness of surgery in surgical robot interaction.

[0005] According to one aspect of the present invention, a surgical robot interaction system is provided, configured on a surgical robot, the surgical robot including a target screen, the system including: a candidate interaction area display module and an interaction command execution module; wherein, the candidate interaction area display module is used to display at least one candidate interaction area on the target screen; the interaction command execution module is used to, in response to detecting a gaze falling on the target screen, execute an interaction command corresponding to the target interaction area where the gaze falls on the target interaction area among the at least one candidate interaction area.

[0006] The technical solution of this invention is configured in a surgical robot, which includes a target screen. A candidate interaction area display module displays at least one candidate interaction area on the target screen, allowing the display of at least one candidate interaction area that the line of sight can fall upon. An interaction command execution module, in response to detecting a line of sight falling on the target screen, executes an interaction command corresponding to the target interaction area within the at least one candidate interaction area, thereby enabling surgical robot interaction. This technical solution, by responding to the detection of a line of sight falling on the target screen and executing an interaction command corresponding to the target interaction area, enables non-contact surgical robot interaction via line of sight, thus solving the problems of high operational load and easy interference with surgical smoothness in surgical robot interaction.

[0007] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a structural block diagram of a surgical robot interaction system provided according to an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of the target screen in a surgical robot interaction system provided according to an embodiment of the present invention.

[0011] Figure 3 This is a structural block diagram of another surgical robot interaction system provided according to an embodiment of the present invention.

[0012] Figure 4 This is a structural block diagram of another surgical robot interaction system provided according to an embodiment of the present invention.

[0013] Figure 5 This is a schematic diagram of at least one alternative risk area in another surgical robot interaction system provided according to an embodiment of the present invention.

[0014] Figure 6 This is a flowchart of an optional example of a surgical robot interaction system provided according to an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The same applies to "target," "original," etc., and will not be repeated here. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Before introducing the embodiments of the present invention, the implementation process of the current solutions for surgical robot interaction and the reasons for the problems of high operational load and easy interference with the smoothness of surgery will be explained by way of example, so as to better understand the solution proposed in the embodiments of the present invention and the reasons for solving the problems of high operational load and easy interference with the smoothness of surgery in surgical robot interaction.

[0018] Currently, surgical robots primarily rely on foot pedals, touchscreens, and physical buttons for interaction. However, foot pedals not only occupy lower limb operating space, but their limited control functions also necessitate frequent pedal switching to achieve complex interactions, increasing the workload on the surgical robot's operating objects and resulting in a high operational load for surgical robot interactions. On the other hand, interaction via touchscreens and physical buttons requires interrupting the current surgical action, severely disrupting the smoothness of the surgery, thus causing surgical robot interactions to easily interfere with the smoothness of the surgery.

[0019] To address this, embodiments of the present invention, in response to the detection of a gaze falling on the target screen, execute an interaction command corresponding to the target interaction area where the gaze falls. This enables non-contact surgical robot interaction via gaze, thereby solving the problems of high operational load and easy interference with surgical smoothness in surgical robot interaction. This will be described in detail below.

[0020] Figure 1 This is a structural block diagram of a surgical robot interaction system provided in an embodiment of the present invention. This embodiment is applicable to surgical robot interaction scenarios.

[0021] See Figure 1The surgical robot interaction system of this invention is configured on a surgical robot, which includes a target screen. The system includes a candidate interaction area display module 110 and an interaction command execution module 120. The candidate interaction area display module 110 is used to display at least one candidate interaction area on the target screen. The interaction command execution module 120 is used to execute an interaction command corresponding to the target interaction area in response to detecting a gaze falling on the target screen.

[0022] Here, a surgical robot can be understood as a robot capable of performing surgery; a surgical robot may include a master end and a slave end, and the patient can control the slave end to perform surgery on the patient through the operation of the master end, and the patient can interact with the surgical robot through the surgical robot interaction system configured on the master end; a surgical robot may be, for example, a laparoscopic surgical robot.

[0023] The target screen can be understood as a screen capable of interacting with the surgical robot; for example, the target screen could be the console screen in the main unit of the surgical robot; see also Figure 2 The target screen may include only the main console screen, or it may include an information display sub-screen and a 3D model display sub-screen. The information display sub-screen can be used to display processing cases and other content. Figure 2 (The information retrieved in the image), the sub-screen can display a three-dimensional reconstruction model of the surgical object's structure or three-dimensional medical images such as computed tomography (CT).

[0024] The alternative interaction area can be understood as an area that is selected as the target interaction area, that is, the area of ​​the interaction instruction corresponding to the alternative execution area. The alternative interaction area may include at least one of the following: the instrument area corresponding to each alternative instrument, the function area corresponding to each function button (e.g., the function interface (UI) button of the console edge system), the anatomical structure area, the medical information display area, the annotation information display area, the case processing display area, and the event processing information display area.

[0025] In this embodiment of the invention, the alternative interactive area display module 110 can display at least one alternative interactive area on the target screen. The display layout of the at least one alternative interactive area on the target screen can be seen, for example, as follows: Figure 2 As shown.

[0026] A line of sight can be understood as the line of sight falling on the target screen; a line of sight can be the line of sight of a certain object being operated on, or it can be any object on the target screen that the line of sight falls on.

[0027] The target interaction area can be understood as the alternative interaction area for the requirement to execute the corresponding interaction command.

[0028] Interactive instructions can be understood as instructions for interacting with a surgical robot; interactive instructions may include, for example, instructions for displaying medical information, instructions for annotating anatomical structures, instructions for displaying case processing information, instructions for displaying event processing information, instructions for triggering functions corresponding to functional areas, instructions for controlling target instruments, instructions for hiding medical information, instructions for switching surgical instruments, and camera control instructions (instructions for controlling the content displayed in the anatomical structure area), etc.

[0029] In this embodiment of the invention, interaction can be achieved by moving the gaze and interacting with at least one alternative interactive area displayed on the target screen. That is, the surgical robot interaction system can employ gaze tracking technology to capture the gaze position on the target screen. For example, a gaze tracker can be used (the installation location of the gaze tracker on the target screen is shown in [reference]). Figure 2 The feedback obtained by the interaction instruction execution module 120 (as shown) is the position of the line of sight on the target screen (e.g., the coordinates of the line of sight on the target screen), so that the interaction instruction execution module 120 can respond to the detection of the line of sight on the target screen, determine the target interaction area (the target interaction area where the line of sight is located), and then execute the interaction instruction corresponding to the target interaction area.

[0030] In this embodiment of the invention, the execution of the interaction command can be a first-level trigger, for example, in response to the detection of a gaze falling on the target screen, if the gaze falls on the target interaction area for a third duration and the third duration is longer than a third preset duration, the interaction command corresponding to the target interaction area is executed. To ensure the accuracy of the interaction, a composite trigger interaction mechanism can also be adopted, that is, the execution of the interaction command can be a first-level trigger plus a second-level confirmation, for example, if the gaze falls on the target interaction area for a fourth duration and the fourth duration is longer than a fourth preset duration, a second-level confirmation is entered (at this time, visual feedback can be triggered for the target interaction area, such as magnifying and / or highlighting the target interaction area, so that the user can understand the target interaction area that the interaction permission command can target), that is, in response to the interaction permission command, the interaction command corresponding to the target interaction area is executed. It should be noted that the terms "first duration," "second duration," "third duration," and "fourth duration" in the context are all essentially durations. They are simply different names used to distinguish durations in different application scenarios, and are not specific limitations on their actual content. Similarly, the terms "first preset duration," "second preset duration," "third preset duration," and "fourth duration" in the context are all preset durations.

[0031] It should be noted that, considering that some interactive commands may cause changes to the content displayed on the target screen, such as when executing an interactive command corresponding to an anatomical structure area to display medical information on the target screen, the target screen may add the display of medical information. Therefore, after executing an interactive command corresponding to the target interactive area, the alternative interactive area display module 110 may update at least one alternative interactive area to be displayed (or update at least one alternative interactive area to be displayed if the content displayed on the target screen changes).

[0032] For example, during the operation of the surgical robot by the user, if an error occurs due to an interruption mechanism such as overspeed disconnection, at least one alternative interaction area can be updated. The updated alternative interaction area includes the area corresponding to the Clear Error UI button. In response to detecting a gaze on the target screen, the interaction instruction execution module 120 executes the clear error instruction corresponding to the area corresponding to the Clear Error UI button, in response to an interaction permission instruction issued by the user through gesture, pedal, voice or other sensor mechanism, to avoid distraction during error clearing, thereby improving the smoothness of the surgery.

[0033] It should be noted that the solution of this invention not only eliminates the need for the operator to frequently switch foot pedals, thereby reducing the operational load when the operator interacts with the surgical robot and solving the problem of high operational load in surgical robot interaction, but also eliminates the need for the operator to manually trigger physical buttons and touch screens, thus avoiding interruption of surgical actions and solving the problem of easy interference with the smoothness of surgery. Furthermore, since it is a non-contact interaction through line of sight, it can also improve the human-computer interaction efficiency of the surgical robot.

[0034] The technical solution of this invention is configured in a surgical robot, which includes a target screen. A candidate interaction area display module displays at least one candidate interaction area on the target screen, allowing the display of at least one candidate interaction area that the line of sight can fall upon. An interaction command execution module, in response to detecting a line of sight falling on the target screen, executes an interaction command corresponding to the target interaction area within the at least one candidate interaction area, thereby enabling surgical robot interaction. This technical solution, by responding to the detection of a line of sight falling on the target screen and executing an interaction command corresponding to the target interaction area, enables non-contact surgical robot interaction via line of sight, thus solving the problems of high operational load and easy interference with surgical smoothness in surgical robot interaction.

[0035] An optional technical solution includes a surgical robot comprising a master end and a slave end, with the system configured at the master end and at least one alternative instrument installed at the slave end; an interaction command execution module comprising: a third interaction command execution submodule, configured to, in response to detecting a gaze falling on a target screen, execute an interaction command corresponding to the functional area where the gaze falls in the target interaction area of ​​the at least one alternative interaction area, if the target interaction area is a functional area, to trigger the function corresponding to the functional area; and / or, a fourth interaction command execution submodule, configured to, in response to detecting a gaze falling on the target screen, execute an interaction command corresponding to the instrument area where the gaze falls in the target interaction area of ​​the at least one alternative interaction area, if the target interaction area is an instrument area, to control the target instrument corresponding to the instrument area among the at least one alternative instrument.

[0036] The master end can be understood as the end of the surgical robot that can be operated by the object.

[0037] The term "end" can be understood as the end of a surgical robot that can perform surgery on the surgical object.

[0038] Alternate instruments can be understood as alternative controlled instruments; at least one alternative instrument can be a set of surgical instruments required to perform surgery on the surgical subject; each alternative instrument can correspond to an instrument area.

[0039] A functional area can be understood as an area where the corresponding function can be triggered.

[0040] In this embodiment of the invention, the third interactive instruction execution submodule can respond to the detection of a gaze falling on the target screen, and, if the target interactive area is a functional area, execute the interactive instruction corresponding to the functional area to trigger the function corresponding to the functional area.

[0041] The instrument area can be understood as the area where the corresponding target instrument can be controlled.

[0042] The target instrument can be understood as the alternative instrument corresponding to the instrument area.

[0043] In this embodiment of the invention, the fourth interactive instruction execution submodule can respond to the detection of a gaze falling on the target screen. If the gaze falls on the target interactive area, which is the device area, the module executes an interactive instruction corresponding to the device area (the interactive instruction may, for example, instruct the device activation state of the target device to be set to activated). This allows the module to control the target device corresponding to the device area, specifically, for example, a device group including the target device corresponding to the device area, thereby improving the dynamic adaptation capability of the interaction to the device.

[0044] In this embodiment of the invention, when controlling a target device, the device area can be displayed (e.g., highlighted or emphasized) to indicate that the target device is being controlled.

[0045] In this embodiment of the invention, by executing interactive instructions corresponding to the functional area, triggering the function corresponding to the functional area, and / or controlling the target instrument corresponding to the instrument area, interaction with the surgical robot for function triggering and / or control of the target instrument can be realized.

[0046] Figure 3 This is a structural block diagram of another surgical robot interaction system provided by an embodiment of the present invention. This embodiment is based on the above-mentioned technical solutions and optimized. In this embodiment, optionally, the interaction instruction execution module includes: a gaze area determination submodule, used to determine a gaze area on the target screen centered on the gaze position when the gaze falls on the target screen in response to detecting a gaze falling on the target screen, for at least one candidate interaction area where the gaze falls on the target interaction area, if the target interaction area is an anatomical structure area; a medical information acquisition submodule, used to perform anatomical structure semantic recognition on the gaze area to obtain medical information of the anatomical structure within the gaze area; and a first interaction instruction execution submodule, used to execute the interaction instruction corresponding to the anatomical structure area to display the medical information on the target screen.

[0047] The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0048] For details, see Figure 3 The surgical robot interaction system of this embodiment is configured on a surgical robot, which includes a target screen. The system includes: a candidate interaction area display module 210 and an interaction command execution module 220. The interaction command execution module 220 includes a gaze area determination submodule 221, a medical information acquisition submodule 222, and a first interaction command execution submodule 223. The candidate interaction area display module 210 is used to display at least one candidate interaction area on the target screen. The gaze area determination submodule 221 is used to, in response to detecting a gaze falling on the target screen, determine a gaze area centered on the gaze position on the target screen, provided the gaze falls on a target interaction area among the at least one candidate interaction area and the target interaction area is an anatomical structure area. The medical information acquisition submodule 222 is used to perform anatomical structure semantic recognition on the gaze area to obtain medical information about the anatomical structures within the gaze area. The first interaction command execution submodule 223 is used to execute the interaction command corresponding to the anatomical structure area to display the medical information on the target screen.

[0049] The anatomical structure area can be understood as the area used to display anatomical structures, or it can be understood as the area that displays the surgical field of view; the anatomical structure area can be the area displayed by the image of the anatomical structure (surgical field of view) provided by the endoscope system, etc.

[0050] The gaze position can be understood as the position where the gaze falls on the target screen.

[0051] The viewing area can be understood as the area on the target screen centered on the viewing position; the viewing area can be, for example, a region of preset size and / or preset shape centered on the viewing position.

[0052] In this embodiment of the invention, the gaze region determination submodule 221 can determine the gaze region in response to detecting a gaze falling on the target screen, for the target interaction area, if the target interaction area is an anatomical structure area.

[0053] In this embodiment of the invention, the gaze area determination submodule 221 can adopt an intelligent wake-up strategy. That is, in response to detecting a gaze falling on the target screen, if the target interaction area is an anatomical structure area, it determines a fourth duration for the gaze to fall on the anatomical structure area. If the fourth duration is longer than a fourth preset duration, and in response to an interaction permission command, it performs subsequent steps such as determining the gaze area. This ensures that the timing of medical information display matches the cognitive needs of the user, reduces visual interference, effectively solves the problem of the user's attention being easily distracted, reduces the frequency of medical information switching caused by the user's gaze switching, and thus improves surgical efficiency.

[0054] Anatomical structures can be understood as the anatomical structures displayed within the visual area.

[0055] Medical information can be understood as medical information related to anatomical structures; medical information may include at least one of the following: the name of the anatomical structure, a brief description of the structure, the distribution of blood vessels, the course of nerves, vital signs, and the state of the structure.

[0056] In this embodiment of the invention, the medical information acquisition submodule 222 can perform anatomical structure semantic recognition on the visual area (specifically, the image of the content displayed within the visual area) to obtain medical information. For example, the image of the content displayed within the visual area can be input into an image segmentation network such as U-Net to perform anatomical structure semantic recognition and obtain medical information.

[0057] In this embodiment of the invention, the first interactive instruction execution submodule 223 can execute interactive instructions corresponding to the anatomical structure region to display medical information on the target screen.

[0058] In this embodiment of the invention, displaying medical information on the target screen can be done in real time. That is, after executing the interactive command corresponding to the anatomical structure area to display medical information on the target screen, the anatomical structure semantic recognition of the field of vision area can be performed in real time to obtain medical information of the anatomical structure within the field of vision area, so as to update the displayed medical information based on the obtained medical information.

[0059] The technical solution of this invention includes a gaze area determination submodule. In response to detecting a gaze falling on the target screen, for at least one candidate interaction area where the gaze falls on the target interaction area, if the target interaction area is an anatomical structure area, a gaze area centered on the gaze position on the target screen is determined, thus defining a certain range for the area requiring anatomical structure semantic recognition. A medical information acquisition submodule performs anatomical structure semantic recognition on the gaze area to obtain medical information about the anatomical structures within the gaze area, thereby determining the medical information to be displayed. A first interaction instruction execution submodule executes the interaction instruction corresponding to the anatomical structure area to display the medical information on the target screen. Compared to solutions that require the user to independently recognize the anatomical structure information in the gaze area, leading to information overload and high cognitive load, this technical solution automatically identifies and displays medical information on the target screen, eliminating the need for the user to independently recognize the anatomical structure information in the gaze area. This avoids information overload and high cognitive load, solves the problem of the separation between medical information display and the surgical scene, and allows the user to directly access the displayed medical information without switching gaze positions.

[0060] An optional technical solution, the surgical robot interaction system, further includes: a labeling information acquisition module and an anatomical structure labeling module; wherein, the labeling information acquisition module is used to acquire labeling information in response to a labeling instruction, wherein the labeling information includes at least one of labeling content, labeling timestamp, and spatial location of the anatomical structure; the anatomical structure labeling module is used to label the anatomical structure according to the labeling information.

[0061] The annotation instruction can be understood as an instruction to annotate the anatomical structure; annotations can be generated in various ways such as voice commands, gesture confirmation, or foot switch.

[0062] The annotation information can be understood as information used to annotate anatomical structures; the annotation information can be at least one of the following: teaching annotation information, information exchange annotation information, surgical experience annotation information, and surgical process annotation information; the annotation information includes at least one of the following: annotation content, annotation timestamp, spatial location of anatomical structure, precautions, and characteristic description of anatomical structure (which can be generated by artificial intelligence (AI)).

[0063] The annotation content can be understood as the annotation of anatomical structures; for example, the annotation content can be obtained by converting the voice command corresponding to the annotation instruction into text.

[0064] A timestamp for annotation can be understood as the timestamp indicating that an anatomical structure is annotated in response to an annotation instruction; for example, a timestamp could be the timestamp corresponding to a surgical video frame in response to an annotation instruction. This surgical video frame can be understood as the video frame corresponding to the surgical field of view displayed in the anatomical structure area in response to the annotation instruction, that is, the video frame corresponding to the image displayed in the anatomical structure area in response to the annotation instruction.

[0065] Spatial location can be understood as the position of an anatomical structure in space; location information can be, for example, the position of the anatomical structure displayed in the anatomical structure area, the position of the anatomical structure displayed on the target screen, the position of the anatomical structure in the 3D navigation model corresponding to the organ or tissue targeted in the surgery, the position of the anatomical structure in the world coordinate system, the position of the anatomical structure in the object coordinate system of the surgical object, and so on.

[0066] In this embodiment of the invention, the annotation information acquisition module can respond to the annotation instruction and acquire annotation information including at least one of the annotation content, annotation timestamp and spatial location.

[0067] In this embodiment of the invention, the anatomical structure annotation module can annotate anatomical structures according to the annotation information, so as to annotate the content that needs to be annotated during the operation through the annotation information converted from data. In this way, the annotation of anatomical structures can be applied in multiple scenarios. For example, in the case of information exchange between the operation target and the assistant target, the assistant target can directly obtain the operation target's intention based on the field of view of the shared annotation information such as the imaging trolley, and also provide an efficient and accurate platform for information exchange during the surgical process, improving the efficiency of communication between the targets and the accuracy of intention transmission. For another example, the annotation of anatomical structures can provide standardized multimedia data support for subsequent surgical teaching review and academic exchanges.

[0068] For example, in gallbladder surgery, the line-of-sight region determination submodule, in response to detecting a line of sight falling on the target screen, determines that the line of sight falls on an anatomical structure region, and thus determines the line-of-sight region; the medical information acquisition submodule performs anatomical structure semantic recognition on the line-of-sight region to obtain medical information about the gallbladder artery within the line-of-sight region, including the blood vessels of the gallbladder artery; the first interaction instruction execution submodule executes the interaction instruction corresponding to the anatomical structure region to display the outline of the blood vessel for the gallbladder artery on the target screen; the annotation information acquisition module, in response to the annotation instruction generated by the operation object through voice input, acquires annotation information, including the gallbladder blood vessel protection zone and precautions related to the gallbladder blood vessels; the anatomical structure annotation module associates the annotation information with the corresponding surgical video frame and annotates the spatial position of the three-dimensional model corresponding to the gallbladder artery to achieve annotation of the anatomical structure.

[0069] In this embodiment of the invention, after annotating the anatomical structures, the annotation results can be archived to facilitate timely marking and archiving of key nodes in the surgery, providing a reference basis for other subsequent marking-related scenarios.

[0070] The solution of this invention, by annotating anatomical structures based on annotation information during interaction with the surgical robot, can avoid the time-consuming and labor-intensive problem of offline annotation after surgery, which leads to high manpower and resource costs. In addition, the solution of this invention also helps to improve the efficiency of communication between objects and the accuracy of intent transmission, and provides data support for subsequent surgical teaching review and academic exchanges.

[0071] Another optional technical solution, the surgical robot interaction system, further includes: a case determination module and a case display module; wherein, the case determination module is used to determine the case of the event corresponding to the target rule when the medical information meets the target rule in the rule base; the case display module is used to display the case.

[0072] A rule base can be understood as a rule base that includes target rules; a rule base may include at least one rule, and at least one rule includes the target rule; a rule base may be preset; a rule base may include rules corresponding to at least one relevant indicator such as surgical stage, activated instrument type, alarm status, and abnormal signs of the surgical subject.

[0073] The target rule can be understood as the rule that medical information is satisfied.

[0074] An event can be understood as an event corresponding to the target rule; an event can be at least one event such as an event during the surgical phase (such as bleeding), an instrument activation event, an alarm event, or an abnormal physical sign event of the surgical subject.

[0075] A case study can be understood as a case of handling an event. Specifically, a case study can be understood as a case obtained after handling a corresponding event in history.

[0076] In this embodiment of the invention, the processing case determination module can determine a processing case when the medical information meets the target rules in the rule base. For example, when the medical information meets the target rules in the rule base, the processing case determination module can retrieve processing cases associated with the event from the knowledge base through a case retrieval engine.

[0077] In this embodiment of the invention, the processing case determination module can also determine the processing case if the content displayed in the anatomical structure area satisfies the target rule in the rule base.

[0078] In this embodiment of the invention, the processing case display module can display processing cases. The processing case display module can display processing cases on the target screen and also on the secondary screen of the console.

[0079] In this embodiment of the invention, by identifying and displaying processing cases, reference processing cases are provided for dealing with events, so as to provide dynamic adaptation capabilities when facing risk events, thereby helping the processing object to handle the corresponding events in a timely manner through processing cases.

[0080] Based on the above solution, another optional technical solution, the surgical robot interaction system, further includes: an event processing information determination module and an event processing information display module; wherein, the event processing information determination module is used to determine event processing information based on the processing case, wherein the event processing information includes at least one of the following: event association structure, recommended instruments, attention indicators, and event processing trajectory; the event processing information display module is used to display the event processing information.

[0081] Event handling information can be understood as information related to the handling of events.

[0082] Event-related structures can be understood as anatomical structures associated with an event; for example, they can be associated organs and / or associated tissues, etc.

[0083] Recommended equipment can be understood as the equipment recommended for handling an incident.

[0084] The metrics to focus on can be understood as the metrics that are important for handling event requirements.

[0085] An event handling trajectory can be understood as the operational path that can be followed to handle an event; an event handling trajectory can exist in the form of an operation guide or operation steps, etc.

[0086] In this embodiment of the invention, the event processing information determination module can determine event processing information, including at least one of the following: event association structure, recommended equipment, attention indicators, and event processing trajectory, based on the processing case.

[0087] In this embodiment of the invention, the event processing information display module can display event processing information; the event processing information display module can display event processing information on the target screen or the console sub-screen, and the event processing information can also be displayed in a manner corresponding to each type of information content (at least one of event-related structure, recommended instruments, attention indicators and event processing trajectory), for example, when the information content includes an event-related structure, the corresponding event-related structure is highlighted in the anatomical structure area.

[0088] In this embodiment of the invention, only event handling information may be displayed, without displaying handling cases.

[0089] In this embodiment of the invention, the case determination module can also determine whether the target rule meets the deactivation rule if the medical information meets the target rule in the rule base. If the target rule meets the deactivation rule, other content such as medical information displayed on the target screen can be frozen or deactivated (e.g., hidden) to avoid interfering with the view of event processing.

[0090] For example, when bleeding time appears in the field of vision, the case determination module detects that the medical information meets the target rules corresponding to the sudden drop in blood pressure and bleeding events in the rule base, determines that the target rules meet the deactivation rules, freezes and deactivates the medical information displayed on the target screen, and determines the case for the sudden drop in blood pressure and bleeding events. The case display module displays the case. The event processing information determination module determines the event processing information based on the case, which includes indicators of concern such as blood pressure and heart rate, recommended instruments (hemostatic instruments), and operating instructions. The event processing information display module highlights the indicators of concern such as blood pressure and heart rate, highlights the instrument area corresponding to the hemostatic instrument, and displays the operating instructions on the secondary screen of the console.

[0091] The technical solution of this invention, through the determination and display of event handling information, can efficiently integrate handling cases into event handling information when an event occurs, and promptly display event handling information that can be used for reference in handling the event. By displaying event handling information, the cognitive load in the face of an event is reduced, and decision-making is quickly provided when facing an event, especially in the face of a crisis event, thereby helping to improve the decision-making efficiency of the operator in the face of an event.

[0092] Another optional technical solution, the surgical robot interaction system, further includes: a structural position determination module; wherein, the structural position determination module is used to determine the structural position of the anatomical structure on the target screen; and a first interaction instruction execution submodule, specifically used to execute interaction instructions corresponding to the anatomical structure area, with the goal of displaying medical information at the structural position with the unobstructed anatomical structure area as the target.

[0093] Here, structural location can be understood as the position of the anatomical structure on the target screen.

[0094] In this embodiment of the invention, the structure location determination module can determine the structure location.

[0095] In this embodiment of the invention, the first interactive instruction execution submodule can execute interactive instructions corresponding to the anatomical structure region to render and display medical information with the goal of displaying the unobstructed anatomical structure region at the structural location (semi-transparent or with highlighted structural edges).

[0096] The technical solution of this invention, by displaying medical information with the goal of displaying the anatomical structure area without obscuring it at the structural location, can avoid the display of medical information obscuring the display of the anatomical structure area.

[0097] Figure 4 This is a structural block diagram of another surgical robot interaction system provided by an embodiment of the present invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, at least one alternative risk area is displayed on the target screen, each alternative risk area including at least a portion of the alternative interaction areas; the interaction instruction execution module includes: a target interaction area determination submodule, used to determine, in response to detecting a line of sight falling on the target screen, that the line of sight in at least one alternative interaction area falls on the target interaction area; a target risk area determination submodule, used to determine, from the at least one alternative risk area, the target risk area where the target interaction area is located; and a second interaction instruction execution submodule, used to execute the interaction instruction corresponding to the target interaction area according to the target risk area. The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0098] For details, see Figure 4The surgical robot interaction system of this embodiment is configured on a surgical robot. The surgical robot includes a target screen, on which at least one alternative risk area is displayed. Each alternative risk area includes at least a portion of the alternative interaction areas. The system includes: an alternative interaction area display module 310 and an interaction command execution module 320. The interaction command execution module 320 includes a target interaction area determination submodule 321, a target risk area determination submodule 322, and a second interaction command execution submodule 323. The alternative interaction area display module 310 is used to display at least one alternative interaction area on the target screen. The target interaction area determination submodule 321 is used to determine, in response to detecting a line of sight falling on the target screen, that the line of sight in at least one alternative interaction area falls on a target interaction area. The target risk area determination submodule 322 is used to determine, from at least one alternative risk area, the target risk area where the target interaction area is located. The second interaction command execution submodule 323 is used to execute an interaction command corresponding to the target interaction area based on the target risk area.

[0099] Here, alternative risk areas can be understood as areas that are considered as target risk areas; alternative interaction areas within different alternative risk areas can correspond to different risks.

[0100] In this embodiment of the invention, at least one alternative risk area is displayed on the target screen, and each alternative risk area includes at least a portion of the alternative interaction area. The at least one alternative interaction area can be divided according to risk, and the area where the at least portion of the alternative interaction area is located under a given area is taken as the alternative risk area corresponding to the risk of the at least portion of the alternative interaction area. That is, the at least one alternative risk area displayed on the target screen can be understood as dividing the at least one alternative interaction area displayed into different alternative risk areas.

[0101] The target risk area can be understood as the alternative risk area where the target interaction area is located.

[0102] In this embodiment of the invention, the target interaction area determination submodule 321 can determine the target interaction area in response to detecting a line of sight falling on the target screen; the target risk area determination submodule 322 can determine the target risk area from at least one alternative risk area. It should be noted that the target interaction area is located in the target risk area at this time; the second interaction instruction execution submodule 323 can execute the interaction instruction corresponding to the target interaction area according to the target risk area.

[0103] The technical solution of this invention involves displaying at least one candidate risk area on the target screen, each candidate risk area including at least a portion of at least one candidate interaction area; a target interaction area determination submodule, in response to detecting a gaze falling on the target screen, determines that the gaze of the gaze in the at least one candidate interaction area falls on the target interaction area; a target risk area determination submodule, from the at least one candidate risk area, determines the target risk area where the target interaction area is located; and a second interaction instruction execution submodule, based on the target risk area, executes the interaction instruction corresponding to the target interaction area. This technical solution, by executing the interaction instruction based on the target risk area, ensures that the execution of the interaction instruction conforms to the risk of the corresponding target risk area, thereby improving the safety of surgical robot interaction.

[0104] An optional technical solution includes a second interactive instruction execution submodule, comprising: a first interactive instruction execution unit, configured to determine a first duration for which the line of sight falls on the target interactive area when the target risk area is a first risk area, and to execute an interactive instruction corresponding to the target interactive area when the first duration is longer than a first preset duration; and / or, a second interactive instruction execution unit, configured to determine a second duration for which the line of sight falls on the target interactive area when the target risk area is a second risk area, and to execute an interactive instruction corresponding to the target interactive area when the second duration is longer than a second preset duration, and in response to an interactive permission instruction; wherein the risk corresponding to the first risk area is less than the risk corresponding to the second risk area.

[0105] The first risk area can be understood as an area where the risk is lower than that corresponding to the second risk area; the first risk area can also be called a low-risk area. The first risk area may, for example, include various functional areas from at least one alternative interaction area, such as see [reference needed]. Figure 5 The area where each functional area of ​​the target screen edge is located can be considered as the first risk area. Figure 5 (Low-risk areas).

[0106] The first duration can be understood as the duration for which the gaze remains on the target interaction area.

[0107] The first preset duration can be understood as the maximum duration of the first period of time when no interaction command corresponding to the target interaction area is executed. It should be noted that the preset duration can be an adjustable parameter, for example, it can be adjusted in response to a preset duration adjustment command, and the preset duration can be updated based on the adjustment result.

[0108] In this embodiment of the invention, the first interactive instruction execution unit can determine a first duration when the target risk area is a first risk area, and execute an interactive instruction corresponding to the target interactive area when the first duration is longer than a first preset duration.

[0109] The second risk zone can be understood as the area where the risk is greater than that corresponding to the first risk zone.

[0110] The second duration can be understood as the duration for which the gaze remains on the target interaction area.

[0111] The second preset duration can be understood as the maximum duration of the second duration when the interaction command corresponding to the target interaction area is not executed.

[0112] An interaction permission instruction can be understood as an instruction that allows the execution of an interaction instruction; interaction permission instructions can be generated through micro-gesture recognition (such as finger micro-movement), foot pedal light touch, voice keywords, or other sensor mechanisms.

[0113] In this embodiment of the invention, the second interactive instruction execution unit can determine a second duration when the target risk area is a second risk area, and execute an interactive instruction corresponding to the target interactive area when the second duration is longer than a second preset duration and in response to an interactive permission instruction.

[0114] The solution of this invention executes the interaction command when the target risk area is a first risk area and the first duration is longer than the first preset duration; and / or, when the target risk area is a second risk area and the second duration is longer than the second preset duration and in response to the interaction permission command, the interaction command is executed. This ensures that the execution of the interaction command meets the execution conditions of the corresponding target risk area, thereby further improving the safety of surgical robot interaction.

[0115] Based on the above solution, another optional technical solution is that the second risk area includes a first sub-area and a second sub-area, and the risk corresponding to the first sub-area is less than the risk corresponding to the second sub-area; when the target risk area is the second sub-area, the interaction instruction is an instruction other than the operation instruction.

[0116] The first sub-region can be understood as a region where the risk is greater than that of the second risk region but less than that of the second sub-region. The first sub-region can also be called a medium-risk region. For example, the first sub-region may include various device regions from at least one alternative interaction region, such as see [reference needed]. Figure 5 The area at the bottom of the target screen, including the instrument operation UI interface for each instrument area, can be used as the first sub-area. Figure 5(Medium-risk areas).

[0117] The second sub-region can be understood as a region with a higher risk than the first sub-region; the second sub-region can also be called a high-risk region. For example, the second sub-region may include at least one anatomical structure region from the alternative interaction regions, see [link to relevant documentation]. Figure 5 The anatomical structure region located at the center of the target screen's field of view can be used as the second sub-region. Figure 5 (High-risk areas).

[0118] Operation instructions can be understood as instructions that produce operational actions; for example, operation instructions can be related to the operation of a surgical object, and operation instructions can also be instructions other than display instructions, etc.

[0119] It is understandable that performing operations directly in the second sub-region, such as the corresponding high-risk anatomical structure area, is prone to safety issues. Therefore, when the target risk area is the second sub-region, the interaction instructions can be instructions other than the operation instructions, in order to avoid the occurrence of safety issues due to operations in the corresponding high-risk area, thereby improving the safety of surgical robot interaction and ensuring the safety redundancy of critical operations.

[0120] In this embodiment of the invention, when the target risk area is a first risk area, the interactive instruction can be at least one of the following: an instruction of the information display type, an instruction of the information hiding type, and an instruction of the operation type. When the target risk area is a first sub-area, the interactive instruction can be at least one of the following: an instruction of the information display type, an instruction of the information hiding type, and an instruction of the operation type (device group switching or activation, or lens control). When the target risk area is a second sub-area, the interactive instruction is an instruction other than an operation instruction (e.g., a labeling instruction or a medical information display instruction).

[0121] The solution of this invention improves the safety of surgical robot interaction by using interactive instructions other than operation instructions when the target risk area is a second sub-region.

[0122] To better understand the technical solutions of the above embodiments of the present invention, an optional example is provided herein. For example, see... Figure 6The surgical robot interaction system integrates a gaze-guided interaction mechanism, a dynamic surgical field spatial binding mechanism, a multimodal intraoperative annotation mechanism, an intelligent information recommendation engine, and an interaction risk control strategy. This enables interaction with the surgical robot, allowing for coordinated control of anatomical structures, surgical instruments, and information display during surgery. Specifically, the gaze-guided interaction mechanism responds to the detection of a gaze falling on the target screen; the dynamic surgical field spatial binding mechanism performs anatomical semantic recognition of the gaze area to obtain medical information and executes corresponding interaction commands to display this information on the target screen; the multimodal intraoperative annotation mechanism generates annotation commands in response to voice commands, gesture confirmations, and / or foot switches, acquires annotation information, and annotates anatomical structures based on this information; the intelligent information recommendation engine recommends at least one of the processed cases and event processing information when the medical information meets the target rules in the rule base; and the interaction risk control strategy executes interaction commands corresponding to the target interaction area based on the target risk area. Specifically, the surgical robot interaction system can use a gaze tracker and a gaze-guided interaction mechanism to detect the presence of a gaze on the target screen. In response to the detection of a gaze on the target screen, it determines the target interaction area where the gaze falls, and performs a focus type determination on the target interaction area, i.e., whether the target interaction area is an anatomical structure area, a functional area, or an instrument area. If the target interaction area is an anatomical structure area, a dynamic surgical field spatial binding mechanism is used, through a semantic understanding engine, to perform anatomical structure semantic recognition on the gaze area, obtain medical information, and execute the interaction commands corresponding to the anatomical structure area to display the medical information on the target screen, thereby achieving surgical field spatial binding. Through a multimodal intraoperative annotation mechanism, in response to annotation... The system provides instructions to annotate anatomical structures for multimodal intraoperative annotation, storing the annotation results in the surgical dataset. An intelligent information recommendation engine recommends and displays functional information such as case processing and event handling information, provided the medical information meets the target rules in the rule base. When the target interaction area is a functional area in the information menu, the system executes the corresponding interaction instructions to trigger the corresponding function, thereby controlling the information system. When the target interaction area is an instrument area in the instrument UI, the system executes the corresponding interaction instructions to control instrument groups and / or endoscopes, including the target instruments corresponding to the instrument area. The surgical robot interaction system employs interaction risk control strategies during the interaction process. This basic solution achieves non-contact, low cognitive load, safe, and efficient surgical robot interaction.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A surgical robotic interaction system, characterized by, The system is configured to a surgical robot, the surgical robot comprising a target screen, the system comprising: an alternative interaction area display module configured to display at least one alternative interaction area on the target screen; an interaction instruction execution module configured to, in response to detecting a line of sight falling on the target screen, execute an interaction instruction corresponding to a target interaction area in which the line of sight falls, among the at least one alternative interaction area; at least one alternative risk area is displayed on the target screen, each of the at least one alternative risk area comprising at least part of the at least one alternative interaction area; the interaction instruction execution module comprises: a target interaction area determination submodule configured to, in response to detecting a line of sight falling on the target screen, determine a target interaction area in which the line of sight falls, among the at least one alternative interaction area; a target risk area determination submodule configured to determine, from the at least one alternative risk area, a target risk area in which the target interaction area is located; a second interaction instruction execution submodule configured to execute an interaction instruction corresponding to the target interaction area according to the target risk area; the second interaction instruction execution submodule comprises: a first interaction instruction execution unit configured to, in the case that the target risk area is a first risk area, determine a first duration in which the line of sight falls on the target interaction area, and execute an interaction instruction corresponding to the target interaction area in the case that the first duration is greater than a first preset duration; a second interaction instruction execution unit configured to, in the case that the target risk area is a second risk area, determine a second duration in which the line of sight falls on the target interaction area, and execute an interaction instruction corresponding to the target interaction area in the case that the second duration is greater than a second preset duration and in response to an interaction permission instruction; wherein the risk corresponding to the first risk area is less than the risk corresponding to the second risk area; the second risk area comprises a first sub-area and a second sub-area, and the risk corresponding to the first sub-area is less than the risk corresponding to the second sub-area; in the case that the target risk area is the second sub-area, the interaction instruction is an instruction other than an operation instruction.

2. The system of claim 1, wherein, the interaction instruction execution module comprises: a line of sight area determination submodule configured to, in response to detecting a line of sight falling on the target screen, determine, for a target interaction area in which the line of sight falls, among the at least one alternative interaction area, a line of sight area on the target screen centered on a line of sight position in which the line of sight falls, in the case that the target interaction area is an anatomical structure area; a medical information obtaining submodule configured to perform anatomical structure semantic recognition on the line of sight area to obtain medical information of an anatomical structure in the line of sight area; a first interaction instruction execution submodule configured to execute an interaction instruction corresponding to the anatomical structure area to display the medical information on the target screen.

3. The system of claim 2, wherein, further comprising: The labeling information acquisition module is configured to acquire labeling information in response to a labeling instruction, wherein the labeling information comprises at least one of labeling content, a labeling timestamp, and a spatial position of the anatomical structure. The anatomical structure labeling module is configured to label the anatomical structure according to the labeling information.

4. The system of claim 2, wherein, Further comprising: The processing case determination module is configured to determine a processing case of an event corresponding to a target rule in a case where the medical information satisfies the target rule in a rule library. The processing case display module is configured to display the processing case.

5. The system of claim 4, wherein, Further comprising: The event processing information determination module is configured to determine event processing information according to the processing case, wherein the event processing information comprises at least one of event-related structures, recommended instruments, attention indicators, and event processing trajectories. The event processing information display module is configured to display the event processing information.

6. The system of claim 2, wherein, Further comprising: The structure position determination module is configured to determine a structure position of the anatomical structure on the target screen. The first interaction instruction execution submodule is configured to execute an interaction instruction corresponding to the anatomical structure region, so as to display the medical information at the structure position in a manner of unoccluded display of the anatomical structure region.

7. The system of claim 1, wherein, The surgical robot comprises a master end and a slave end, the system is configured at the master end, and at least one alternative instrument is installed at the slave end; The interaction instruction execution module comprises: The third interaction instruction execution submodule is configured to, in response to detection of a line of sight falling on the target screen, execute an interaction instruction corresponding to a target interaction region in which the line of sight falls in at least one of the alternative interaction regions, in a case where the target interaction region is a function region, so as to trigger a function corresponding to the function region. And / or, The fourth interaction instruction execution submodule is configured to, in response to detection of a line of sight falling on the target screen, execute an interaction instruction corresponding to a target interaction region in which the line of sight falls in at least one of the alternative interaction regions, in a case where the target interaction region is an instrument region, so as to control a target instrument corresponding to the instrument region in at least one of the alternative instruments.

Images