Surgical robot interaction method, apparatus, robot and medium

Abstract

Embodiments of the present application provide a surgical robot interaction method, device, robot and medium. The method comprises: acquiring surgical image data and user voice instructions in real time; inputting the surgical image data into a preset anomaly detection model to determine anomaly perception information corresponding to the surgical image data; outputting the anomaly perception information through a non-speech prompt mode, the non-speech prompt mode at least including visual prompting; in response to the user voice instructions, outputting corresponding voice response information; wherein the output of the anomaly perception information and the output of the voice response information are performed through mutually independent non-speech channels and voice channels. The method is used to improve the recognition degree and response speed of intraoperative anomaly warning, and enhances the smoothness and overall safety of surgical operation.

Description

Technical Field

[0001] This application relates to the field of surgical robot technology, and in particular to a surgical robot interaction method, device, robot, and medium. Background Technology

[0002] With the intelligent development of surgical robot technology, especially in surgical scenarios requiring high-precision operation and real-time decision support, such as laparoscopic and single-port surgeries, surgeons not only need to operate robotic arms to complete delicate dissection and suturing movements, but also need to monitor dynamic changes within the surgical field of view in real time and make accurate decisions based on real-time information. At the same time, doctors often need to obtain individualized patient information or expert advice through natural language interaction to assist in their judgment.

[0003] In related technologies, surgical robots mostly exist in the form of "assistive operation tools," passively executing the doctor's operation instructions, lacking the ability to actively perceive and intelligently interact. Moreover, existing intraoperative abnormality mechanisms mostly use voice broadcasting to indicate abnormalities, which can easily occupy the doctor's attention resources, interfere with the surgical operation, and affect the safety and efficiency of the surgery. Summary of the Invention

[0004] This application provides a surgical robot interaction method, device, robot, and medium to reduce the risks of traditional surgical robots sharing an audio channel for all interactive information, and the risk of intraoperative warnings being easily masked by operational dialogue or environmental noise. It also reduces the possibility of doctors missing risk prompts while focusing on the operation and the possibility of misaligned command responses caused by interference between multiple types of voice information.

[0005] In a first aspect, embodiments of this application provide a surgical robot interaction method, including:

[0006] Real-time acquisition of surgical image data and user voice commands;

[0007] The surgical image data is input into a preset anomaly detection model to determine the anomaly perception information corresponding to the surgical image data.

[0008] The abnormality perception information is output through a non-voice prompt method, which includes at least a visual prompt method.

[0009] In response to user voice commands, output corresponding voice response information;

[0010] The output of the anomaly perception information and the output of the voice response information are performed through independent non-voice channels and voice channels.

[0011] In one possible implementation, outputting the anomaly perception information via a non-voice prompt method includes:

[0012] The risk level is determined based on the anomaly detection information;

[0013] When the risk level is higher than the preset level, the abnormal perception information is output through a non-voice prompt. The visual prompt includes triggering indicator light flashing and / or displaying real-time surgical image data with warning labels superimposed on the graphical user interface.

[0014] In one possible implementation, the anomaly detection information includes at least one of the following:

[0015] Target categories, which include bleeding areas, abnormal surgical instrument conditions, and foreign bodies;

[0016] The location coordinates corresponding to the target category;

[0017] The bounding rectangle corresponding to the target category;

[0018] The confidence level corresponding to the target category;

[0019] The risk level corresponding to the target category.

[0020] In one possible implementation, the step of responding to a user's voice command and outputting corresponding voice response information includes:

[0021] Parse the user's voice command to obtain the query intent corresponding to the user's voice command;

[0022] By using a pre-set knowledge base, target knowledge text that matches the query intent is obtained;

[0023] A response text matching the target knowledge text is generated using a preset natural language model, and the response text is then synthesized to output the corresponding speech response information.

[0024] In one possible implementation, obtaining target knowledge text matching the query intent through a preset knowledge base includes:

[0025] If the query intent is a professional knowledge query, then the target knowledge text that matches the surgical image data and / or the query intent is obtained through a preset expert knowledge base;

[0026] The preset expert knowledge base includes at least one of the following: structured or semi-structured surgical operation guidelines, complication management plans, and instrument usage guidelines.

[0027] In one possible embodiment, obtaining target knowledge text matching the query intent through a preset knowledge base includes:

[0028] If the query intent is a personalized patient information query, then a target knowledge text matching the query intent is obtained through a pre-built localized patient database. The localized database contains at least one of the patient's medical records, past medical history, allergy history, and laboratory test reports.

[0029] In one possible implementation, the method for constructing the localized patient database includes:

[0030] Before the surgery begins, a pluggable storage medium containing patient data is attached to the surgical robot to build the localized patient database.

[0031] During the procedure, the pluggable storage medium is kept in a read-only state.

[0032] After the procedure, the pluggable storage medium can be physically removed.

[0033] Secondly, embodiments of this application provide a surgical robot interaction device, comprising:

[0034] The acquisition module is used to acquire surgical image data and user voice commands in real time;

[0035] The perception module is used to input the surgical image data into a preset anomaly detection model in order to determine the anomaly perception information corresponding to the surgical image data.

[0036] The prompting module is used to output the abnormality perception information through a non-voice prompting method, wherein the non-voice prompting method includes at least a visual prompting method;

[0037] The response module is used to respond to user voice commands and output corresponding voice response information;

[0038] The prompting module and the response module output information through independent non-voice channels and voice channels, respectively.

[0039] Thirdly, embodiments of this application provide a surgical robot, comprising:

[0040] Pluggable interface, memory, processor, visual prompting unit, and voice output unit;

[0041] The pluggable interface is used to connect a pluggable storage medium, which is configured to remain read-only during the operation and can be physically removed after the operation for the processor to read data.

[0042] The visual prompting unit is used to output visual prompts, and the voice output unit is used to output voice response information;

[0043] The memory stores computer-executed instructions;

[0044] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0046] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0047] The surgical robot interaction method, device, robot, and medium provided in this application synchronously collect surgical image data and user voice commands in real time. Based on a preset abnormality detection model, it automatically identifies intraoperative abnormal perception information and outputs it independently through a non-voice channel such as visual prompts. At the same time, it matches the corresponding voice response to the user's actively input voice command and outputs it through an independent voice channel. This reduces the risk that all interactive information of traditional surgical robots shares an audio channel and that intraoperative warnings are easily masked by operation dialogues or environmental noise. It also reduces the possibility that doctors may miss risk prompts while focusing on the operation and that interference between multiple types of voice information may lead to misaligned command responses. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] Figure 1 A flowchart illustrating the surgical robot interaction method provided in the embodiments of this application. Figure 1 ;

[0050] Figure 2 A flowchart illustrating the surgical robot interaction method provided in the embodiments of this application. Figure 2 ;

[0051] Figure 3 This is an architectural diagram of a surgical robot system provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of the surgical robot interaction device provided in the embodiments of this application;

[0053] Figure 5 This is a schematic diagram of the structure of the surgical robot provided in an embodiment of this application.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0056] First, let me explain the terms used in this application:

[0057] Surgical robots are high-end intelligent medical devices that integrate multimodal image fusion, high-precision mechanical control, intelligent sensing, and human-computer interaction technologies. They break the operational boundaries of traditional surgery, enabling doctors to perform sub-millimeter-level precision surgical movements under their guidance. They can also capture key information such as intraoperative images and operational trajectories through active sensing systems, assisting doctors in avoiding risks and planning pathways. Currently, they are widely used in dozens of sub-surgical scenarios, including neurosurgery, orthopedics, laparoscopy, and vascular intervention. With their core advantages of being minimally invasive, precise, and having a fast recovery, they have become a core supporting tool for promoting the intelligent upgrade of modern minimally invasive surgery.

[0058] Specifically, as a "co-pilot" for doctors during surgery, surgical robots need to have the ability to perceive abnormal conditions in the surgical field of vision (such as bleeding, foreign objects, and instrument status) in real time, as well as the ability to respond to doctors' natural language questions and provide expert knowledge and personalized information to patients, so as to improve the safety and efficiency of surgery without interfering with the doctor's main operating procedures.

[0059] However, existing surgical robots have a relatively centralized way of organizing intraoperative perception information and interactive responses, and the way of outputting abnormal information is singular. They are difficult to take into account the usage characteristics of two different tasks: continuous perception and on-demand question and answer. Doctors are more likely to be disturbed by unnecessary voice during high-intensity surgeries.

[0060] Based on this, this application provides a surgical robot interaction method. By synchronously and parallelly acquiring real-time intraoperative image data and voice commands actively input by the doctor, and relying on a pre-trained anomaly detection model, it automatically identifies hidden anomalies in the images and generates exclusive anomaly perception information. All risk warning content is output in a targeted manner through non-voice modes such as visual cues. At the same time, a layered interaction mechanism that matches the doctor's operation commands with exclusive voice feedback reduces the risk of traditional surgical robots sharing an audio channel for all interactive information and intraoperative warnings being easily masked by operation dialogue or environmental noise. It also reduces the possibility of doctors missing risk warnings while focusing on operation and the possibility of misaligned command responses due to interference between multiple types of voice information.

[0061] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0062] Figure 1 A flowchart illustrating the surgical robot interaction method provided in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the specific implementation process of this surgical robot interaction method includes at least steps S101 to S104, which are described in detail below:

[0063] Step S101: Acquire surgical image data and user voice commands in real time.

[0064] For example, during the operation, on the one hand, high-definition image data streams synchronously output by the surgical endoscope throughout the operation are collected in real time. The image data fully covers the surgical operation area, instrument end posture, surrounding tissue status and other full-dimensional visual information within the endoscopic field of view. All image data are continuously transmitted to the cache module of the interactive system at a preset frame rate, serving as the sole input data source for subsequent anomaly detection.

[0065] On the other hand, the voice signals output by doctors through head-mounted microphones or operating console pickup modules are collected in real time throughout the entire surgical procedure. After the voice signals are processed by pre-noise reduction to filter out the operating noise of the equipment and environmental background noise during the operation, user voice command data with uniform format is generated, and key semantic information such as the operation intention and target parameters in the command are completely preserved.

[0066] Step S102: Input the surgical image data into a preset anomaly detection model to determine the anomaly perception information corresponding to the surgical image data.

[0067] For example, the real-time image data to be processed in the cache queue is pushed to the model inference module frame by frame in the order of acquisition time. After the pre-processing operations of size normalization and pixel value standardization are completed, each frame of image is completely input into the lightweight target detection model that has been trained and calibrated with full labeled dataset in advance. The model outputs the recognition results and confidence parameters of all target categories in the corresponding frame of image, and generates structured perception information covering various abnormal scenes in the operation.

[0068] In one possible embodiment, the model performs frame-by-frame detection on each frame of the surgical image, identifies abnormal targets or abnormal states in the surgical field, and outputs the corresponding detection results. These detection results are then post-processed by the model to form anomaly perception information, which may include anomaly-related characterization information. For example, the anomaly detection model can also adopt a model that performs temporal joint analysis on video segments, suppressing false detections in a single frame by introducing motion information from adjacent frames, as long as the corresponding anomaly perception information can be determined based on the surgical image data.

[0069] Optionally, the lightweight object detection model can be a lightweight version of the YOLO series (such as YOLOv8n, YOLOv8s, etc.), or it can be an object detection model with real-time inference capabilities, such as MobileNet-SSD and NanoDet, which can run resident on edge computing hardware. This application does not limit this.

[0070] In one specific embodiment, the YOLOv8n model can be used as the anomaly detection model. Specifically, the input image (i.e., surgical image data) of the YOLOv8n model is 640×640 pixels, and the output categories include bleeding areas, abnormal surgical instrument states, and foreign bodies. This YOLOv8n model is trained on a training set containing approximately 10,000 labeled surgical images using a stochastic gradient descent optimizer with an initial learning rate of 0.001 for 100 epochs. The final mean accuracy (mAP@0.5) on the validation set reaches 0.92. The training dataset includes images of different surgical procedures such as laparoscopy and single-port surgery, with annotations including target category, location coordinates, bounding box, and confidence score. Those skilled in the art can transfer-train the model according to actual surgical scenarios, obtaining an anomaly detection model suitable for specific surgical procedures without creative effort.

[0071] Step S103: Output the abnormality perception information through non-voice prompts, which include at least visual prompts.

[0072] For example, after obtaining anomaly perception information, instead of converting the anomaly content into a voice broadcast statement, the anomaly perception information is directly sent to the visual prompting unit, which then generates corresponding visual prompt content based on the anomaly perception information. Specifically, the visual prompting unit can be integrated into the surgical robot console display, endoscopic image display terminal, head-mounted display device, or other visualization output components adapted to the surgical scenario.

[0073] In addition to basic visual cues, the non-voice mode can be expanded to include other non-voice output forms such as tactile cues or graphical status bar cues. It is only necessary to ensure that the non-voice cues at least cover the visual cues. This type of non-voice cues is used to transmit abnormal perception information entirely through non-voice carriers, avoiding signal interference to the subsequent voice interaction link from the bottom layer. All cues can be implemented through visual cues such as highlighting, pop-up alarms, and color coding, directly and accurately conveying the corresponding abnormal perception information to the operating doctor.

[0074] Step S104: In response to the user's voice command, output the corresponding voice response information.

[0075] The output of anomaly perception information and the output of voice response information are carried out through independent non-voice channels and voice channels.

[0076] For example, after completing the semantic parsing and intent verification of the voice command, a standardized response content that completely corresponds to the command content is generated by matching the preset intraoperative interaction knowledge base. The response content strictly follows the professional expression norms of the surgical scenario and removes redundant embellishment information to ensure the accuracy of information transmission.

[0077] User voice commands can cover two types of scenarios: one is intraoperative information query commands, which can be used to retrieve key intraoperative information such as surgical imaging parameters, equipment operating status, and real-time patient vital signs; the other is knowledge question and answer commands, which can be used to query professional content such as surgical operation procedures, instrument parameter descriptions, and clinical treatment references.

[0078] The generated standardized response content is completely fed into the offline-deployed speech synthesis unit, and converted into speech response information with a stable tone and moderate speech rate through a low-noise speech synthesis algorithm adapted to the medical scenario.

[0079] Finally, the converted voice response information is directly pushed to the audio output link and played to the operating doctor through a medical headset, a directional speaker on the control console, or a dedicated audio output device for the surgical robot. An independent volume threshold limit is set during the audio output process to avoid excessive output volume from interfering with the surgical environment.

[0080] In this embodiment, by using a dual-channel isolation interaction mechanism that outputs intraoperative abnormality perception information independently in non-voice mode and responds to user-initiated voice commands through a dedicated voice channel, the problem of missed warnings and interference with voice command responses caused by the sharing of multiple types of information in traditional surgical robots is overcome. This significantly improves the efficiency of intraoperative abnormality warnings and the smoothness of human-computer interaction.

[0081] In one possible implementation, the specific process of outputting anomaly perception information through a non-voice prompt method may further include steps S201 and S202, which are detailed below:

[0082] Step S201: Analyze the risk level corresponding to the anomaly perception information;

[0083] Step S202: When the risk level is higher than the preset level, the abnormal perception information is output through non-voice prompts. Visual prompts include triggering indicator light flashing and / or displaying real-time surgical image data with warning labels superimposed on the graphical user interface.

[0084] For example, the anomaly perception information is output by the anomaly detection model and then enters the risk assessment module. The risk assessment module can use a rule table, threshold table or classifier to merge the anomaly features and generate risk level results.

[0085] For example, the risk assessment module calculates the risk level based on a rule table, with the following rules:

[0086] If the target category is "bleeding area" and the confidence level is ≥0.7, and the number of consecutive detection frames is ≥5, then the risk level is "high".

[0087] If the target category is "foreign object" and the confidence level is ≥0.6, and the number of consecutive detection frames is ≥3, then the risk level is "medium".

[0088] If the target category is "abnormal surgical instrument status" and the confidence level is ≥0.75, the risk level is "high".

[0089] The preset level can be set to "medium". When the risk level is "high" or "medium", a non-voice prompt will be triggered.

[0090] Different risk levels correspond to different indicator light flashing frequencies and colors. For example, red flashes rapidly for high risk and yellow flashes slowly for medium risk.

[0091] In some embodiments, if the risk level is higher than a preset risk level, a prompt control signal is sent to a non-voice output unit (e.g., a visual output unit) to drive an indicator light installed on the main control panel of the surgical robot, the frame of the display terminal, or the auxiliary prompt panel to flash at a preset frequency. The flashing frequency, on / off duty cycle, and duration can be set in association with the risk level.

[0092] Meanwhile, the graphical user interface calls up real-time surgical image data and overlays warning labels onto the image screen. These warning labels can be highlighted boxes, arrows, semi-transparent exclamation mark layers, or color-coded text, and they correspond to the spatial location of abnormal areas. The real-time surgical image data used for display can come directly from the endoscopic imaging unit, input to the display terminal via an image transmission interface, and the interface rendering module completes the overlay display without obscuring key surgical field information.

[0093] Optionally, it can also be determined whether the abnormality perception information meets the preset prompt triggering conditions. The triggering conditions may include one or more of the following: the risk level corresponding to the abnormality perception information, the confidence level corresponding to the abnormality perception information, or the abnormality category corresponding to the abnormality perception information.

[0094] For example, if the target category corresponding to the perceived information is a preset category of interest (such as bleeding or foreign body) and the confidence level is greater than a preset confidence level threshold (such as 0.6); or if the number of frames in which the target is continuously detected in multiple consecutive frames of images exceeds a preset frame number threshold (such as 5 frames), a risk warning can be triggered. This risk warning can be in the form of a visual cues.

[0095] Optionally, if the above-mentioned abnormal perception information prompt triggering conditions are met, the visual prompt unit (such as an indicator light) set on the surgical robot is controlled to output visual prompt information in a preset flashing mode to avoid interfering with the surgeon's attention during the surgical operation.

[0096] In addition, different colors (such as red and yellow) and different flashing frequencies can be set to distinguish different types of sensory information such as bleeding and foreign objects.

[0097] In this embodiment, by classifying and judging abnormal perception information, a highly recognizable non-voice prompt with flashing indicator lights and superimposed image warning signs is triggered only when the risk level exceeds a preset threshold. This solves the problem of visual information overload caused by indiscriminately pushing all abnormalities, and greatly improves the accuracy of prompting high-risk intraoperative abnormalities and the doctor's response efficiency.

[0098] In one possible implementation, the anomaly detection information includes at least one of the following:

[0099] Target categories include bleeding areas, abnormal surgical instrument conditions, and foreign bodies.

[0100] The location coordinates corresponding to the target category;

[0101] The bounding rectangle corresponding to the target category;

[0102] Confidence level corresponding to the target category;

[0103] The risk level corresponding to the target category. For example, the detected anomaly perception information output by the anomaly detection model may include the target category corresponding to the anomaly, where the target category may include bleeding areas, abnormal surgical instruments, and foreign objects, etc.

[0104] In some embodiments, the anomaly perception information may further include the location coordinates of the target category corresponding to the anomaly, the bounding rectangle corresponding to the target category, the confidence level corresponding to the target category, and the risk level corresponding to the target category.

[0105] In some embodiments, after determining the abnormal perception information of surgical image data through the abnormality detection model, the location coordinates of the abnormal region are marked by the smallest bounding rectangle that fits the edge contour of the abnormal region, and the abnormality category, the confidence level corresponding to the abnormality category, and the risk level corresponding to the abnormality category are marked in the blank area at the edge of the image or in a non-critical field of view that does not interfere with the abnormality recognition.

[0106] In some embodiments, the anomaly perception information adopts a structured array data organization method to be compatible with multi-target parallel output, which makes it easy to bind multiple anomaly objects in the same frame of image to independent identification parameters, location information and attribute fields to complete recording and hierarchical display.

[0107] In this embodiment, by decomposing the abnormal perception information into structured fields such as target category, corresponding location coordinates, bounding rectangle, and confidence level, the problem that unstructured intraoperative image abnormal information cannot be directly adapted to the visual prompt output is solved, which greatly improves the accuracy of visual prompts and the speed at which doctors can identify abnormal locations.

[0108] In one possible implementation, the specific process of responding to a user's voice command and outputting corresponding voice response information may further include steps S301 to S303, which are detailed below:

[0109] Step S301: Parse the user's voice command to obtain the query intent corresponding to the user's voice command;

[0110] Step S302: Obtain target knowledge text that matches the query intent through a preset knowledge base;

[0111] Step S303: Generate a response text that matches the target knowledge text using a preset natural language model, and perform speech synthesis on the response text to output the corresponding speech response information.

[0112] For example, after receiving the raw signal of the user's voice command output by the sound pickup module, it first undergoes pre-audio processing such as bandpass filtering and environmental noise cancellation, and then is sent to a speech recognition model that has been specially trained on medical scenario corpus to convert the continuous speech signal into structured text content that conforms to the medical professional terminology standard.

[0113] The converted text content undergoes multi-dimensional semantic parsing and intent classification operations, sequentially completing technical term matching, instruction boundary segmentation, and intent label mapping to accurately pinpoint the specific query intent corresponding to the user's voice command and filter out invalid semantic fragments that are not related to the operation.

[0114] After determining the user's query intent, the system retrieves a dedicated surgical interaction knowledge base deployed offline on-premises. Through precise matching of intent tags, it quickly locates and retrieves the target knowledge text that perfectly matches the current query intent. The entire process does not rely on external cloud data transmission links, thus avoiding the risk of data leakage.

[0115] The retrieved target knowledge text is fed into a lightweight natural language processing model adapted to the medical scenario to optimize sentence fluency, calibrate professional expressions, and adapt to colloquial language, generating a standardized response text with rigorous logic and concise expression. The response text is then fed into an offline speech synthesis unit, which converts it into a speech signal with a stable tone and moderate speech rate through a low-noise speech synthesis algorithm adapted to the medical scenario, and outputs it as speech response information.

[0116] Optionally, the target text corresponding to the user's query intent can be input into a pre-set natural language model. Then, the natural language model, combined with the contextual information of the current surgical scenario (including but not limited to the current surgical stage and abnormality perception information), performs natural language processing on the retrieved content to generate a response text that conforms to spoken expression habits and fits the current medical terminology context. The response text is then converted into a speech signal through speech synthesis and broadcast to the user (i.e., the doctor) through a speech playback unit.

[0117] In this embodiment, by performing intent parsing on user voice commands, matching corresponding target knowledge text from a dedicated knowledge base, and then generating voice responses through a natural language model, the problem of ambiguous semantic recognition and response content not meeting clinical needs in traditional surgical robot voice interaction is solved, thereby improving the accuracy of intraoperative voice command response and the smoothness of interaction.

[0118] In one possible implementation, the specific process of obtaining target knowledge text matching the query intent through a preset knowledge base may further include the following steps, detailed below:

[0119] If the query intent is a professional knowledge query, the target knowledge text that matches the surgical image data and / or the query intent will be obtained through a preset expert knowledge base;

[0120] The pre-set expert knowledge base includes at least one of the following: structured or semi-structured surgical operation procedures, complication management plans, and instrument usage guidelines.

[0121] For example, when conducting professional knowledge queries, the semantic category recognition is first performed on the query intent obtained from the user's voice command parsing. The recognized semantic tags and the synchronously collected real-time surgical image data are directly sent to the preset expert knowledge base. The image data can directly output key scene information such as the current location of the surgical field, the real-time position of the surgical instruments, the marked abnormal areas, or the current stage of the surgical operation. The query intent can accurately pinpoint the specific knowledge topic that the doctor is currently concerned about.

[0122] The system extracts high-dimensional image features from surgical image data and performs cross-modal joint matching operations with semantic features mapped from query intent. This allows for dual-constraint retrieval of scene and intent across the entire expert knowledge base. Specifically, a surgical stage recognition model is first used to classify real-time surgical image data, obtaining current surgical scene labels such as "gallbladder separation," "vascular exposure," and "suture hemostasis." Then, the scene label is combined with the semantic label corresponding to the query intent as a joint query key, which is matched against the metadata of entries in the expert knowledge base to retrieve target knowledge text relevant to both the current scene and the query intent. Each entry in the expert knowledge base is pre-labeled with applicable surgical scene and intent labels. For example, the entry "bleeding management plan" is labeled with the scene "bleeding" and the intent "hemostasis," thereby improving retrieval accuracy and precisely locating knowledge entries that perfectly match the current intraoperative real-time scene, directly outputting target knowledge text conforming to clinical professional standards. The pre-stored expert knowledge base contains various types of structured or semi-structured medical professional content, covering three core data dimensions: surgical operation standards, complication management plans, and instrument usage guidelines, allowing for targeted content retrieval based on different query scenarios.

[0123] In some embodiments, when the query intent is directed to routine surgical procedures such as hemostasis, suturing, or the handling of surgical instruments, the target knowledge text directly extracts the corresponding standardized operation segments from the pre-stored structured surgical operation specifications in the preset expert knowledge base. The content is completely consistent with the official operation guidelines for the corresponding surgical procedure.

[0124] In some embodiments, when the query intent is directed to intraoperative emergencies such as bleeding control, tissue damage management, or abnormal detachment of surgical instruments, the corresponding standardized treatment content is directly extracted from the pre-stored complication management plans in the preset expert knowledge base, providing doctors with compliant emergency operation references.

[0125] In some embodiments, when the query intent is directed to equipment usage requirements such as surgical instrument assembly, energy output parameter settings, or endoscope operation methods, the corresponding operation instructions are directly extracted from the pre-stored surgical instrument usage guidelines in the preset expert knowledge base, and the content fully matches the official operation requirements of the corresponding model of surgical robot.

[0126] In this embodiment, by targeting the intent to query professional knowledge, relying on a preset expert knowledge base that includes surgical operation specifications, complication management plans, and instrument usage guidelines, the system matches surgical image data with the query intent to output target knowledge text. This solves the problems of traditional surgical robot voice knowledge queries being detached from the real-time intraoperative scenario and returning generalized and non-targeted content, and significantly improves the scenario adaptability and clinical reference value of intraoperative professional knowledge query results.

[0127] In one possible implementation, the specific process of obtaining target knowledge text that matches the query intent through a preset knowledge base may also include the following steps, detailed below:

[0128] If the query intent is to retrieve personalized patient information, the target knowledge text matching the query intent is obtained through a pre-built localized patient database. The localized patient database contains at least one of the patient's medical records, past medical history, allergy history, and laboratory test reports.

[0129] For example, when the query intent is identified as a patient personalized information query type, the local dedicated data retrieval link is directly triggered without initiating any data transmission requests to the external network. The target knowledge text that perfectly matches the current query intent is accurately retrieved from the localized patient database that has been pre-authorized and deployed. All retrieval operations are completed within the local hardware of the surgical end.

[0130] Optionally, in some feasible embodiments, the localized patient database is generated synchronously by the hospital information system before the operation. When importing data, the patient's name, hospital number, and surgery number are associated and bound, and index fields are established for medical record information, past medical history, allergy history, and laboratory test reports.

[0131] The localized database contains structured and encrypted patient data covering multiple core clinical information categories, including at least four core types: complete electronic medical records throughout the patient's lifecycle, comprehensive records of past medical history, detailed records of drug and consumable allergies, and preoperative comprehensive laboratory and examination reports. All data has undergone anonymization and access control verification, allowing only authorized physicians to access it during surgery. Furthermore, this structure ensures that patient data retrieval and intraoperative question-and-answer responses maintain independent data sources, enabling offline access to relevant patient information and reducing reliance on external network connections, thereby improving the continuity and stability of intraoperative queries and responses.

[0132] In this embodiment, by matching the corresponding target knowledge text from a locally deployed dedicated database containing medical records, past medical history, allergy history, and laboratory test reports based on the patient's personalized information query intent, the problem of traditional intraoperative patient information query relying on cloud transmission and having the risk of data leakage is solved, which greatly improves the security of patient privacy data and the response speed of intraoperative data retrieval.

[0133] In one feasible implementation, the method for constructing the aforementioned localized patient database includes the following steps, detailed below:

[0134] Before the surgery begins, a pluggable storage medium containing patient data is attached to the surgical robot to build a localized patient database.

[0135] During the procedure, the pluggable storage medium is kept in a read-only state.

[0136] After the procedure, the pluggable storage media can be physically removed.

[0137] For example, before the surgery begins, a localized patient database can be built using a pluggable storage medium pre-mounted to the surgical robot's dedicated encrypted interface. Specifically, the pluggable storage medium contains the patient's complete clinical data, and after being mounted on the surgical robot, it automatically generates a local index without writing any original patient data to the surgical robot's public storage partition.

[0138] In some embodiments, during the operation, the pluggable storage medium remains in a read-only state, and all operation commands are unable to modify, delete or overwrite the original patient data in the medium. No new temporary data files are generated, and no original clinical fields beyond the index call range are sent back to the surgical robot. By locking the read-only access permissions of the pluggable storage medium throughout the operation, the risk of data being accidentally tampered with during the operation is eliminated from the hardware level.

[0139] In some embodiments, after the surgery, the pluggable storage media can be physically removed directly from the surgical robot's dedicated encrypted interface. After the removal operation is completed, the surgical robot only retains a temporarily generated search index cache locally, without any original sensitive patient data remaining.

[0140] In this embodiment, by constructing a localized patient database based on a preoperatively mounted pluggable read-only storage medium, locking read-only permissions throughout the operation, and supporting direct physical removal after the operation, the problem of easy leakage and tampering of traditional surgical patient data due to long-term retention is solved, and the storage security and compliance of sensitive medical data of patients during the operation are greatly improved.

[0141] Please see Figure 2 , Figure 2 A flowchart illustrating a surgical robot interaction method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, the interaction of the surgical robot includes two main branches: a detection prompting branch and a question-and-answer interaction branch. The detection prompting branch is a function that runs continuously, while the question-and-answer interaction branch is a function that is triggered on demand.

[0142] Specifically, in the detection and prompting branch, real-time surgical image data can be obtained through the endoscope used during the operation. Then, a lightweight model is used to detect the real-time surgical image data frame by frame to obtain abnormal perception information during the operation. If the abnormal perception information meets the preset triggering conditions, the corresponding visual output unit (such as an indicator light) on the surgical robot is controlled to flash in a non-voice mode according to the preset mode to provide an abnormal prompt.

[0143] In the question-and-answer interaction branch, a pluggable storage medium can be inserted before the surgery begins. The hardware then reads the patient's individualized data, segments and vectorizes this data, and constructs a localized retrieval knowledge base specific to the patient. Upon receiving the doctor's voice command, semantic parsing is performed to determine the query intent. If the query intent pertains to expert knowledge, the corresponding target knowledge text is retrieved based on the pre-set expert knowledge base. If the query intent pertains to patient data, the corresponding target knowledge text is retrieved from the localized knowledge base. Finally, the obtained target knowledge text is processed through a language model to produce the corresponding response text, which is then synthesized into speech and broadcast to the doctor.

[0144] In one feasible embodiment, before a laparoscopic surgery begins, the surgeon inserts a pluggable storage medium containing the patient's medical records, past medical history, and allergy history into the reading interface of the surgical intelligent agent hardware device. The hardware device automatically reads the patient's individualized data from the pluggable storage medium and constructs a localized, enhanced knowledge base specific to that patient. After the surgery begins, the hardware device acquires the image data collected by the endoscope in real time and inputs it into a lightweight target detection model for frame-by-frame detection.

[0145] When the surgery reaches a certain stage, the lightweight model detects a bleeding area in the real-time imaging data, and if the confidence level continuously exceeds a preset threshold, the trigger unit determines that the prompt trigger condition is met. It then controls the indicator light on the surgical robot to flash rapidly in red or another color to provide a visual prompt. This allows the surgeon to notice the prompt with their peripheral vision and decide whether to intervene. The entire process is conducted without any voice announcements and does not interfere with the surgeon's current surgical operation.

[0146] During the surgery, the doctor asked via voice, "Does this patient have a history of drug allergies?" The voice interaction module recognized and semantically analyzed the voice command, determining that the query type was a patient individual information query. It retrieved information matching "allergy history" from the localized retrieval enhancement knowledge base, and input the search results into a large language model for naturalization processing, generating the response text "The patient's past records show an allergy to penicillin-type drugs, with no other drug allergies." This text was then synthesized and broadcast to the doctor through a speaker.

[0147] In addition, doctors can ask questions related to surgical expertise, such as, "What are the general principles for handling bleeding during surgery?" If the query is identified as a question related to surgical expertise, the doctor can retrieve a treatment plan from the expert knowledge base that matches the current perceived information (degree of bleeding, anatomical location), process it using a large language model, and then broadcast it to the doctor in speech for reference and decision-making.

[0148] After the surgery, the pluggable storage medium is removed, and the patient-specific localized retrieval enhancement knowledge base constructed in this embodiment is then cleared. The hardware device is restored to its initial state without any patient data, waiting for the next surgery to insert a new pluggable storage medium for data switching.

[0149] Please see Figure 3 , Figure 3 An architecture diagram of a surgical robot system provided in this application embodiment is shown below. Figure 3As shown, the surgical robot includes a detection and prompting link 310, a question-and-answer interaction link 320, and a knowledge production and output 330. Specifically, the detection and prompting link 310 sequentially passes through endoscopic image acquisition, a lightweight target detection model, and a prompt trigger judgment unit, ultimately driving the visual prompting unit to complete the intraoperative visual prompting of flashing indicator lights. The question-and-answer interaction link 320 sends the processed content to the query type judgment unit through a voice acquisition and semantic parsing module. The query type judgment unit can retrieve the corresponding information from an expert knowledge base based on Retrieval-Augmented Generation (RAG) retrieval or from a pluggable storage medium containing patient data. Finally, the relevant content is sent to the large language model naturalization processing module for content optimization, and then handed over to the speech synthesis and image unit to output the corresponding interactive feedback. The modules are connected by a clear data flow to form a complete intraoperative intelligent auxiliary operation system.

[0150] Figure 4 This is a schematic diagram of the structure of the surgical robot interaction device provided in the embodiments of this application, as shown below. Figure 4 As shown, the surgical robot interaction device 40 provided in this embodiment includes:

[0151] The acquisition module 410 is used to acquire surgical image data and user voice commands in real time;

[0152] The perception module 420 is used to input surgical image data into a preset anomaly detection model in order to determine the anomaly perception information corresponding to the surgical image data.

[0153] The prompting module 430 is used to output abnormality perception information through non-voice prompts, and the non-voice mode includes at least visual prompts;

[0154] The response module 440 is used to respond to user voice commands and output corresponding voice response information.

[0155] The prompting module 430 and the response module 440 output information through independent non-voice channels and voice channels, respectively.

[0156] In one possible implementation, the above-mentioned prompting module 420 is further configured to determine the risk level based on the anomaly perception information;

[0157] When the risk level is higher than the preset level, abnormal perception information will be output through non-voice prompts. Visual prompts include triggering indicator light flashing and / or displaying real-time surgical image data with warning labels overlaid on the graphical user interface.

[0158] In one possible implementation, the above-mentioned anomaly detection information includes at least one of the following:

[0159] Target categories include bleeding areas, abnormal surgical instrument conditions, and foreign bodies.

[0160] The location coordinates corresponding to the target category;

[0161] The bounding rectangle corresponding to the target category;

[0162] Confidence level corresponding to the target category;

[0163] Risk level corresponding to the target category.

[0164] In one possible implementation, the response module 440 is further configured to:

[0165] Parse the user's voice command to obtain the query intent corresponding to the user's voice command;

[0166] By using a pre-defined knowledge base, target knowledge text that matches the query intent can be obtained;

[0167] The system generates response text that matches the target knowledge text using a pre-defined natural language model, and then performs speech synthesis on the response text to output the corresponding speech response information.

[0168] In one possible implementation, the response module 440 is further configured to, if the query intent is a professional knowledge query, obtain target knowledge text that matches the surgical image data and / or the query intent through a preset expert knowledge base.

[0169] The pre-set expert knowledge base includes at least one of the following: structured or semi-structured surgical operation procedures, complication management plans, and instrument usage guidelines.

[0170] In one possible implementation, the response module 440 is further configured to:

[0171] If the query intent is to retrieve personalized patient information, the target knowledge text matching the query intent is obtained through a pre-built localized patient database. The localized patient database contains at least one of the patient's medical records, past medical history, allergy history, and laboratory test reports.

[0172] In one possible implementation, the device provided in this embodiment further includes a construction module for attaching a pluggable storage medium containing patient data to the surgical robot before the start of surgery, so as to build a localized patient database.

[0173] During the procedure, the pluggable storage medium is kept in a read-only state.

[0174] After the procedure, the pluggable storage media can be physically removed.

[0175] The surgical robot interaction device provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0176] Figure 5 This is a structural schematic diagram of the surgical robot provided in this application. Figure 5 As shown, the surgical robot 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the surgical robot 50 also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0177] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described surgical robot interaction method.

[0178] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0179] In one specific implementation, the surgical robot also includes a pluggable interface 505 for connecting a pluggable storage medium for the processor 501 to read data.

[0180] In one specific implementation, the surgical robot also includes a visual output unit 506, which is used to output visual cues.

[0181] In one specific implementation, the surgical robot also includes a voice output unit 507, which is used to output voice response information.

[0182] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0183] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0184] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0185] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described surgical robot interaction method.

[0186] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, the above-described surgical robot interaction method is implemented.

[0187] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0188] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0189] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0190] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0192] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0194] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from its scope.

Claims

1. A surgical robot interaction method, characterized in that, include: Real-time acquisition of surgical image data and user voice commands; The surgical image data is input into a preset anomaly detection model to determine the anomaly perception information corresponding to the surgical image data. The abnormality perception information is output through a non-voice prompt method, which includes at least a visual prompt method. In response to user voice commands, output corresponding voice response information; The output of the anomaly perception information and the output of the voice response information are performed through independent non-voice channels and voice channels.

2. The method according to claim 1, characterized in that, The step of outputting the anomaly perception information through a non-voice prompt method includes: The risk level is determined based on the anomaly detection information; When the risk level is higher than the preset level, the abnormal perception information is output through a non-voice prompt. The visual prompt includes triggering indicator light flashing and / or displaying real-time surgical image data with warning labels superimposed on the graphical user interface.

3. The method according to claim 2, characterized in that, The anomaly detection information includes at least one of the following: Target categories, which include bleeding areas, abnormal surgical instrument conditions, and foreign bodies; The location coordinates corresponding to the target category; The bounding rectangle corresponding to the target category; The confidence level corresponding to the target category; The risk level corresponding to the target category.

4. The method according to any one of claims 1 to 3, characterized in that, The response to the user's voice command, outputting corresponding voice response information, includes: Parse the user's voice command to obtain the query intent corresponding to the user's voice command; By using a pre-set knowledge base, target knowledge text that matches the query intent is obtained; A response text matching the target knowledge text is generated using a preset natural language model, and the response text is then synthesized to output the corresponding speech response information.

5. The method according to claim 4, characterized in that, The step of obtaining target knowledge text that matches the query intent through a preset knowledge base includes: If the query intent is a professional knowledge query, then the target knowledge text that matches the surgical image data and / or the query intent is obtained through a preset expert knowledge base; The preset expert knowledge base includes at least one of the following: structured or semi-structured surgical operation guidelines, complication management plans, and instrument usage guidelines.

6. The method according to claim 4, characterized in that, The step of obtaining target knowledge text that matches the query intent through a preset knowledge base includes: If the query intent is a personalized patient information query, then a target knowledge text matching the query intent is obtained through a pre-built localized patient database. The localized patient database contains at least one of the patient's medical records, past medical history, allergy history, and laboratory test reports.

7. The method according to claim 6, characterized in that, The method for constructing the localized patient database includes: Before the surgery begins, a pluggable storage medium containing patient data is attached to the surgical robot to build the localized patient database. During the procedure, the pluggable storage medium is kept in a read-only state. After the procedure, the pluggable storage medium can be physically removed.

8. A surgical robot interaction device, characterized in that, include: The acquisition module is used to acquire surgical image data and user voice commands in real time; The perception module is used to input the surgical image data into a preset anomaly detection model in order to determine the anomaly perception information corresponding to the surgical image data. The prompting module is used to output the abnormality perception information through a non-voice prompting method, wherein the non-voice prompting mode includes at least a visual prompting. The response module is used to respond to user voice commands and output corresponding voice response information; The prompting module and the response module output information through independent non-voice channels and voice channels, respectively.

9. A surgical robot, characterized in that, include: Pluggable interface, memory, processor, visual prompting unit, and voice output unit; The pluggable interface is used to connect a pluggable storage medium, which is configured to remain read-only during the operation and can be physically removed after the operation for the processor to read data. The visual prompting unit is used to output visual prompts, and the voice output unit is used to output voice response information; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images