AR-based surgical robot on-site troubleshooting system and method, storage medium

Abstract

This invention provides an AR-based on-site troubleshooting system and method for surgical robots, along with a storage medium. It can establish virtual troubleshooting operation models corresponding one-to-one with various fault causes based on historical data. Through video analysis results and fault codes, it deduces the current fault cause and finds the corresponding virtual troubleshooting operation model. This model can then be overlaid onto the surgical robot using an AR device, merging the actual troubleshooting steps with the pre-planned virtual 3D model of the troubleshooting path. This provides real-time and effective guidance for personnel to troubleshoot, shortening troubleshooting time, improving troubleshooting efficiency, reducing the error rate, and shortening surgical time.

Description

Technical Field

[0001] This invention relates to the field of medical device troubleshooting technology, and in particular to an AR-based on-site troubleshooting system and method for surgical robots, as well as a storage medium. Background Technology

[0002] In recent years, surgical robot systems have become powerful tools to help doctors complete surgeries (such as minimally invasive surgeries) due to their clear stereoscopic visual feedback and flexible control, and have been widely used in the field of medical surgery.

[0003] Please refer to Figure 1 Current surgical robot systems typically include a surgeon's console 10, an imaging cart 20, and a patient-side operating table 30. The patient-side operating table 30 usually contains multiple robotic arms (not labeled), including a visual robotic arm for mounting or connecting an endoscope and at least one surgical robotic arm for mounting or connecting surgical instruments. Before surgery, the surgical robot system performs a self-test to determine its initial position and status. After the manual installation of the tamper (not shown), surgical instruments (not shown), endoscope (not shown), and sterile bag (not shown), it can enter the master-slave surgical operation state. The surgeon can input control commands from the surgeon's console 10 to control the patient-side operating table 30 to drive the robotic arms to perform surgery on the patient. Improper placement of instruments before and during surgery, master-slave operation, or unexpected conditions encountered by the machine itself can all lead to on-site malfunctions. To ensure the efficiency and safety of the surgery, these malfunctions need to be promptly and accurately resolved.

[0004] However, current on-site troubleshooting solutions for surgical robots have the following technical problems:

[0005] 1. The surgical robot system has its own safety fault alarm and troubleshooting instructions. Therefore, when a fault occurs, the surgical robot system will display a fault code on the display device, and the personnel on duty need to troubleshoot according to the fault code. However, due to the large number and complexity of fault codes in the surgical robot system, troubleshooting personnel cannot quickly locate and eliminate the fault in the surgical robot. 2. The troubleshooting methods and procedures are highly dependent on the skills and subjective judgment of professional personnel. Skills training is time-consuming and costly.

[0006] 3. For doctors lacking proficiency and for resolving unexpected problems, on-site staff need to troubleshoot the issues immediately and accurately. This results in high labor costs, and the on-site staff rely on subjective judgment and experience to troubleshoot, leading to varying troubleshooting efficiencies between individuals.

[0007] 4. The troubleshooting process cannot be monitored or responded to, which results in the inability to accurately and efficiently resolve the fault. Summary of the Invention

[0008] The purpose of this invention is to provide an AR-based surgical robot on-site troubleshooting system and method, and a readable storage medium, which can solve at least some of the technical problems existing in the prior art.

[0009] To achieve the above objectives, the present invention provides an AR-based on-site troubleshooting system for surgical robots, comprising:

[0010] The fault handling system is used to establish virtual fault troubleshooting operation models corresponding to various causes of surgical robot failures based on historical data.

[0011] A video analysis system is used to collect and analyze the operation videos of the surgical robot before and during the operation to obtain the first behavioral data before the failure occurs;

[0012] A data interaction system is used to interact with the video analysis system and the motion control system of the surgical robot to obtain the status data of the surgical robot in real time.

[0013] A fault analysis system is used to analyze the first behavioral data and the state data in order to find a virtual fault troubleshooting operation model corresponding to the current fault cause from the fault handling system based on the analysis results.

[0014] AR devices are used to overlay the virtual fault-solving operation model determined by the fault analysis system onto the surgical robot through coordinate mapping, thereby guiding relevant personnel to perform fault-solving operations.

[0015] Optionally, the fault analysis system is further configured to send each step of the fault troubleshooting operation in the virtual fault troubleshooting operation model to the AR device in a step-by-step manner; the video analysis system is further configured to collect and analyze the fault troubleshooting operation video of the relevant personnel at the current step to obtain the second line of data, and determine whether the fault troubleshooting operation of the relevant personnel at the current step is valid, and only when it is determined to be valid will the fault analysis system send the next fault troubleshooting operation in the virtual fault troubleshooting operation model to the AR device.

[0016] Optionally, the fault analysis system is further configured to, when the video analysis system determines that the fault troubleshooting operation of the relevant personnel in the current step is invalid, cause the AR device to guide the relevant personnel to repeat the fault troubleshooting operation of the current step, or to perform comprehensive analysis on the second behavior data, the first behavior data and the status data to obtain the current fault cause and the corresponding virtual fault troubleshooting operation model again.

[0017] Optionally, the surgical robot includes a patient-side operating table and a doctor-side control console. The video analysis system includes a first camera device and a second camera device. The first camera device is used to capture operation videos and the status of each component of the patient-side operating table within the area of ​​the patient-side operating table. The second camera device is used to capture operation videos and the status of each component of the doctor-side control console within the area of ​​the doctor-side control console.

[0018] Optionally, the AR device has a binocular vision module used to establish the coordinate mapping.

[0019] Optionally, the video analysis system or the fault handling system is further configured to: classify historical operation videos according to different fault causes to obtain corresponding classified videos, and model single-frame and multi-frame images of each classified video to obtain feature models and video models of each fault cause;

[0020] The video analysis system is also used to capture the operation video of the surgical robot in real time, and, starting from the time of the failure, use the video model of the current failure cause to extract the operation video captured before the failure occurred, and use the feature model of the current failure cause to perform behavioral data analysis on the extracted operation video to obtain the first behavioral data corresponding to the current failure cause.

[0021] Optionally, when the surgical robot malfunctions, it automatically generates a fault code corresponding to the current cause of the malfunction, and the fault analysis system obtains the virtual fault troubleshooting operation model based on the fault code, the first behavior data, and the status data; or, the AR device has a trigger button for manually triggering the fault analysis system when the surgical robot malfunctions, so that the fault analysis system analyzes and obtains a fault code corresponding to the current cause of the malfunction.

[0022] Optionally, the surgical robot includes a foot pedal, a robotic arm, an endoscope, a power unit, a trocar, and surgical instruments. The robotic arm includes a master hand on the doctor's control console and a slave hand on the patient's operating table. The causes of failure include at least one of the following: surgical posture mismatch, master-slave posture mismatch, endoscope malfunction causing fixed field of view, no energy output when the foot pedal is stepped on, power unit failing to return to zero, trocar detachment, extension and retraction joint of the robotic arm reaching its limit position, and instruments failing to release.

[0023] Based on the same inventive concept, the present invention also provides a method for troubleshooting surgical robots on-site, comprising:

[0024] Based on historical data, virtual troubleshooting operation models are pre-established to correspond to various causes of surgical robot failures.

[0025] Real-time acquisition and analysis of the surgical robot's operation videos before and during surgery to obtain the first behavioral data before the malfunction occurs;

[0026] Real-time acquisition of the status data of the surgical robot;

[0027] The first behavioral data and the state data are analyzed to identify the virtual fault troubleshooting operation model that causes the current fault from all the pre-established virtual fault troubleshooting operation models based on the analysis results.

[0028] By using coordinate mapping on AR devices, the identified virtual troubleshooting operation model is superimposed onto the surgical robot through virtual-real fusion to guide relevant personnel in troubleshooting operations.

[0029] Optionally, the identified virtual troubleshooting operation model is superimposed onto the surgical robot through virtual-real fusion step by step. At the same time, the video of the relevant personnel's current troubleshooting operation is collected and analyzed to obtain the second line of data. It is then determined whether the relevant personnel's current troubleshooting operation is effective. Only when it is determined to be effective is the next troubleshooting operation in the virtual troubleshooting operation model sent to the AR device.

[0030] Optionally, the on-site troubleshooting method for the surgical robot further includes:

[0031] When it is determined that the troubleshooting operation of the relevant personnel in the current step is invalid, the AR device guides the relevant personnel to repeat the troubleshooting operation of the current step, or the second behavior data, the first behavior data and the status data are comprehensively analyzed to obtain the corresponding current fault cause and the corresponding virtual fault troubleshooting operation model again.

[0032] Optionally, the on-site troubleshooting method for the surgical robot further includes:

[0033] Historical operation videos are classified according to different fault causes to obtain corresponding classified videos. Single-frame and multi-frame images of each classified video are modeled to obtain feature models and video models for each fault cause.

[0034] While acquiring real-time video of the surgical robot's operation, starting from the moment the fault occurs, the video model of the current fault cause is used to extract the operation video acquired before the fault occurred, and the feature model of the current fault cause is used to perform behavioral data analysis on the extracted operation video to obtain the first behavioral data corresponding to the current fault cause.

[0035] Based on the same inventive concept, the present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the surgical robot on-site troubleshooting method described in the present invention.

[0036] Compared with the prior art, the technical solution of the present invention has at least the following beneficial effects:

[0037] 1. The technical solution of this invention can establish virtual fault-solving operation models corresponding to various fault causes in surgical robots based on historical data. After real-time acquisition and analysis of the surgical robot's pre-operative and intra-operative operation videos and status data, the obtained status data and the analyzed behavioral data of the surgical robot before the fault occurred can be integrated and analyzed. From all the established virtual fault-solving operation models, the virtual fault-solving operation model corresponding to the current fault cause can be found. Then, through the virtual-real fusion of AR devices, the found virtual fault-solving operation model can be superimposed onto the surgical robot to guide relevant personnel in fault-solving operations. This fault-solving method can quickly locate and resolve faults in the surgical robot with the assistance of AR devices, without relying on the skills and subjective judgment of professional personnel, thus reducing the cost of fault-solving and improving the accuracy of fault-solving.

[0038] 2. The found virtual troubleshooting operation model can be superimposed onto the surgical robot step by step. This allows for real-time video acquisition and analysis of the current operations of the personnel involved in troubleshooting, and real-time judgment of the validity (i.e., correctness) of the current operations. This ensures the accuracy of the troubleshooting operation steps, saves troubleshooting time, and increases the safety of the surgery.

[0039] 3. When it is determined that the current operation of the relevant personnel in troubleshooting is invalid, the cause of the fault and / or the virtual fault troubleshooting operation model are further analyzed by combining the current operation data of the relevant personnel (i.e., the second action data) with the obtained first action data and status data. This can avoid the problem of new faults that cannot be eliminated due to improper operation of relevant personnel during the troubleshooting process, further reduce the fault troubleshooting error rate and shorten the operation time. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an existing surgical robot system.

[0041] Figure 2 This is a schematic diagram of the architecture design of the on-site troubleshooting system for a surgical robot according to a specific embodiment of the present invention.

[0042] Figure 3This is a schematic diagram of the layout of the surgical robot on-site troubleshooting system and the surgical robot according to a specific embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram illustrating the behavior code classification of a surgical robot according to a specific embodiment of the present invention.

[0044] Figure 5 This is a flowchart of a video analysis method according to a specific embodiment of the present invention.

[0045] Figure 6 This is a schematic diagram of a video analysis method for the main hand joint according to a specific embodiment of the present invention.

[0046] Figure 7 This is a schematic diagram of a video analysis method for the zero position of the power box according to a specific embodiment of the present invention.

[0047] Figure 8 This is a flowchart illustrating a specific embodiment of the surgical robot troubleshooting method.

[0048] Figure 9 This is an example illustrating the causes (i.e., types of faults) that the surgical robot on-site fault troubleshooting system of a specific embodiment of the present invention can eliminate.

[0049] Figure 10 This is a schematic diagram of the AR device in the surgical robot on-site troubleshooting system according to a specific embodiment of the present invention.

[0050] Figure 11 This is a schematic diagram of the troubleshooting process for a device with no energy output, according to a specific embodiment of the present invention. Detailed Implementation

[0051] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described to avoid confusion with the invention. It should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, the provision of these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art. The same reference numerals denote the same elements throughout. It should be understood that when an element is referred to as "connected" or "coupled" to other elements, it may be directly connected to other elements, or there may be intervening elements. Conversely, when an element is referred to as "directly connected" to other elements, there are no intervening elements. Although the terms "left side," "right side," etc., may be used to describe various elements, components, and / or portions, these elements, components, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, or portion from another element, component, or portion. Therefore, without departing from the teachings of this invention, the “left-side” element, component, or portion discussed below may be referred to as a “right-side” element, component, or portion. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “comprising” is used to identify the presence of features, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0052] The technical solution proposed by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0053] Please refer to Figure 2 and Figure 3 One embodiment of the present invention provides an AR-based on-site troubleshooting system for surgical robots, which is used to troubleshoot malfunctions of the corresponding surgical robot (or surgical robot system). The surgical robot can be any existing surgical robot system.

[0054] As an example, the surgical robot includes a doctor's console 10, an image carriage 20, a patient's operating table 30, auxiliary equipment 40, and a hospital bed 50. The doctor's console 10, the image carriage 20, and the patient's operating table 30 can communicate with each other using any suitable method (e.g., wireless via a local area network or wired via a data cable).

[0055] The hospital bed 50 is positioned in a suitable location near the operating table 30 at the patient end to facilitate sitting, lying down, or reclining for the patient. Auxiliary equipment 40 is also positioned near the operating table 30 at the patient end and may include, for example, a sterile table (where surgical instruments can be placed), a ventilator, an anesthesia machine, an extracorporeal circulation machine, or some vital sign monitoring devices.

[0056] The patient-side operating table 30 (also known as the slave trolley) is used for performing related surgeries on patients. It includes a power unit (not shown), surgical instruments 32, an endoscope, a trocar, and at least one slave hand 31. The slave hand 31, also known as a tool arm, adjustment arm, slave robotic arm, slave operator hand, etc., is used to mount or connect the surgical instruments 32, the trocar (not shown), and the endoscope (not shown). One end of each slave hand 31 can be connected to the same passive adjustment arm, which enables the slave hand to move up, down, and translate. Each slave hand has multiple joints, allowing for extension, retraction, and rotation. The endoscope is used to acquire images of the surgical environment, including human tissues and organs, surgical instruments 32, blood vessels, and body fluids. The surgical instruments 32 may include scalpels, forceps, trocars, and suture needles, and can perform various functions, including clamping, resection, cutting, suturing, and anastomosis. The trocar is usually fixed to the slave hand, and the surgical instruments 32 are mounted on the trocar, facilitating the installation and replacement of the surgical instruments 32. The power box (which can be a motor) is the power source for the operating table at the patient end. It can drive the patient's hand to make corresponding movements according to the instructions of the doctor's control panel. In order to ensure the accuracy of the use of the surgical robot, the power box usually needs to be set to "zero position" and needs to automatically return to "zero position".

[0057] The doctor's console 10 is used to control the patient's operating table 30, drive the hand 31, and operate the surgical instruments 32 to perform the corresponding surgery on the patient under the operation of the surgeon. The doctor's console 10 mainly has four functional modules, from top to bottom: 1) Stereoscopic image observation window (unlabeled), which provides the surgeon with high-definition stereoscopic images during surgery for observing lesions and surgical instruments 32 at the patient-side operating table 30; 2) Master hand 11, which is the surgical action command unit and can be operated with one or two hands. During surgery, the surgeon holds the end of the master hand 11 and sends the desired movement position and posture (clamping) command to the slave hand 31. There is a master-slave mapping relationship between the surgeon's operation on the master hand 11 on the doctor's console 10 and the operation produced by the slave hand 31 according to the command issued by the doctor's console 10; 3) Control panel (unlabeled), which is used to adjust some preoperative surgical robot parameters, such as endoscope angle, zoom ratio, etc.; 4) Foot pedals 12, usually multiple, used to adjust commonly used surgical actions during surgery, disconnect master-slave motion mapping, endoscope motion control, start the patient-side operating table 30, etc.

[0058] Before surgery, the machine self-checks to determine its initial position and status. After installing the trocar, surgical instruments 32, endoscope, and sterile bag, it can enter the master-slave surgical operation state. During the operation, the surgeon can frequently operate the master hand 11 to control the slave hand 31, thereby achieving the master-slave surgical effect. The surgeon can also control the master-slave operation, endoscope control, slave hand switching, and electrosurgical energy control by releasing and opening the foot pedal 12. Obviously, improper positioning before and during surgery, master-slave operation, or unexpected situations encountered by the surgical robot itself (including surgical posture mismatch, master-slave posture mismatch, endoscope malfunction causing fixed field of view, no energy output when stepping on the energy instrument pedal, power box unable to return to zero, trocar detachment, extension joint reaching its limit position, instruments unable to be released, etc.) can all lead to on-site malfunctions of the surgical robot. One of the design objectives of the AR-based on-site fault diagnosis system for surgical robots in this embodiment is to ensure that on-site personnel can troubleshoot malfunctions in a timely and accurate manner, reduce the experience requirements for troubleshooting personnel, and ultimately ensure the efficiency and safety of the surgery.

[0059] Specifically, the AR-based on-site troubleshooting system for surgical robots in this embodiment includes an AR device 60, a video analysis system 70, a fault handling system 80, a data interaction system 81, and a fault analysis system 82.

[0060] The fault handling system 80 is used to establish virtual fault-solving operation models that correspond one-to-one with various fault causes (or multiple fault types) of the surgical robot based on historical data. Thus, once a fault occurs and the specific fault cause is analyzed, the corresponding virtual fault-solving operation model can be found and superimposed onto the physical surgical robot using the AR device 60 through virtual-real fusion, guiding the personnel on duty to handle the fault immediately and correctly. The fault handling system 80 may include a data storage unit (not shown) and a fault model generation unit (not shown). The fault model generation unit is used to analyze historical data and establish corresponding virtual fault-solving operation models based on the analysis results. The data storage unit can capture, update, and store historical data (including clinical surgical operation data, machine parameters, and fault data, etc.), and can also store the various fault causes, various virtual fault-solving operation models, and the mapping relationship between fault causes and virtual fault-solving operation models analyzed by the fault model generation unit.

[0061] The video analysis system 70 is used to collect and analyze the operation videos of the surgical robot before, during, and even after surgery to obtain the initial behavioral data before a malfunction occurs. The video analysis system 70 may include a first camera device 71, a second camera device 72, and a video analysis device 73. The first camera device 71 is positioned near the doctor's control console 10 to collect video of the doctor's operation of the main hand 11, control panel, and foot pedal 12, as well as the status of each component. The second camera device 72 is positioned near (e.g., above) the patient's operating table 30 to collect video of the operation of the area on the patient's operating table 30, as well as the status of each component, including videos of the hands and other components operating according to instructions from the doctor's control console 10, and videos of personnel troubleshooting various components on the patient's operating table 30. The video analysis device 73 may be any suitable device or module, such as a data processor, and can analyze the videos collected by the first camera device 71 and the second camera device 72 to obtain corresponding behavioral data, such as extracting video from a preset time period before the malfunction occurs to analyze whether the operation steps on the patient's operating table 30 are correct and effective.

[0062] Alternatively, please refer to Figure 4 In this embodiment, the video analysis system 70 can obtain corresponding behavior codes after analyzing the operation video, such as... Figure 4 As shown in the second column of the table, in these behavior codes, the first digit represents the corresponding module in the surgical robot, the second digit represents the joint, and the last three digits represent the behavior type.

[0063] Further optional, please refer to Figure 5 The video analytics system 70 is also used to perform the following steps:

[0064] S1. Classify historical operation videos according to different fault causes to obtain corresponding classified videos, and model single-frame and multi-frame images of each classified video to obtain feature models and video models of each fault cause.

[0065] S2, while acquiring the operation video of the surgical robot in real time, takes the time of the fault occurrence as the starting point and uses the video model of the current fault cause to extract the operation video acquired before the fault occurred. In other words, the acquired operation video is segmented.

[0066] S3, use the feature model of the current fault cause to perform behavioral data analysis on the extracted operation video to obtain the first behavioral data corresponding to the current fault cause.

[0067] As an example, the behavior of the surgical instruments 32 on the surgical robot in the time period prior to the malfunction is a necessary condition for the current malfunction to occur. Therefore, analyzing and recognizing the operation videos of the master and slave hands can more accurately diagnose the cause of the malfunction (i.e., the cause of the malfunction). For details, please refer to... Figure 5 and Figure 6 In the process of analyzing and recognizing the operation video of the main hand, the video analysis system 70, during the execution of step S1, classifies the historical operation video according to different fault causes to obtain the corresponding main hand classification video. For the main hand classification video, it extracts single-frame images of the main hand for modeling, generating a main hand feature model T1. It also extracts the number of frames (i.e., multi-frame images) and node changes [D] of frequently shaking the main hand's posture joints for modeling, generating a main hand video model V1. During step S2, for the real-time acquired operation video of the main hand's 1-7 joints, taking the fault occurrence time as the starting point, it extracts video data from the operation video of the main hand's 1-7 joints based on the duration / frame count of the main hand video model V1, capturing the video data for a period before the fault occurrence. During step S3, based on the main hand feature model T1, it extracts the position coordinates of 7 joint points from the captured video, assuming the joint coordinates of the i-th joint are P1. i =(X i ,Y i Z i ), i = 1~7, calculate the change in distance between key points in two adjacent frames, accumulate the distance changes, and denot them as [D i ]. When [D]–[D i When the distance difference is less than or equal to ε, the corresponding behavior code (e.g., 216XX) is determined and sent to the fault analysis system 82. Here, P1 is the coordinate of the first frame joint point of the i-th joint, P1' is the coordinate of the second frame joint point of the i-th joint, and ε is the threshold used to determine whether the distance difference is within the required range.

[0068] In this example, the video analysis system 70 can use the process described above to analyze and recognize the operation video of the master hand, and to analyze and recognize the operation video of each slave hand.

[0069] As another example, analyzing and recognizing the operation video of the power box at zero position can also more accurately diagnose the cause of the fault (i.e., the cause of the fault). For details, please refer to... Figure 5 and Figure 7 The polar coordinates of the four isolation plate joints of the power box are predefined as P2. i (r,θ i ), (i = 1, 2, 3, 4), where:

[0070]

[0071] Therefore, during the analysis and behavior recognition of the operation video of the power box zero position, the video analysis system 70, when performing step S1, classifies the historical operation video according to different fault causes to obtain the corresponding power box classification video. For the power box classification video, it extracts a single-frame image of the power box (the image includes four isolation plate joints) for modeling, generating a power box feature model T2. It also extracts the number of image frames and the cumulative value of polar coordinate changes [θ] during the power box self-test, generating a power box video model V2. During step S2, for the real-time acquired power box isolation plate joint video, taking the fault occurrence time as the starting point, it extracts video data from a period before the fault occurs based on the power box video model V2. During step S3, based on the power box feature model T2, it extracts the polar coordinates of the four isolation plate joints of the power box from the extracted video, and accumulates the angle changes, denoted as [θ]. i ]. When [θ]–[θ i When θ ≤ ε, the self-test is considered complete. The polar coordinate θ value of the last frame image after the self-test is completed is then checked. If it is 0, the self-test is successful; otherwise, a behavior code (e.g., 116XX) is determined and sent to the fault analysis system. Here, P2 is the first frame joint polar coordinate of the i-th isolation plate joint, P2' is the second frame joint polar coordinate of the i-th isolation plate joint, and ε is the threshold value used to determine if the angle difference is within this range.

[0072] As another example, analyzing and recognizing the behavior of pedal operation videos can also more accurately diagnose the cause of malfunctions (i.e., the cause of the malfunction). For details, please refer to... Figure 5 and Figure 8In the process of analyzing and recognizing the operation video of the pedal, the video analysis system 70, during step S1, classifies the historical operation video according to different causes of failure to obtain corresponding pedal classification videos. For each pedal classification video, it extracts a single-frame image of the pedal and models it to generate a pedal feature model T3. It also extracts multiple-frame images of the pedal and changes in pedal travel to generate a pedal video model V3. During step S2, it extracts video data from the real-time acquired pedal operation video, taking the time of failure as the starting point, based on the pedal video model V3, within a certain period before the failure. During step S3, it compares and analyzes the extracted video data based on the pedal feature model T3.

[0073] Please continue to refer to this. Figure 2 and Figure 3 The data interaction system 81 is used to interact with the video analysis system 70 and the motion control system of the surgical robot (not shown) to obtain the status data of the surgical robot in real time. The motion control system of the surgical robot is mainly used to control the slave hand to move according to the operation of the master hand.

[0074] Please continue to refer to this. Figure 2 and Figure 3 The fault analysis system 82 integrates and analyzes the behavioral data obtained from the video analysis system 70 and the status data obtained from the data interaction system to determine the cause of the current fault. Based on the determined cause, it then finds the corresponding virtual fault-solving operation model from the fault handling system 80. Please refer to... Figure 9 The current causes of failure include at least one of the following: surgical posture mismatch, master-slave posture mismatch, endoscope malfunction causing fixed field of view, no energy output when stepping on the pedal, power box unable to return to zero position, puncture card loosening, extension joint of robotic arm reaching its limit position, and instrument unable to be released.

[0075] As an example, please refer to Figure 9 Based on the video analysis results of the video analysis system 70 ("frequent shaking of the master hand's posture joints") and the corresponding status data in the data interaction system 81, the fault analysis system 82 determines the cause of the fault as "master hand posture mismatch." Furthermore, based on the video analysis results of the video analysis system 70 ("frequent shaking of the master hand's posture joints") and the determined cause of the fault ("master hand posture mismatch"), it determines the corresponding virtual fault-solving model (i.e., fault-solving method) as "re-inserting the surgical instruments." After this virtual fault-solving model is superimposed onto the actual surgical robot via the AR device 60, relevant personnel can view relevant guidance prompts regarding "re-inserting the surgical instruments" through the AR device 60.

[0076] Please continue to refer to this. Figure 2 and Figure 3 The AR device 60 is used to overlay the virtual fault-solving operation model determined by the fault analysis system 82 onto the surgical robot through coordinate mapping and virtual-real fusion, so as to guide relevant personnel in performing fault-solving operations. The AR device 60 can be AR glasses, AR helmets, or other similar devices.

[0077] Please refer to Figure 10 As an example of this embodiment, the AR device 60 is an AR glasses device with a binocular vision module 61 and a trigger button 62. The binocular vision module 61 is used to realize coordinate mapping, and the trigger button 62 is used to manually trigger a fault-solving indicator. The coordinate system (X5, Y5, Z5) of the binocular vision module 61 of the AR glasses can establish a coordinate mapping relationship with the coordinate system (X3, Y3, Z3) of the center of the frame of the AR glasses through mechanical positioning. The center of the frame of the AR glasses has corresponding coordinates in the world coordinate system (X0, Y0, Z0). Therefore, a corresponding coordinate mapping relationship can be established between the coordinate system (X5, Y5, Z5) of the binocular vision module 61 of the AR glasses and the world coordinate system (X0, Y0, Z0). Further combining... Figure 3 The coordinate system (X5, Y5, Z5) of the AR glasses is mapped to the coordinate system (X2, Y2, Z2) of the operating table 30 at the patient end in the world coordinate system (X0, Y0, Z0). The coordinate system (X5, Y5, Z5) of the AR glasses is also mapped to the coordinate system (X4, Y4, Z4) of the doctor's control console 10 at the world coordinate system (X0, Y0, Z0) (this coordinate mapping relationship is known). Thus, the coordinate system (X2, Y2, Z2) at the patient end and the coordinate system (X4, Y4, Z4) at the doctor's control console 10 can be mapped.

[0078] Optionally, when the surgical robot malfunctions, after manually triggering the trigger button 62 of the AR device 60, the fault analysis system 82 can also obtain the fault code corresponding to the cause of the current malfunction through analysis.

[0079] Of course, in other embodiments of the present invention, when the surgical robot itself has a fault alarm and prompt function, the trigger button 62 on the AR glasses can also be omitted. When the surgical robot malfunctions, it automatically generates a fault code corresponding to the current fault cause. Thus, the fault analysis system 82 determines the current fault cause based on the fault code and obtains the virtual fault troubleshooting operation model corresponding to the current fault cause based on the first behavior data of the video analysis system 70 and the status data of the data interaction system 81.

[0080] Optionally, the fault analysis system 82 is also used to distribute each step of the fault-solving operation in its determined virtual fault-solving operation model to the AR device 60, so that relevant personnel can perform fault-solving step by step. This allows relevant personnel to quickly locate and resolve a fault in the surgical robot, which helps to ensure the accuracy of each step of fault-solving. Moreover, in cases where fault-solving is relatively complex, it can reduce costs, shorten time, and improve accuracy without relying on the skills and subjective judgment of professional personnel.

[0081] In other embodiments of the present invention, based on the fault analysis system 82 sending each step of the fault troubleshooting operation in its determined virtual fault troubleshooting operation model to the AR device 60 step by step, the video analysis system 70 is further used to collect and analyze video of relevant personnel performing the current step of the fault troubleshooting operation to obtain second action data, and to determine whether the current step of the fault troubleshooting operation by the relevant personnel is valid. Only when it is determined to be valid will the fault analysis system send the next step of the fault troubleshooting operation in the virtual fault troubleshooting operation model to the AR device 60. When the video analysis system 70 determines that the current step of the fault troubleshooting operation by the relevant personnel is invalid, the fault analysis system 82 can enable the AR device 60 to guide the relevant personnel to repeat the current step of the fault troubleshooting operation until the fault troubleshooting operation is correctly executed. Alternatively, the fault analysis system 82 can again perform comprehensive analysis on the second action data, the first action data, and the status data obtained by the data interaction system 81 obtained by the video analysis system 70 to obtain the current fault cause and the corresponding virtual fault troubleshooting operation model again, that is, update the virtual fault troubleshooting operation model.

[0082] In other words, in this embodiment, the behavioral data obtained by the video analysis system 70 includes first behavioral data and second behavioral data. The first behavioral data is the behavioral data of the surgical robot before the malfunction occurs, which includes the behavioral data generated by the surgical robot before and during the operation to perform the surgery. The second behavioral data is the behavioral data during the troubleshooting process, which includes the operational behavior data of relevant personnel when troubleshooting the surgical robot and the behavioral data of the surgical robot.

[0083] This embodiment enables real-time and accurate troubleshooting guidance for relevant personnel, and provides real-time data feedback on the troubleshooting process, ensuring the accuracy of troubleshooting steps, saving troubleshooting time, increasing surgical safety, and reducing costs and improving accuracy when troubleshooting is complex, without relying on the skills and subjective judgment of professional personnel.

[0084] It is understood that the AR device 60, video analysis system 70, fault handling system 80, data interaction system 81, and fault analysis system 82 can be implemented in a single module or device; or, any one of the AR device 60, video analysis system 70, fault handling system 80, data interaction system 81, and fault analysis system 82 can be split into multiple modules or devices; or, at least some of the functions of one or more of the AR device 60, video analysis system 70, fault handling system 80, data interaction system 81, and fault analysis system 82 can be combined with at least some of the functions of one or more other devices and implemented in a single module or device. According to embodiments of the present invention, at least one of the video analysis system 70, fault handling system 80, data interaction system 81, and fault analysis system 82 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in hardware or firmware, or in a suitable combination of software, hardware, and firmware implementations. Alternatively, at least one of the video analysis system 70, fault handling system 80, data interaction system 81, and fault analysis system 82 may be implemented at least partially as a computer program module, which can perform the functions of the corresponding module when the program is run by a computer.

[0085] Please refer to Figure 2 and Figure 11 An embodiment of the present invention also provides a method for troubleshooting surgical robots on-site, which can be implemented using the surgical robot on-site troubleshooting system of the present invention. The method includes:

[0086] A1, the fault handling system 80 pre-establishes virtual fault troubleshooting operation models corresponding to various fault causes of the surgical robot based on historical data;

[0087] A2, the video analysis system 70 collects and analyzes the operation videos of the surgical robot before and during the operation in real time to obtain the first behavioral data of the surgical robot before the fault occurs. At the same time, the data interaction system 81 interacts with the video analysis system 70 and the motion control system of the surgical robot to obtain the status data of the surgical robot in real time. Among them, step A3 can be further executed: the data interaction system 81 or the video analysis system 70 or a person determines whether the status data indicates a fault in the motion control system of the surgical robot. If not, step A4 is executed, and the trigger button 62 of the AR device 60 is manually clicked to trigger the fault analysis system 82. If yes, step A5 is executed, and the surgical robot automatically triggers a fault code to trigger the fault analysis system 82.

[0088] A6, the fault analysis system 82 analyzes fault codes, first line data and status data to determine the current fault cause based on the analysis results, and then finds the virtual fault troubleshooting operation model of the current fault cause from all the pre-established virtual fault troubleshooting operation models.

[0089] A7 uses the coordinate mapping of AR device 60 to overlay the identified virtual troubleshooting operation model onto the surgical robot through virtual-real fusion, in order to guide relevant personnel in troubleshooting operations.

[0090] Optionally, in step A6, the fault analysis system 82 sends the identified virtual fault-solving operation model to the AR device 60 step by step, so that the AR device 60 can overlay it onto the surgical robot through virtual-real fusion, and relevant personnel can perform fault-solving operations step by step. Therefore, this method also includes step A8: the video analysis system 70 simultaneously collects and analyzes the video of the relevant personnel's current step of fault-solving operation to obtain second-line data, and judges whether the relevant personnel's current step of fault-solving operation is effective based on the second-line data. If it is determined to be effective, the fault analysis system 82 sends the next step of fault-solving operation in the virtual fault-solving operation model to the AR device 60. If it is determined to be invalid, the AR device 60 guides the relevant personnel to repeat the current step of fault-solving operation. Alternatively, the fault analysis system 82 performs comprehensive analysis of the second-line data, the first-line data, and the status data to obtain the corresponding current fault cause and the corresponding virtual fault-solving operation model again, until the current step of fault-solving operation of the surgical robot is effectively completed, and then proceeds to the next step of fault-solving operation. This cycle continues until all faults of the surgical robot are eliminated, and then the fault-solving work ends, and the surgery continues.

[0091] Optionally, in the on-site troubleshooting method for the surgical robot in this embodiment, step A2 further includes:

[0092] First, the historical operation videos are classified according to different fault causes to obtain corresponding classified videos. Then, single-frame and multi-frame images of each classified video are modeled to obtain feature models and video models for each fault cause.

[0093] Secondly, while acquiring real-time video of the surgical robot's operation, starting from the moment the fault occurs, the video model of the current fault cause is used to extract the operation video acquired before the fault occurred, and the feature model of the current fault cause is used to perform behavioral data analysis on the extracted operation video to obtain the first behavioral data corresponding to the current fault cause.

[0094] It should be understood that in the above embodiments, the videos collected by the video analysis system are mainly videos of the preoperative, intraoperative, and troubleshooting stages. However, the technical solution of the present invention is not limited to this. In other embodiments of the present invention, when the operation of the surgical robot after surgery may affect the safety of the next surgery, the video analysis system can also further collect postoperative videos for postoperative on-site troubleshooting, thereby ensuring the correct use of the surgical robot after surgery.

[0095] As an example, please refer to further details. Figure 4 , Figures 8-9 When the data interaction system 81 obtains a no-energy-output fault code, it acquires pedal signal data S and energy device data E through the motion control system. The video analysis system 70 analyzes and determines whether the downward pedal travel before the fault occurred was greater than or equal to H (i.e., the position where the pedal signal was triggered). Based on the fault code from the data interaction system 81, the data from the motion control system, and the analysis results from the video analysis system 70, the fault analysis system 82 provides a corresponding virtual fault-solving operation model, and then instructs the corresponding fault-solving operation on the AR device, as follows:

[0096] When the analysis results show that the foot pedal has a signal, there is an energy device, and the foot pedal is in place, the current fault is determined to be no energy output. Based on the analysis results, the AR device will then indicate "replace the energy device".

[0097] When the analysis results indicate that the foot pedal has a signal and there is an energy device, but the foot pedal is not pressed down to the correct position, the cause of the current fault is determined to be no energy output. Based on the analysis results, the AR device will then indicate "the foot pedal is pressed down to the correct position".

[0098] When the analysis result shows that there is no signal from the foot pedal but there is an energy device and the foot pedal is in place, the current fault is determined to be no energy output. Based on the analysis result, the AR device will then indicate "Replace foot pedal".

[0099] When the analysis results show that there is a signal from the foot pedal but no energy device, and the foot pedal is in place, the current fault is determined to be no energy output. Based on the analysis results, the AR device will then indicate "replace the energy device".

[0100] In addition, the causes of failures, analysis results, and corresponding troubleshooting methods obtained in other examples can be referenced. Figure 4 and Figure 9 This will not be elaborated upon here.

[0101] Based on the same inventive concept, one embodiment of the present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the on-site troubleshooting method for surgical robots described in the present invention.

[0102] Furthermore, the storage medium can be any medium capable of containing, storing, transmitting, propagating, or transmitting computer programs. For example, the storage medium can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media, specific examples of which include: magnetic storage devices, such as magnetic tape or hard disk drives (HDDs); optical storage devices, such as optical discs (CDROMs); memory, such as random access memory (RAM) or flash memory; and / or wired or wireless communication links.

[0103] In summary, the technical solution of this invention can establish virtual troubleshooting operation models corresponding one-to-one with various fault causes based on historical data. By analyzing the results of a video analysis system and machine fault codes, it can deduce the current fault cause and find the corresponding virtual troubleshooting operation model. This model can then be superimposed onto the surgical robot using an AR device. In other words, it integrates the actual troubleshooting steps with the virtual 3D model corresponding to the pre-planned troubleshooting path, providing real-time and effective guidance to relevant personnel (also known as on-site personnel) to troubleshoot, shortening troubleshooting time, improving efficiency, reducing error rates, and shortening surgical time. Furthermore, the video analysis system can provide real-time feedback and analysis of the effectiveness of the troubleshooting operations performed by relevant personnel. Only when the operations are effective is the virtual 3D model of the next troubleshooting step superimposed onto the AR device. This real-time data feedback ensures the accuracy of the troubleshooting steps, saves time, and increases surgical safety.

[0104] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the scope of the present invention.

Claims

1. An AR-based on-site troubleshooting system for surgical robots, characterized in that, include: The fault handling system is used to establish virtual fault troubleshooting operation models corresponding to various causes of surgical robot failures based on historical data. A video analysis system is used to collect and analyze the operation videos of the surgical robot before and during the operation to obtain the first behavioral data before the failure occurs; A data interaction system is used to interact with the video analysis system and the motion control system of the surgical robot to obtain the status data of the surgical robot in real time. A fault analysis system is used to analyze the first behavioral data and the state data in order to find a virtual fault troubleshooting operation model corresponding to the current fault cause from the fault handling system based on the analysis results. AR devices are used to overlay the virtual fault-solving operation model determined by the fault analysis system onto the surgical robot through coordinate mapping, thereby guiding relevant personnel to perform fault-solving operations.

2. The on-site troubleshooting system for surgical robots as described in claim 1, characterized in that, The fault analysis system is also used to send each step of the fault troubleshooting operation in the virtual fault troubleshooting operation model to the AR device in a step-by-step manner; the video analysis system is also used to collect and analyze the fault troubleshooting operation video of the relevant personnel in the current step to obtain the second line of data, and to determine whether the fault troubleshooting operation of the relevant personnel in the current step is valid, and only when it is determined to be valid will the fault analysis system send the next fault troubleshooting operation in the virtual fault troubleshooting operation model to the AR device.

3. The on-site troubleshooting system for surgical robots as described in claim 2, characterized in that, The fault analysis system is also used to, when the video analysis system determines that the fault troubleshooting operation of the relevant personnel in the current step is invalid, cause the AR device to guide the relevant personnel to repeat the fault troubleshooting operation of the current step, or to perform comprehensive analysis on the second behavior data, the first behavior data and the status data to obtain the current fault cause and the corresponding virtual fault troubleshooting operation model again.

4. The on-site troubleshooting system for surgical robots as described in claim 1, characterized in that, The surgical robot includes a patient-side operating table and a doctor-side control console. The video analysis system includes a first camera device and a second camera device. The first camera device is used to capture operation videos and the status of each component of the patient-side operating table within the area of ​​the patient-side operating table. The second camera device is used to capture operation videos and the status of each component of the doctor-side control console within the area of ​​the doctor-side control console.

5. The on-site troubleshooting system for surgical robots as described in claim 4, characterized in that, The AR device has a binocular vision module, which is used to establish the coordinate mapping.

6. The surgical robot on-site troubleshooting system as described in any one of claims 1-5, characterized in that, The video analysis system or the fault handling system is further configured to: classify historical operation videos according to different fault causes, obtain corresponding classified videos, and model single-frame and multi-frame images of each classified video to obtain feature models and video models of each fault cause. The video analysis system is also used to capture the operation video of the surgical robot in real time, and, starting from the time of the failure, use the video model of the current failure cause to extract the operation video captured before the failure occurred, and use the feature model of the current failure cause to perform behavioral data analysis on the extracted operation video to obtain the first behavioral data corresponding to the current failure cause.

7. The on-site troubleshooting system for surgical robots as described in claim 6, characterized in that, When the surgical robot malfunctions, it automatically generates a fault code corresponding to the current cause of the malfunction. The fault analysis system obtains the virtual fault troubleshooting operation model based on the fault code, the first behavior data, and the status data. Alternatively, the AR device has a trigger button for manually triggering the fault analysis system when the surgical robot malfunctions, so that the fault analysis system can analyze and obtain a fault code corresponding to the current cause of the malfunction.

8. The on-site troubleshooting system for surgical robots as described in claim 6, characterized in that, The surgical robot includes a foot pedal, a robotic arm, an endoscope, a power unit, a trocar, and surgical instruments. The robotic arm includes a master hand on the doctor's control console and a slave hand on the patient's operating table. The causes of failure include at least one of the following: surgical posture mismatch, master-slave posture mismatch, endoscope malfunction causing fixed field of view, no energy output when the foot pedal is stepped on, the power unit failing to return to zero, the trocar becoming loose, the extension joint of the robotic arm reaching its limit position, and instruments failing to release.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the following steps: Based on historical data, virtual troubleshooting operation models are pre-established to correspond to various causes of surgical robot failures. Real-time acquisition and analysis of the surgical robot's operation videos before and during surgery to obtain the first behavioral data before the malfunction occurs; Real-time acquisition of the status data of the surgical robot; The first behavioral data and the state data are analyzed to identify the virtual fault troubleshooting operation model that causes the current fault from all the pre-established virtual fault troubleshooting operation models based on the analysis results. By using coordinate mapping on AR devices, the identified virtual troubleshooting operation model is superimposed onto the surgical robot through virtual-real fusion to guide relevant personnel in troubleshooting operations.

10. The storage medium as claimed in claim 9, characterized in that, The identified virtual troubleshooting operation model is superimposed onto the surgical robot using a virtual-real fusion method. Simultaneously, the video of the relevant personnel's current troubleshooting operation is collected and analyzed to obtain the second line of data. The effectiveness of the relevant personnel's current troubleshooting operation is then determined. Only when the operation is deemed effective is the next troubleshooting operation in the virtual troubleshooting operation model sent to the AR device.

11. The storage medium as claimed in claim 10, characterized in that, When it is determined that the troubleshooting operation of the relevant personnel in the current step is invalid, the AR device guides the relevant personnel to repeat the troubleshooting operation of the current step, or the second behavior data, the first behavior data and the status data are comprehensively analyzed to obtain the corresponding current fault cause and the corresponding virtual fault troubleshooting operation model again.

12. The storage medium as described in any one of claims 9-11, characterized in that, Also includes: Historical operation videos are classified according to different fault causes to obtain corresponding classified videos. Single-frame and multi-frame images of each classified video are modeled to obtain feature models and video models for each fault cause. While acquiring real-time video of the surgical robot's operation, starting from the moment the fault occurs, the video model of the current fault cause is used to extract the operation video acquired before the fault occurred, and the feature model of the current fault cause is used to perform behavioral data analysis on the extracted operation video to obtain the first behavioral data corresponding to the current fault cause.

Images