Unintended operation detection method, detection system, robot system, and storage medium

Abstract

The application provides a kind of unintended operation detection method, unintended operation detection system, surgical robot system, readable storage medium and computer equipment, the unintended operation detection method includes: obtaining the image information of operation end and the operation area adjacent to the operation end;At least one joint of the operation end is obtained Additional detection data;According to the image information and the additional detection data, whether the operation end generates unintended operation is judged based on preset algorithm.It is configured in this way, the position of the doctor and the surrounding obstacles can be detected based on image information, and whether the doctor and the operation end are separated during the operation can be known, based on additional detection data, whether the doctor and the operation end are separated can also be known, and further, whether the movement of the operation end is caused by unintended operation can be known.The dual detection of image information and additional detection data improves the reliability of detection, and avoids the state misjudgment caused by the detection result of a single sensor.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a method, detection system, robotic system, and storage medium for detecting unexpected operations. Background Technology

[0002] Surgical robotic systems enable surgeons to observe tissue features within a patient's body from a master control console using 2D or 3D displays. They can then remotely control the master arm on the console, driving the robotic arms and surgical instruments on the slave robot via master-slave mapping to perform the surgery. Surgeons can perform minimally invasive surgeries with the same methods and sensations as traditional surgeries, significantly reducing the difficulty of these procedures, improving efficiency and safety, and representing a breakthrough in remote surgery.

[0003] If the primary operator makes unexpected movements during surgery, the slave device's follow-up movements can easily lead to surgical risks. However, there is currently no specific technology for detecting unexpected movements of the primary operator. Related technologies, such as collision detection technology and methods to prevent the primary operator from being interfered with by external collisions and sending unexpected movements, include:

[0004] 1) Collision detection between robotic arms and between robotic arm joints of the slave robot, such as using OBB bounding box collision detection;

[0005] 2) Use a fence to prevent external collisions and interference with the main control console.

[0006] The defects or shortcomings of these related detection technologies include:

[0007] 1) OBB bounding box collision detection is mainly for collision detection between rigid body structures with regular convex polyhedral shapes whose motion information is known. It is not suitable for detecting external collision interference.

[0008] 2) Using fences cannot completely prevent external collisions and interference. If an external collision occurs in an area not protected by the fence, it will cause the main operator to move unexpectedly. Summary of the Invention

[0009] The purpose of this invention is to provide a method, system, surgical robot system, readable storage medium, and computer device for detecting unexpected operations, in order to solve the problem that the prior art cannot detect unexpected movements of the main manipulator of a surgical robot.

[0010] To address the aforementioned technical problems, this invention provides a method for detecting unexpected operations, comprising:

[0011] Acquire image information of the operating terminal and the operating area adjacent to the operating terminal;

[0012] Acquire additional detection data for at least one joint of the operating end; and

[0013] Based on the image information and the additional detection data, a preset algorithm is used to determine whether the operating terminal has generated an unexpected operation.

[0014] Optionally, the additional detection data includes contact stress data between the operator and the operating end; the step of determining whether the operating end has generated unexpected operation based on the image information and the additional detection data and a preset algorithm includes:

[0015] Based on the image information and the contact stress data, the connection state between the operator and the operating end is obtained based on a preset algorithm; if the connection state is that the operator and the operating end are separated, and the movement speed of the operating end exceeds a preset threshold, it is determined that the operating end has performed an unexpected operation.

[0016] Optionally, the additional detection data includes contact stress data between the operator and the operating end; the additional detection data also includes joint angle data, velocity data, acceleration data, and joint driving torque data; the step of determining whether the operating end produces unexpected operation based on the image information and the additional detection data and a preset algorithm includes:

[0017] The calculated torque data of the joint is obtained using the angle data, velocity data, and acceleration data.

[0018] Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end is obtained;

[0019] The operational state between the operator and the operating end is obtained based on the contact stress data and the external torque data.

[0020] Based on the image information and the operation status, a preset algorithm is used to determine whether the operation terminal has been impacted by an external force; if so, it is determined that the operation terminal has performed an unexpected operation.

[0021] Optionally, the joint driving torque data is obtained based on the current feedback of the joint motor, or the joint driving torque data is obtained based on the detection of the joint torque sensor.

[0022] Optionally, if it is determined that the operating terminal has generated an unexpected operation, the unexpected operation detection method further includes: providing a warning message, and / or restricting the master-slave mapping between the operating terminal and the slave device.

[0023] Optionally, the preset algorithm includes at least one of the weighted average method, the Kalman filter algorithm, and the artificial neural network algorithm.

[0024] To address the aforementioned technical problems, the present invention also provides a method for detecting unexpected operations, comprising:

[0025] Acquire contact stress data between the operator and the operating device;

[0026] Acquire angle data, velocity data, acceleration data, and joint driving torque data of at least one joint of the operating end;

[0027] The calculated torque data of the joint is obtained using the angle data, velocity data, and acceleration data.

[0028] Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end is obtained;

[0029] The operational state between the operator and the operating end is obtained based on the contact stress data and the external torque data.

[0030] If the operating state is such that the operating torque data applied by the operator to the operating terminal is not equal to the external torque data, then it is determined that the operating terminal has generated an unexpected operation.

[0031] To address the aforementioned technical problems, the present invention also provides an unexpected operation detection system, comprising: a visual sensor, an additional sensor, and a control device;

[0032] The visual sensor is used to acquire image information of the operating terminal and the operating area adjacent to the operating terminal.

[0033] The additional sensor is used to acquire additional detection data for at least one joint of the operating end;

[0034] The control device is used to determine, based on the image information and the additional detection data, whether the operating terminal has generated an unexpected operation according to a preset algorithm.

[0035] Optionally, the additional sensor includes a stress sensor disposed on the operating end, and the additional detection data includes contact stress data between the operator and the operating end, and the stress sensor is used to acquire the contact stress data;

[0036] The control device determines the connection state between the operator and the operating end based on the image information and the contact stress data using a preset algorithm; if the connection state is that the operator and the operating end are separated, and the movement speed of the operating end exceeds a preset threshold, then it is determined that the operating end has performed an unexpected operation.

[0037] Optionally, the additional sensors include a stress sensor, a joint angle sensor, a joint velocity sensor, and a joint acceleration sensor disposed on the operating end; the additional sensors also include a joint torque sensor or a joint motor current sensor disposed on the operating end.

[0038] The additional detection data includes contact stress data between the operator and the operating end; the additional detection data also includes joint angle data, velocity data, acceleration data, and joint driving torque data; the stress sensor is used to acquire the contact stress data; the joint angle sensor is used to acquire the angle data; the joint velocity sensor is used to acquire the velocity data; and the joint acceleration sensor is used to acquire the acceleration data.

[0039] The joint driving torque data is obtained based on the current fed back by the joint motor current sensor, or the joint driving torque data is obtained based on the detection by the joint torque sensor;

[0040] The control device is configured to: obtain calculated torque data of the joint using the angle data, velocity data, and acceleration data; obtain external torque data of the joint on the operating end based on the calculated torque data and the joint driving torque data; obtain the operating state between the operator and the operating end based on the contact stress data and the external torque data; determine whether the operating end has been impacted by an external force based on the image information and the operating state; if so, determine that the operating end has performed an unexpected operation.

[0041] Optionally, the unexpected operation detection system further includes an alarm device; when the control device determines that the operation terminal has generated an unexpected operation, the alarm device is triggered to display an alarm message.

[0042] Optionally, the control device is configured to communicate with the master-slave mapping module. When the control device determines that the operation terminal has generated an unexpected operation, the control device is configured to drive the master-slave mapping module to stop mapping the operation terminal's action to the slave device.

[0043] To address the aforementioned technical problems, the present invention also provides an unexpected operation detection system, comprising: a control device, a stress sensor, a joint angle sensor, a joint velocity sensor, and a joint acceleration sensor disposed on the operating end; the unexpected operation detection system further comprises a joint torque sensor or a joint motor current sensor disposed on the operating end;

[0044] The stress sensor is used to acquire contact stress data between the operator and the operating end; the joint angle sensor is used to acquire the angle data; the joint velocity sensor is used to acquire the velocity data; and the joint acceleration sensor is used to acquire the acceleration data.

[0045] The joint driving torque data is obtained based on the current fed back by the joint motor current sensor, or the joint driving torque data is obtained based on the detection by the joint torque sensor;

[0046] The control device is configured to: obtain calculated torque data of the joint using the angle data, velocity data, and acceleration data; obtain external torque data of the joint on the operating end based on the calculated torque data and the joint driving torque data; obtain the operating state between the operator and the operating end according to the contact stress data and the external torque data; if the operating state is that the operating torque data applied by the operator to the operating end is not equal to the external torque data, then determine that the operating end has generated an unexpected operation.

[0047] To address the aforementioned technical problems, the present invention also provides a surgical robot system, which includes a master end device and a slave end device. The master end device includes an operating end and also includes the unexpected operation detection system described above.

[0048] To address the aforementioned technical problems, the present invention also provides a readable storage medium having a program stored thereon, which, when executed, implements the steps of the unexpected operation detection method described above.

[0049] To address the aforementioned technical problems, the present invention also provides a computer device comprising a processor and a readable storage medium as described above, wherein the processor is configured to execute the program stored on the readable storage medium.

[0050] In summary, in the unexpected operation detection method, unexpected operation detection system, surgical robot system, readable storage medium, and computer device provided by the present invention, the unexpected operation detection method includes: acquiring image information of the operating end and the operating area adjacent to the operating end; acquiring additional detection data of at least one joint of the operating end; and determining whether the operating end has generated an unexpected operation based on the image information and the additional detection data and a preset algorithm.

[0051] This configuration allows for the detection of the doctor's and surrounding obstacles based on image information, and also determines whether the doctor has become detached from the operating device during surgery. Additional detection data further confirms this detachment and indicates whether the device's movement is caused by unintended actions. This dual detection of image information and additional data enhances reliability and avoids misjudgments caused by relying on a single sensor. Furthermore, the additional detection data reveals whether the device is subjected to unexpected external interference during movement. Therefore, it accurately determines whether the device has engaged in unintended actions, effectively preventing surgical accidents caused by the doctor becoming detached from the device, leading to unintended movement and subsequent tracking by the slave device. This improves the safety and controllability of robotic surgery. Attached Figure Description

[0052] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0053] Figure 1 This is a schematic diagram illustrating an application scenario of a surgical robot system;

[0054] Figure 2 This is a schematic diagram of the unexpected operation detection system according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of an additional sensor according to an embodiment of the present invention;

[0056] Figure 4 This is a flowchart illustrating the steps of the weighted average method according to an embodiment of the present invention.

[0057] Figure 5 This is a flowchart of the Kalman filtering algorithm according to an embodiment of the present invention;

[0058] Figure 6 This is a flowchart of the steps of the artificial neural network algorithm according to an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of the operating terminal according to an embodiment of the present invention;

[0060] Figure 8 This is a flowchart of the unexpected operation detection method according to an embodiment of the present invention;

[0061] Figure 9 This is a flowchart of the computational model of the unexpected operation detection method according to an embodiment of the present invention.

[0062] In the attached image:

[0063] 100-Main device; 101-Operating end; 102-Imaging equipment; 103-Foot-operated surgical control device; 104-End loop; 200-Slave device; 201-Base; 210-Instrument arm; 221-Surgical instrument; 222-Endoscope; 300-Image carriage; 302-Display device; 400-Support device; 410-Patient; 420-Ventilator and anesthesia machine; 430-Instrument table; 500-Visual sensor; 600-Additional sensor. Detailed Implementation

[0064] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to facilitate and clarify the explanation of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and may sometimes use different scales.

[0065] As used herein, the singular forms “a,” “an,” and “the” include plural objects; the term “or” is generally used to include the meaning of “and / or”; the term “a number” is generally used to include the meaning of “at least one”; and the term “at least two” is generally used to include the meaning of “two or more”. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature; “one end” and “the other end,” and “proximal end” and “distal end” generally refer to two corresponding parts, which include not only endpoints. The terms “proximal end” and “distal end” are defined herein relative to the relative positional relationship between the operator, such as a surgeon or clinician, and the patient. The term “proximal end” refers to the position of the element closer to the operator, and the term “distal end” refers to the position of the element closer to the patient or lesion and therefore further away from the operator. Furthermore, the terms "installed," "connected," and "attached," as used in this invention, and the term "set" on one element from another, should be interpreted broadly. They generally only indicate a connection, coupling, cooperation, or transmission relationship between the two elements, which can be direct or indirect through an intermediate element. They should not be construed as indicating or implying a spatial relationship between the two elements, meaning one element can be located inside, outside, above, below, or to one side of another element, unless otherwise explicitly stated. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Additionally, directional terms such as above, below, up, down, upward, downward, left, and right are used relative to exemplary embodiments as shown in the figures, with upward or upper directions pointing towards the top of the corresponding figure, and downward or lower directions pointing towards the bottom of the corresponding figure.

[0066] The purpose of this invention is to provide a method, system, surgical robot system, readable storage medium, and computer device for detecting unexpected operations, in order to solve the problem that the prior art cannot detect unexpected movements of the main operator of a surgical robot.

[0067] The following description refers to the accompanying drawings.

[0068] Figure 1An application scenario of a surgical robot system is illustrated. The surgical robot system includes a master-slave teleoperated surgical robot, comprising a master device 100 (i.e., a doctor's control device), a slave device 200 (i.e., a patient's control device), a main controller, and a support device 400 (e.g., an operating table) for supporting the surgical object during surgery. It should be noted that in some embodiments, the support device 400 may be replaced with other surgical operating platforms; this invention is not limited thereto.

[0069] The master device 100 is the master end of the teleoperated surgical robot and includes an operating end 101 (such as a master hand) mounted thereon. The operating end 101 is used to receive hand movement information from the operator as motion control signal input for the entire system. Optionally, the main controller is also mounted on the master device 100. Preferably, the master device 100 also includes an imaging device 102, which provides the operator with stereoscopic images and surgical field images for surgical operations. The surgical field images include the type and number of surgical instruments, their position in the abdomen, and the morphology and arrangement of the patient's organs and tissues, as well as the blood vessels of surrounding organs and tissues. Optionally, the master device 100 also includes a foot-operated surgical control device 103, through which the operator can input related operation commands such as electrocautery and electrocoagulation.

[0070] The slave device 200 is the specific execution platform for the remotely operated surgical robot, and includes a base 201 and surgical execution components mounted thereon. The surgical execution components include an instrument arm 210 and instruments, the instruments being mounted or connected to the end of the instrument arm 210. Further, the instruments include surgical instruments 221 (such as a high-frequency electrosurgical unit) for performing specific surgical operations and endoscopes 222 for assisting observation.

[0071] In one embodiment, the instrument arm 210 includes an adjustment arm and a working arm. The working arm is a mechanical fixed-point mechanism used to drive the instrument to move around the mechanical fixed point to perform minimally invasive surgical treatment or imaging operations on the patient 410 on the support device 400. The adjustment arm is used to adjust the position and orientation of the mechanical fixed point in the workspace. In another embodiment, the instrument arm 210 is a mechanism with at least six degrees of freedom in a spatial configuration used to drive the surgical instrument 221 to move around an active fixed point under program control. The surgical instrument 221 is used to perform specific surgical operations, such as clamping, cutting, and scissing. It should be noted that, since the surgical instrument 221 and endoscope 222 have a certain volume in practice, the aforementioned "fixed point" should be understood as a stationary region. Of course, those skilled in the art can understand the "fixed point" according to the prior art.

[0072] The main controller is communicatively connected to both the master device 100 and the slave device 200. It controls the movement of the surgical execution component based on the movement of the operating end 101. Specifically, the main controller includes a master-slave mapping module. This module acquires the end-effector pose of the operating end 101 and a predetermined master-slave mapping relationship to obtain the desired end-effector pose of the surgical execution component, thereby controlling the instrument arm 210 to drive the instrument to the desired end-effector pose. Furthermore, the master-slave mapping module also receives instrument function operation commands (such as electrocautery, electrocoagulation, etc.) and controls the energy driver of the surgical instrument 221 to release energy for electrocautery, electrocoagulation, and other surgical operations. In some embodiments, the main controller also receives force information received by the surgical execution component (e.g., the force information of human tissues and organs on the surgical instrument) and feeds this force information back to the operating end 101, allowing the operator to more intuitively feel the feedback force of the surgical operation.

[0073] Furthermore, the medical robot system also includes an image cart 300. The image cart 300 includes an image processing unit (not shown) communicatively connected to the endoscope 222. The endoscope 222 is used to acquire surgical field images within the cavity (referring to the patient's body cavity). The image processing unit is used to perform image processing on the surgical field images acquired by the endoscope 222 and transmit them to the imaging device 102 so that the operator can observe the surgical field images. Optionally, the image cart 300 also includes a display device 302. The display device 302 is communicatively connected to the image processing unit and is used to provide the operator (e.g., a nurse) with real-time display of the surgical field images or other auxiliary display information.

[0074] Optionally, in some surgical applications, the surgical robot system may also include auxiliary components such as a ventilator and anesthesia machine 420 and an instrument table 430 for use during surgery. Those skilled in the art can select and configure these auxiliary components according to existing technology, which will not be described in detail here.

[0075] It should be noted that the surgical robot system disclosed in the above examples is only an example of an application scenario and not a limitation on the application scenario of the surgical robot system. The surgical robot system is not limited to a master-slave teleoperated surgical robot, but can also be a single-end surgical robot system in which the operator directly operates the surgical robot to perform surgery. This invention is not limited to this.

[0076] Unintended movement of the operating terminal 101 during surgery can cause dangerous following motion of the slave device 200. Detecting and protecting against unintended movement of the operating terminal 101 can greatly reduce the risk of surgery. The inventors have discovered that unintended movement of the operating terminal 101 typically includes two types: one is when the doctor releases their grip on the operating terminal 101, causing it to lose control but continue moving due to inertia; the other is when a person or obstacle from the external environment impacts the operating terminal 101, causing it to deviate from its intended position and produce unintended movement.

[0077] Based on the above-described surgical robot system, this embodiment of the invention provides a method for detecting unexpected operations, comprising:

[0078] Step S1: Obtain image information of the operation terminal 101 and the operation area adjacent to the operation terminal 101;

[0079] Step S2: Obtain additional detection data for at least one joint of the operating end 101; and

[0080] Step S3: Based on the image information and the additional detection data, determine whether the operating terminal has generated an unexpected operation based on a preset algorithm.

[0081] Please refer to Figure 2 This embodiment also provides an unexpected operation detection system for implementing the unexpected operation detection method described above. The unexpected operation detection system includes a vision sensor 500, an additional sensor, and a control device (not shown); the control device is communicatively connected to both the vision sensor 500 and the additional sensor. The control device can be provided independently or integrated into or attached to the surgical robot system, for example, it can be integrated into the main controller of the surgical robot system.

[0082] Step S1 can be implemented using a vision sensor 500, which acquires image information of the operating end 101 and the operating area adjacent to the operating end 101. The vision sensor 500 may include, for example, devices commonly used in the art such as a camera. Furthermore, those skilled in the art can set the range of the operating area adjacent to the operating end 101 according to actual conditions. It is understood that by acquiring image information, the control device can analyze and calculate the positional relationship between the doctor's hand and the end of the operating end 101, thereby determining whether the doctor has detached from the operating end 101, and analyzing and calculating whether there is any collision between the operating end 101 and external objects (such as other people or obstacles besides the operating doctor).

[0083] Step S2 can be implemented using an additional sensor, which acquires additional detection data of at least one joint of the operating end 101. Based on this additional detection data, the control device can also determine whether the doctor has disengaged from the operating end 101. Furthermore, in some applications, the control device can also determine whether the force applied to the operating end 101 is as expected based on the additional detection data. For example, if the operating end 101 is subjected to an impact, the additional detection data of the joints acquired by the additional sensor will be abnormal. Therefore, the control device can determine that the force applied to the operating end 101 is unexpected.

[0084] Step S3 can be implemented through a control device.

[0085] This configuration allows for the detection of the doctor's and surrounding obstacles based on image information, and also determines whether the doctor has become detached from the operating terminal 101 during surgery. Additional detection data further confirms this, indicating whether the movement of the operating terminal 101 is caused by unintended actions. This dual detection of image information and additional detection data enhances the reliability of the detection, avoiding misjudgments caused by relying on a single sensor. Furthermore, the additional detection data reveals whether the operating terminal 101 is subjected to unexpected external interference during movement. Therefore, it is possible to accurately determine whether the operating terminal 101 has performed unintended actions, effectively preventing surgical accidents caused by the doctor becoming detached from the operating terminal 101 during surgery, leading to unintended movement of the operating terminal 101 and subsequent tracking by the slave device 200. This improves the safety of robotic surgery and ensures the controllability of the surgical procedure.

[0086] Please refer to Figure 3 Optionally, in one embodiment, the additional sensor includes a stress sensor 600 disposed on the operating end 101, and the additional detection data includes contact stress data between the operator and the operating end 101. The stress sensor 600 is used to acquire the contact stress data. Correspondingly, in the unexpected operation detection method, step S3 includes: obtaining the connection state between the operator and the operating end 101 based on the image information and the contact stress data using a preset algorithm; if the connection state is that the operator is detached from the operating end 101, and the movement speed of the operating end 101 exceeds a preset threshold, then it is determined that the operating end 101 has performed an unexpected operation. Specifically, when it is detected that the doctor's hand has detached from the end of the operating end 101, but the operating end 101 still has a large movement speed (movement speed exceeding a preset threshold), it is determined that the operating end 101 has performed an unexpected operation.

[0087] Furthermore, a stress sensor 600 can be installed within the end ring 104 of the operating end 101, which can acquire contact stress data of the kneading tabs of the end ring 104. The contact force caused by the doctor's hand within the end ring 104 changes the magnitude of the contact stress data of the kneading tabs, thereby allowing estimation of whether the doctor's hand is within the end ring 104 based on the value of the contact stress data. Since using image information or contact stress data alone to determine whether the doctor's hand is within the end ring 104 has a certain margin of error (for example, the doctor's hand may indeed be within the end ring 104, but the stress may be smaller when the hand is relaxed), combining the image information and the contact stress data according to a preset algorithm for analysis and calculation can obtain an accurate engagement state between the operator and the operating end 101, avoiding misjudgments caused by using the detection results of a single sensor.

[0088] Optionally, the preset algorithm includes at least one of the weighted average method, Kalman filter algorithm, and artificial neural network algorithm. The following is in conjunction with... Figures 4 to 6 The weighted average method, Kalman filter algorithm, and artificial neural network algorithm are explained respectively.

[0089] Figure 4 The flowchart of the weighted average method is shown. Based on the detection result z1 (such as image information) obtained by sensor 1 (e.g., vision sensor 500), estimated state 1 can be obtained; based on the detection result z2 (such as contact stress data) obtained by sensor 2 (e.g., stress sensor 600), estimated state 2 can be obtained.

[0090] Therefore, the final target estimated state z can be obtained through the weighted average method:

[0091]

[0092] Where w1 and w2 are the weights of sensor 1 and sensor 2 in the state estimation calculation, respectively, and can be set by those skilled in the art according to actual conditions. It is understood that in the example using vision sensor 500 and stress sensor 600 as sensors, the final target estimation state z is the combined state between the operator and the operating terminal 101. It should be understood that the above example only shows the final target estimation state z obtained by weighted averaging the detection results of two sensors. In other embodiments, this can also be extended to obtaining the final target estimation state z by weighted averaging the detection results of multiple sensors. Those skilled in the art can understand this based on existing technology, and it will not be elaborated here.

[0093] Figure 5The flowchart of the Kalman filtering algorithm is shown. By performing Kalman filtering on two or more estimated states obtained from the detection results of two or more sensors, the final estimated target state can be obtained.

[0094] The Kalman filter algorithm combines the sensor's detection result in the current cycle with the estimated state in the previous cycle to obtain the estimated state for the current cycle, thus reducing the estimation error caused by sensor detection errors. For example... Figure 5 As shown in the example, an initial state needs to be set first, which serves as the estimated state for the next cycle. Then, at the beginning of the next cycle, the sensor's detection result and the estimated state from the previous cycle are used to calculate the current estimated state according to the state update equation and output it. This also updates the estimated state for the next cycle. The specific formula is: Current estimated state = Estimated state from the previous cycle + Kalman gain factor × (Sensor detection result for the current cycle - Estimated state from the previous cycle for the current cycle).

[0095] It needs to be understood. Figure 2 The example only shows the final target estimation state obtained by processing the detection results of two sensors through the Kalman filter algorithm. In other embodiments, the method can also be extended to obtain the final target estimation state by processing the detection results of multiple sensors through the Kalman filter algorithm. Those skilled in the art can understand this based on the prior art, and it will not be elaborated here.

[0096] Figure 6 The flowchart illustrates the steps of an artificial neural network algorithm. By inputting the detection results from two or more sensors into a trained BP (backpropagation) neural network, the final state estimate output by the network is obtained. The neural network can be divided into three layers: an input layer, a hidden layer, and an output layer. Each neuron in these three layers is connected to neurons in adjacent layers via neural lines, and each neuron is only responsible for acquiring input information, making judgments, and providing output information.

[0097] It needs to be understood. Figure 6 The example only shows that the detection results of two sensors are used to obtain the final target estimation state through an artificial neural network algorithm. In other embodiments, the method can also be extended to obtain the final target estimation state through the detection results of multiple sensors. Those skilled in the art can understand this based on the existing technology, and it will not be elaborated here.

[0098] Optionally, in another embodiment, the additional sensors include a stress sensor 600, a joint angle sensor, a joint velocity sensor, and a joint acceleration sensor disposed on the operating end 101; the additional sensors also include a joint torque sensor or a joint motor current sensor disposed on the operating end 101; the additional detection data includes contact stress data between the operator and the operating end 101; the additional detection data also includes joint angle data, velocity data, acceleration data, and joint driving torque data; the stress sensor 600 is used to acquire the contact stress data; the joint angle sensor is used to acquire the angle data; the joint velocity sensor is used to acquire the velocity data; and the joint acceleration sensor is used to acquire the acceleration data. Preferably, the joint driving torque data is obtained based on the current fed back by the joint motor current sensor, or the joint driving torque data is obtained based on the detection by the joint torque sensor.

[0099] In a corresponding method for detecting unexpected operations, step S3 includes:

[0100] Step S31: Use the angle data, velocity data, and acceleration data to obtain the calculated torque data of the joint;

[0101] Step S32: Based on the calculated torque data and the joint driving torque data, obtain the external torque on the joint on the operating end 101;

[0102] Step S33: Obtain the operating state between the operator and the operating end 101 based on the contact stress data and the external torque;

[0103] Step S34: Based on the image information and the operation state, determine whether the operation terminal has been impacted by an external force using a preset algorithm; if so, determine that the operation terminal has performed an unexpected operation.

[0104] It should be noted that the stress sensor 600, joint angle sensor, joint speed sensor, joint acceleration sensor, joint torque sensor, and joint motor current sensor are all common sensors in the field. Their structure and detection principle can be understood by those skilled in the art based on existing technology, and this invention will not elaborate on them.

[0105] In step S31, based on angle data (i.e., joint position), velocity data, and acceleration data, the external torque acting on the joint of the operating end 101 can be obtained through a dynamic model algorithm, thereby determining whether the operating end 101 is subjected to unexpected external disturbance forces during movement. The following section combines... Figure 7 Please provide an explanation.

[0106] like Figure 7As shown, in some embodiments, the operating end 101 is a multi-joint connected robotic arm structure, with mutual forces between the joints. The force situation of each joint can be derived using the Newton-Euler equations as follows:

[0107] Figure 7 In the example shown, f i Let n be the force acting on joint i. i F is the torque (calculated torque) acting on joint i. i Let N be the force acting on link i. i Let be the torque acting on link i;

[0108] According to Newton's equations, we have: F i =m*a, where m is the mass of the connecting rod and a is the linear acceleration of the connecting rod;

[0109] According to Euler's equations: Where I is the moment of inertia of the connecting rod, ω i The angular velocity of the connecting rod.

[0110] According to the force equilibrium equation of the connecting rod, F i =f i -R*f i+1 , where R is the rotation matrix between coordinate system i and coordinate system i+1 corresponding to the two joints;

[0111] According to the torque equilibrium equation of the connecting rod, we get: N i =n i -n i+1 +P ci ×f i -(P i+1 -P ci )×f i+1 , where P ci Let i be the position of the link's center of mass in coordinate system i corresponding to joint i;

[0112] Therefore, the calculated torque data n of joint i can be obtained based on the joint angle data, velocity data, and acceleration data. i Size.

[0113] In step S32, the calculated torque data is subtracted from the joint's driving torque data to obtain the external torque data acting on the joint: τ ext =τ exp -τ fdb ; where τ ext This represents the external torque data acting on the joint, τ exp τ represents the calculated torque data of the joint obtained from the joint dynamics model. fdbThis represents the driving torque data of the joint. The driving torque data of the joint can be obtained based on the current fed back by the joint motor current sensor, or it can be obtained directly from the joint torque sensor.

[0114] In step S33, the operating torque data applied by the operator to the operating end 101 can be obtained based on the contact stress data. This operating torque data is then compared with the external torque data τ experienced by the joint. ext By comparing the two data points, the operational status between the operator and the control terminal 101 can be obtained. A normal operational status should be that the operating torque data and the external torque data τ are compared. ext Equal, that is, the external torque data τ acting on the joint. ext The operating torque data applied by the operator should be equal.

[0115] In step S34, if the operating torque data applied by the operator to the operating terminal 101 is not equal to the external torque data, then from the perspective of torque calculation, it is determined that the operating terminal 101 may be subjected to other external impacts. Based on the preset algorithm combined with image information analysis to determine whether there is an external person or object impacting the operating terminal 101, it can be comprehensively determined whether the operating terminal 101 has been impacted externally; if so, it is determined that the operating terminal 101 has performed an unexpected operation. The preset algorithm here may include at least one of the weighted average method, Kalman filter algorithm, and artificial neural network algorithm, as detailed in the foregoing description.

[0116] Furthermore, the unexpected operation detection system also includes a warning device; when the control device determines that the operating terminal 101 has generated an unexpected operation, the warning device is triggered to display a warning message. The warning device may include warning lights, buzzers, displays, or other warning devices commonly used in the art. Corresponding warning messages may include sound, light, and text prompts. In some embodiments, the control device is also communicatively connected to the master-slave mapping module of the main controller of the surgical robot system. When the control device determines that the operating terminal 101 has generated an unexpected operation, the control device drives the master-slave mapping module to stop mapping the action of the operating terminal 101 to the slave device 200, preventing the slave device 200 from generating dangerous following actions. Correspondingly, in the unexpected operation detection method, if it is determined that the operating terminal 101 has generated an unexpected operation, the unexpected operation detection method further includes: displaying a warning message, and / or restricting the master-slave mapping between the operating terminal 101 and the slave device. The entire unexpected operation detection process and calculation model flow are as follows: Figure 8 and Figure 9 As shown.

[0117] In another embodiment, the unexpected operation detection system may not include the vision sensor 500, but includes a control device, a stress sensor 600, a joint angle sensor, a joint velocity sensor, and a joint acceleration sensor disposed on the operating end 101; the unexpected operation detection system also includes a joint torque sensor or a joint motor current sensor disposed on the operating end 101. The control device is configured to: obtain calculated torque data of the joint using the angle data, velocity data, and acceleration data; obtain external torque data of the joint on the operating end based on the calculated torque data and the joint driving torque data; obtain the operation state between the operator and the operating end according to the contact stress data and the external torque data; if the operation state is that the operating torque data applied by the operator to the operating end 101 is not equal to the external torque data, then it is determined that the operating end has generated an unexpected operation.

[0118] Accordingly, the unexpected operation detection method includes:

[0119] Acquire contact stress data between the operator and the operating terminal 101;

[0120] Acquire angle data, velocity data, acceleration data, and joint driving torque data of at least one joint of the operating terminal 101;

[0121] The calculated torque data of the joint is obtained using the angle data, velocity data, and acceleration data.

[0122] Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end 101 is obtained;

[0123] The operating state between the operator and the operating end 101 is obtained based on the contact stress data and the external torque data.

[0124] If the operating state is such that the operating torque data applied by the operator to the operating terminal 101 is not equal to the external torque data, then it is determined that the operating terminal has generated an unexpected operation.

[0125] In this embodiment, instead of using dual detection of image information and additional detection data, a detection algorithm based on a dynamic model can be used to obtain the external torque on the joint of the operating end 101, and directly determine whether the operating end 101 is subjected to unexpected external interference forces during movement. When an unexpected external interference force is detected, causing the operating end 101 to produce unexpected movement, it is determined that the operating end 101 has performed an unexpected operation. Furthermore, an alarm can be triggered and the movement of the operating end 101 can be locked. For details, please refer to the foregoing description, which will not be repeated here.

[0126] Based on the unexpected operation detection method and unexpected operation detection system described above, this embodiment of the invention also provides a surgical robot system, which includes a master end device 100 and a slave end device 200. The master end device 100 includes an operating end 101 and also includes the unexpected operation detection system described above.

[0127] Furthermore, embodiments of the present invention also provide a readable storage medium storing a program thereon, which, when executed, implements the steps of the unexpected operation detection method described above. Even further, embodiments of the present invention also provide a computer device including a processor and the readable storage medium described above, the processor being used to execute the program stored on the readable storage medium. The readable storage medium can be set independently or integrated into a surgical robot system, such as integrating it into the main controller; the present invention is not limited thereto.

[0128] In summary, the unexpected operation detection method, unexpected operation detection system, surgical robot system, readable storage medium, and computer device provided by this invention include: acquiring image information of the operating end and the operating area adjacent to the operating end; acquiring additional detection data of at least one joint of the operating end; and determining whether the operating end has generated an unexpected operation based on the image information and the additional detection data, using a preset algorithm. With this configuration, the positions of the doctor and surrounding obstacles can be detected based on the image information, and it can be determined whether the doctor has become detached from the operating end during surgery. Based on the additional detection data, it can also be determined whether the doctor has become detached from the operating end, and further, whether the movement of the operating end is caused by an unexpected operation. The dual detection of image information and additional detection data improves the reliability of the detection and avoids misjudgments caused by using the detection results of a single sensor. Furthermore, based on the additional detection data, it can also be determined whether the operating end is subjected to unexpected external interference forces during movement. Therefore, it is possible to accurately determine whether the operating end produces unexpected operations, which can effectively avoid surgical accidents caused by the surgeon becoming detached from the operating end during the operation, resulting in unexpected movement of the operating end and the slave device following. This improves the safety of robotic surgery and ensures the controllability of the surgical operation.

[0129] It should be noted that the above embodiments can be combined with each other. The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A method for detecting unexpected operations, characterized in that, include: Acquire image information of the operating terminal and the operating area adjacent to the operating terminal; Acquire additional detection data for at least one joint of the operating end, the additional detection data including contact stress data between the operator and the operating end, angle data, velocity data, acceleration data, and joint driving torque data of the joint; as well as Based on the image information and the additional detection data, a preset algorithm is used to determine whether the operating terminal has generated an unexpected operation. The steps include: The calculated torque data of the joint is obtained using the angle data, velocity data, and acceleration data. Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end is obtained; The operational state between the operator and the operating end is obtained based on the contact stress data and the external torque data. Based on the image information and the operation status, a preset algorithm is used to determine whether the operation terminal has been impacted by an external force; if so, it is determined that the operation terminal has been impacted by an external force and has caused an unexpected operation.

2. The method for detecting unexpected operations according to claim 1, characterized in that, Based on the image information and the additional detection data, the step of determining whether the operating terminal has generated an unexpected operation according to a preset algorithm includes: Based on the image information and the contact stress data, the connection state between the operator and the operating end is obtained based on a preset algorithm; if the connection state is that the operator and the operating end are separated, and the movement speed of the operating end exceeds a preset threshold, it is determined that the operating end has separated from the operator and an unexpected operation has occurred.

3. The method for detecting unexpected operations according to claim 1, characterized in that, The joint driving torque data is obtained based on the current feedback of the joint motor, or the joint driving torque data is obtained based on the detection of the joint torque sensor.

4. The method for detecting unexpected operations according to claim 1, characterized in that, If it is determined that the operating terminal has generated an unexpected operation, the unexpected operation detection method further includes: providing a warning message, and / or restricting the master-slave mapping between the operating terminal and the slave device.

5. The method for detecting unexpected operations according to claim 1, characterized in that, The preset algorithm includes at least one of the following: weighted average method, Kalman filter algorithm, and artificial neural network algorithm.

6. A method for detecting unexpected operations, characterized in that, include: Acquire contact stress data between the operator and the operating device; Acquire angle data, velocity data, acceleration data, and joint driving torque data of at least one joint of the operating end; The calculated torque data of the joint is obtained using the angle data, velocity data, and acceleration data. Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end is obtained; The operational state between the operator and the operating end is obtained based on the contact stress data and the external torque data. If the operating state is such that the operating torque data applied by the operator to the operating end is not equal to the external torque data, it is determined that the operating end has been impacted by an external force, resulting in an unexpected operation.

7. An unexpected operation detection system, characterized in that, include: Vision sensors, additional sensors, and control devices; The visual sensor is used to acquire image information of the operating terminal and the operating area adjacent to the operating terminal. The additional sensors are used to acquire additional detection data of at least one joint of the operating end; the additional sensors include a stress sensor, a joint angle sensor, a joint velocity sensor, and a joint acceleration sensor disposed on the operating end; the additional sensors also include a joint torque sensor or a joint motor current sensor disposed on the operating end; the additional detection data includes contact stress data between the operator and the operating end, angle data, velocity data, acceleration data, and joint driving torque data of the joint; the stress sensor is used to acquire the contact stress data; The joint angle sensor is used to acquire the angle data; The joint velocity sensor is used to acquire the velocity data; the joint acceleration sensor is used to acquire the acceleration data; The joint driving torque data is obtained based on the current fed back by the joint motor current sensor, or the joint driving torque data is obtained based on the detection by the joint torque sensor; The control device is used to determine, based on the image information and the additional detection data, whether the operating terminal generates an unexpected operation according to a preset algorithm. The control device is configured to obtain the calculated torque data of the joint using the angle data, velocity data, and acceleration data; Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end is obtained; The operational state between the operator and the operating end is obtained based on the contact stress data and the external torque data. Based on the image information and the operation status, it is determined whether the operating terminal has been impacted by an external force; if so, it is determined that the operating terminal has been impacted by an external force and has caused an unexpected operation.

8. The unexpected operation detection system according to claim 7, characterized in that, The control device determines the connection state between the operator and the operating end based on the image information and the contact stress data using a preset algorithm. If the connection state is that the operator and the operating end are separated, and the movement speed of the operating end exceeds a preset threshold, it is determined that the operating end has separated from the operator, resulting in an unexpected operation.

9. The unexpected operation detection system according to claim 7, characterized in that, The unexpected operation detection system also includes an alarm device; when the control device determines that the operation terminal has generated an unexpected operation, the alarm device is triggered to display an alarm message.

10. The unexpected operation detection system according to claim 7, characterized in that, The control device is used to communicate with the master-slave mapping module. When the control device determines that the operation terminal has generated an unexpected operation, the control device is used to drive the master-slave mapping module to stop mapping the operation terminal's action to the slave device.

11. An unexpected operation detection system, characterized in that, include: The system includes a control device, a stress sensor, a joint angle sensor, a joint velocity sensor, and a joint acceleration sensor mounted on the operating end; the unexpected operation detection system also includes a joint torque sensor or a joint motor current sensor mounted on the operating end. The stress sensor is used to acquire contact stress data between the operator and the operating end; The joint angle sensor is used to acquire angle data of at least one joint of the operating end; The joint velocity sensor is used to acquire the velocity data of the joint; The joint acceleration sensor is used to acquire the acceleration data of the joint; The joint driving torque data of the joint is obtained based on the current fed back by the joint motor current sensor, or the joint driving torque data of the joint is obtained based on the detection of the joint torque sensor. The control device is configured to obtain calculated torque data of the joint using the angle data, the velocity data, and the acceleration data; Based on the calculated torque data and the joint driving torque data, the external torque data of the joint on the operating end is obtained; based on the contact stress data and the external torque data, the operating state between the operator and the operating end is obtained; If the operating state is such that the operating torque data applied by the operator to the operating end is not equal to the external torque data, it is determined that the operating end has been impacted by an external force, resulting in an unexpected operation.

12. A surgical robot system, comprising a master end device and a slave end device, wherein the master end device includes an operating end, characterized in that, It also includes the unexpected operation detection system according to any one of claims 7 to 11.

13. A readable storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the steps of the unexpected operation detection method according to any one of claims 1 to 6.

14. A computer device, characterized in that, It includes a processor and a readable storage medium according to claim 13, the processor being configured to execute the program stored on the readable storage medium.

Images