Remote multi-host collaborative control system and surgical robot system

Abstract

The present invention provides a remote multi-master collaborative control system and a surgical robot system, the control system includes an adaptive filtering module, configured to use an adaptive filtering algorithm to smooth the data transmitted between each remote master and slave; a delay synchronization module, configured to synchronize the delay differences between different remote masters and slaves; a trajectory prediction module, configured to predict the next action trajectory of the slave according to the historical operation trajectory data of the remote master when a network delay occurs at any remote master; and an instruction conflict arbitration module, configured to dynamically determine the priority of each remote master and process multi-master instruction conflicts based on the priority of each remote master, so as to generate a single valid instruction to be transmitted to the slave. The present invention can not only realize efficient collaborative operation of multiple doctors, but also significantly improve the real-time and accuracy of remote surgery, and effectively reduce the surgical error rate.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and in particular to a remote multi-master collaborative control system and a surgical robot system. Background Art

[0002] In the existing laparoscopic surgical robot system, common technologies (such as the da Vinci surgical robot) use a single doctor's console (master) to remotely operate the surgical trolley (slave). The doctor remotely controls the surgical instruments and endoscopes installed on the slave's robotic arm through the master operator, and the instructions are transmitted to the slave via the network for execution. The system relies on a high-speed network to support real-time performance.

[0003] Some surgical robot systems allow two local doctor consoles to control the slaves at the same time. For example, one local doctor console is responsible for operating surgical instruments, while the other controls the adjustment of the endoscope lens, and collaborative control is achieved through local high-speed communication.

[0004] However, for the remote control solution of a single doctor console, it is susceptible to network delay and fluctuation, and there is a significant timing asynchrony problem between the master and slave ends. Network delay usually varies between 10ms and 300ms, and network jitter is usually around 30ms. Jitter and packet loss further aggravate the lag in command execution, resulting in inconsistency between the robot arm movement and the intention of the main operator, especially in delicate operations such as cutting and suturing, which can easily cause errors and increase surgical risks. In the prior art, simple prediction algorithms (such as linear interpolation) or fixed delay compensation methods cannot adapt to dynamic network environments and lack real-time and robustness. In addition, the design of a single doctor console limits the collaborative operation of multiple doctors. The remote end only supports single-person control, and cannot realize the division of labor and cooperation between the main surgeon and the assistant doctor (such as one person controlling the surgical instrument and one person adjusting the endoscope lens). This leads to inefficiency in complex surgeries and excessive reliance on the skills of a single operator, which limits the efficiency and flexibility of complex surgeries.

[0005] Although the local dual-console solution avoids remote delays, it is only suitable for local collaboration and is not suitable for remote surgery multi-master scenarios. It cannot meet the needs of telemedicine, and the remote end only supports single-person operation. It lacks multi-master access management, command conflict arbitration, and fault safety mechanisms. Command conflicts are prone to occur, and there are no effective safety response measures in the event of network failure or equipment failure, resulting in the risk of surgical interruption and the inability to achieve multi-doctor collaboration.

[0006] It should be noted that the information disclosed in the background technology section of the invention is only intended to deepen the understanding of the general background technology of the invention, and should not be regarded as an admission or suggestion in any form that the information constitutes prior art already known to those skilled in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a remote multi-master collaborative control system and a surgical robot system, which can not only realize efficient collaborative operation of multiple doctors, but also significantly improve the real-time and accuracy of remote surgery, and effectively reduce the error rate of surgery.

[0008] To achieve the above-mentioned purpose, the present invention provides a remote multi-master collaborative control system, comprising: an adaptive filtering module, configured to use an adaptive filtering algorithm to smooth the data transmitted between each remote master and the slave; a delay synchronization module, configured to synchronize the delay differences between different remote masters and the slaves; a trajectory prediction module, configured to predict the next action trajectory of the slave according to the historical operation trajectory data of the remote master when a network delay occurs at any of the remote masters; and an instruction conflict arbitration module, configured to dynamically determine the priority of each of the remote masters and process multi-master instruction conflicts based on the priority of each of the remote masters, so as to generate a single valid instruction for transmission to the slave.

[0009] Optionally, the adaptive filtering module is configured to adaptively adjust filtering parameters in real time according to the network status of each of the remote master ends, and smooth the data transmitted between each of the remote master ends and the slave ends according to the adjusted filtering parameters.

[0010] Optionally, the delay synchronization module is configured to periodically measure the delay between each of the remote master ends and the slave ends so as to calibrate the instruction timestamps of each of the remote master ends, and control the slave ends to execute the instructions sent by each of the remote master ends in sequence according to the order of the calibrated timestamps of the instructions sent by each of the remote master ends.

[0011] Optionally, the delay synchronization module is also configured to remeasure the delay between each remote master end and the slave end if the absolute value of the deviation between the delay between the remote master end and the slave end measured in the current cycle and the delay between the remote master end and the slave end measured in the previous cycle is greater than a preset deviation threshold.

[0012] Optionally, the trajectory prediction module is configured to predict the next action trajectory of the remote master end when a network delay occurs in any of the remote master ends based on a trajectory prediction model trained based on the historical operation trajectory data of the remote master end and the operation trajectory data of the remote master end in a previous preset time period, and predict the next action trajectory of the slave end based on the next action trajectory of the remote master end and the master-slave mapping relationship between the remote master end and the slave end.

[0013] Optionally, the instruction conflict arbitration module is configured to dynamically determine the priority of each of the remote masters according to the authority of each of the remote masters and a pre-acquired priority rule table.

[0014] Optionally, the remote multi-master collaborative control system provided by the present invention also includes a multi-master access management module, which is configured to receive access requests sent by each of the remote masters, and to verify whether the identity of each remote master requesting access is legal through a digital signature, and to dynamically allocate permissions to each remote master with a legal identity based on surgical requirements and the network quality of each of the remote masters.

[0015] Optionally, the remote multi-master cooperative control system provided by the present invention further includes a fault safety response module, which is configured to detect the status of each of the remote masters in real time and take response measures when a fault occurs in any of the remote masters.

[0016] Optionally, the fault safety response module is configured to control the slave end to enter a safety lock state when a fault occurs in any of the remote master ends, and display fault prompt information on the operation page, wherein the fault prompt information includes the fault type and a list of takeover master ends.

[0017] Optionally, the fail-safe response module is configured to synchronize data of each of the remote master ends and the slave ends to a backup master end in real time, and trigger switching upon detecting a failure of any of the remote master ends, so as to switch the failed remote master end to the backup master end.

[0018] Optionally, the remote multi-master collaborative control system provided by the present invention also includes a dynamic network bandwidth allocation module, which is configured to determine the network bandwidth weight of each remote master end according to the network status and instruction requirements of each remote master end, and dynamically adjust the network bandwidth of each remote master end according to the network bandwidth weight of each remote master end.

[0019] In order to achieve the above-mentioned object, the present invention also provides a surgical robot system, including multiple remote master terminals, slave terminals and the remote multi-master terminal collaborative control system described in any one of the above items.

[0020] Compared with the prior art, the remote multi-master collaborative control system and surgical robot system provided by the present invention have the following beneficial effects: the present invention sets an adaptive filtering module and adopts an adaptive filtering algorithm to smooth the data transmitted between each remote master and slave, thereby reducing the influence of network fluctuations on the transmission of command data sent by the remote master and data such as images or forces returned by the slave, thereby ensuring that the mechanical arm and instruments (including surgical instruments and endoscopes) of the slave can accurately move according to the hand movements of the operator of the remote master, and ensuring that the operator of the remote master can accurately perform the next operation according to the data such as images or forces returned by the slave, thereby significantly improving the real-time and accuracy of remote surgery; by setting a delay synchronization module to synchronize the delay differences between different remote master ends and the slave, it can be ensured that multi-master instructions can be The operation is executed at the correct timing on the slave end, which effectively improves the coordination and accuracy of the operation. By setting a trajectory prediction module, when a network delay occurs on any of the remote master ends, the next action trajectory of the slave end is predicted according to the historical operation trajectory data of the remote master end, which can avoid the impact of network delay or fluctuation on the operation, ensure the accuracy of command transmission, provide reliable support for multi-master collaborative control, and effectively improve the stability of the operation. By setting an instruction conflict arbitration module to dynamically judge the priority of each remote master end and handle multi-master command conflicts based on the priority of each remote master end, so as to generate a single valid instruction and transmit it to the slave end, it can support multiple operators to operate efficiently and collaboratively, solve the command conflict problem during multi-master operation, ensure that the slave end receives a single valid instruction, effectively improve the coordination and safety of remote multi-master surgery scenes, and meet the needs of complex minimally invasive surgery scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a surgical application scenario of a surgical robot system provided in one embodiment of the present invention.

[0022] Figure 2 A schematic diagram of the structure of a master end provided in one embodiment of the present invention.

[0023] Figure 3 A schematic structural diagram of a main operator provided in one embodiment of the present invention.

[0024] Figure 4 This is a schematic structural diagram of a single-arm operating trolley provided in one embodiment of the present invention.

[0025] Figure 5 This is a schematic structural diagram of a multi-arm operating trolley provided in one embodiment of the present invention.

[0026] Figure 6 A schematic diagram of the structure of a slave end provided in one embodiment of the present invention.

[0027] Figure 7 A schematic diagram of the block structure of a remote multi-master collaborative control system provided in one embodiment of the present invention.

[0028] Figure 8 A flowchart of the adaptive filtering module provided in one embodiment of the present invention.

[0029] Fig. 9 A flowchart of a delayed synchronization module provided in accordance with an embodiment of the present invention.

[0030] Fig.10 A flowchart of a trajectory prediction module provided in accordance with an embodiment of the present invention.

[0031] Fig.11 A flowchart of a command conflict arbitration module provided in accordance with an embodiment of the present invention.

[0032] Fig.12 A flowchart of a multi-host access management module provided in one embodiment of the present invention.

[0033] Fig.13 A flowchart of a fail-safe response module provided in accordance with an embodiment of the present invention.

[0034] Fig.14 This is a workflow diagram of a dynamic network bandwidth allocation module provided in one embodiment of the present invention.

[0035] Fig.15 A schematic diagram of the block structure of a remote master terminal provided in one embodiment of the present invention.

[0036] Fig.16 A schematic diagram of the block structure of a slave end provided in one embodiment of the present invention.

[0037] The reference numerals are as follows: local master end 100; adjustment component 110; master manipulator 120; first joint 121; second joint 122; third joint 123; fourth joint 124; fifth joint 125; sixth joint 126; terminal joint 127; trolley component 130; display device 140; slave end 200; robot arm 210; surgical instrument 220; endoscope 230; slave control unit 240; visual capture unit 250; image trolley 300; tool trolley 4 00; anesthesia machine-500; remote master-600; first remote master-600A; second remote master-600B; main control unit-610; doctor observation window-620; remote communication unit-630; remote multi-master collaborative control system-700; adaptive filtering module-710; delay synchronization module-720; trajectory prediction module-730; instruction conflict arbitration module-740; multi-master access management module-750; fault safety response module-760; dynamic network bandwidth allocation module-770; backup master-800. DETAILED DESCRIPTION

[0038] The remote multi-master collaborative control system and surgical robot system proposed by the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will become clearer. It should be noted that the accompanying drawings are in a very simplified form and are not in precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the present invention.

[0039] The core idea of ​​the present invention is to provide a remote multi-master collaborative control system and surgical robot system, which can not only realize efficient collaborative operation of multiple doctors, but also significantly improve the real-time and accuracy of remote surgery, and effectively reduce the error rate of surgery.

[0040] It should be noted that, as can be understood by those skilled in the art, the remote multi-master collaborative control system provided by the present invention can be deployed on the local master, or on the slave, or locally independently of the local master and slave. It should also be noted that each robotic arm of the slave has a one-to-one corresponding master-slave control relationship with each remote master (that is, each remote master controls a robotic arm of the slave and the surgical instrument or endoscope mounted on the robotic arm).

[0041] Please refer to Figure 1 , which is a schematic diagram of a surgical application scenario of a surgical robot system provided by an embodiment of the present invention. Figure 1As shown, the surgical robot system includes a local master terminal 100 (local doctor console), a slave terminal 200 (surgical trolley), an imaging trolley 300, a tool trolley 400, and a remote master terminal 600 (remote doctor console); the local master terminal 100 and the remote master terminal 600 are provided with a master operator 120. The slave terminal 200 has at least two robotic arms 210 (see Figure 5 ), surgical instrument 220 (see Figure 6 ) and endoscope 230 (see Figure 6 ) can be mounted on the robot arm 210 respectively. The operator (for example, a surgeon) remotely operates the master operator 120 of the local master terminal 100 / remote master terminal 600 to perform minimally invasive surgery on the patient on the bed. Among them, the master operator 120 and the robot arm 210 and the surgical instruments 220 and endoscope 230 mounted on the robot arm 210 form a master-slave control relationship. Specifically, the robot arm 210 and the surgical instruments 220 and endoscope 230 mounted on the robot arm 210 move according to the movement of the master operator 120 during the operation, that is, according to the operation of the operator's hand. Furthermore, the master operator 120 also receives the force information of the human tissues and organs on the surgical instruments 220 and feeds it back to the operator's hand, so that the operator can feel the surgical operation more intuitively. The local master end 100 / remote master end 600 both have a display device 140, which is connected to the endoscope 230 mounted on the robot arm 210 of the slave end 200 for communication, and can receive and display images collected by the endoscope 230. The operator controls the movement of the robot arm 210 and the surgical instrument 220 and the endoscope 230 mounted on the robot arm 210 through the main operator 120 according to the images displayed on the display device 140 on the local master end 100 / remote master end 600. The endoscope 230 and the surgical instrument 220 enter the patient's position through the wound on the patient's body respectively.

[0042] Please continue to refer to Figure 1 ,like Figure 1 As shown, the surgical robot system also includes auxiliary components such as a ventilator and an anesthesia machine 500 for use in surgery. It should be noted that those skilled in the art can select and configure these auxiliary components according to actual needs, and no further description will be given here.

[0043] Please continue to refer to Figure 2 , which is a schematic diagram of the structure of the master end provided by one embodiment of the present invention. Figure 2As shown, the local master end 100 / remote master end 600 both include an adjustment component 110, a main operator 120, a trolley component 130, and a display device 140. The main operator 120 detects the operator's hand motion information through the control handle at its end as the motion control input of the entire system. The trolley component 130 is a basic bracket for installing other components. The trolley component 130 has movable casters and can be moved or fixed as needed. A foot switch is also installed on the trolley component 130 to detect the switch control signal sent by the operator. The adjustment component 110 can electrically adjust the position of the main operator 120, the display device 140, the operator's armrest and other devices, that is, the adjustment component 110 has a human-machine parameter adjustment function. The display device 140 can provide the operator with the surgical picture (intracavitary picture) detected by the endoscope 230, and provide the operator with reliable image information for performing surgical operations. During the operation, the operator sitting in front of the local master terminal 100 / remote master terminal 600 is located outside the sterilization area. The operator controls the surgical instrument 220 and the endoscope 230 by operating the control handle at the end of the master operator 120. The operator observes the surgical screen transmitted back through the display device 140, and controls the movement of the mechanical arm 210 and instruments (including the surgical instrument 220 and the endoscope 230) of the slave terminal 200 with both hands to complete various operations, thereby achieving the purpose of performing surgery on the patient. At the same time, the operator can control some actions through the foot switch, such as completing the relevant operation inputs such as electrocuting and electrocoagulation through the foot switch.

[0044] Please continue to refer to Figure 3 , which is a schematic diagram of the structure of the main operator 120 provided in one embodiment of the present invention. Figure 3 As shown, the master manipulator 120 includes a first joint 121, a second joint 122, a third joint 123, a fourth joint 124, a fifth joint 125, a sixth joint 126 and a terminal joint 127 which are connected in sequence, wherein the first joint 121, the second joint 122 and the third joint 123 are position joints, the fourth joint 124 is a redundant follow-up joint, and the fifth joint 125, the sixth joint 126 and the terminal joint 127 are posture joints. The first joint 121, the second joint 122 and the third joint 123 and other position joints reflect the position change of the terminal of the master manipulator 120, and the axes of the fifth joint 125, the sixth joint 126 and the terminal joint 127 intersect at one point, which is the position Cartesian terminal point. The movement of the fifth joint 125, the sixth joint 126 and the terminal joint 127 will not affect the change of the terminal position of the master manipulator 120 (i.e., the position Cartesian terminal point), but will only affect the change of the terminal posture of the master manipulator 120.

[0045] Furthermore, the slave end 200 may be composed of multiple Figure 4(which is a structural schematic diagram of a single-arm surgical trolley provided by one embodiment of the present invention) is composed of a single-arm surgical trolley. Of course, the slave end 200 can also be as follows Figure 5 (which is a structural schematic diagram of a multi-arm surgical trolley provided by one embodiment of the present invention) The multi-arm surgical trolley shown in FIG. Each mechanical arm 210 of the slave end 200 can carry different types of surgical instruments 220 or endoscopes 230 to respond to operation signals of the local master end 100 / remote master end 600.

[0046] Please continue to refer to Figure 6 , which is a schematic diagram of the structure of the slave end 200 provided in one embodiment of the present invention. Figure 6 As shown, the endoscope 230 can be loaded on one of the mechanical arms 210 of the slave end 200, and the mechanical arm 210 has multiple degrees of freedom for controlling the movement of the endoscope 230. The endoscope 230 is mainly used to collect surgical images (including human tissues and organs, surgical instruments 220, etc.) and transmit them to the display device 140 on the local master end 100 / remote master end 600 for display. In addition, the slave end 200 also has other mechanical arms 210 for controlling the movement of the surgical instruments 220.

[0047] Please continue to refer to Figure 7 , which is a block diagram of a remote multi-master cooperative control system provided by an embodiment of the present invention. Figure 7 As shown, the remote multi-master collaborative control system 700 provided by the present invention includes: an adaptive filtering module 710, configured to use an adaptive filtering algorithm to smooth the data transmitted between each remote master end 600 and the slave end 200; a delay synchronization module 720, configured to synchronize the delay differences between different remote master ends 600 and the slave ends 200; a trajectory prediction module 730, configured to predict the next action trajectory of the slave end 200 according to the historical operation trajectory data of the remote master end 600 when a network delay occurs in any of the remote master ends 600; and an instruction conflict arbitration module 740, configured to dynamically determine the priority of each of the remote master ends 600 and process multi-master instruction conflicts based on the priority of each of the remote master ends 600, so as to generate a single valid instruction to be transmitted to the slave end 200.

[0048] Therefore, the present invention can reduce the influence of network fluctuations on the transmission of command data sent by the remote master end 600 and data such as images or forces returned by the slave end 200 by setting an adaptive filtering module 710 and using an adaptive filtering algorithm to smooth the data transmitted between each remote master end 600 and the slave end 200, thereby ensuring that the robot arm 210 and instruments (including surgical instruments 220 and endoscope 230) of the slave end 200 can accurately move according to the hand movement of the operator of the remote master end 600, and can ensure that the operator of the remote master end 600 can accurately perform the next operation according to the data such as images or forces returned by the slave end 200, thereby significantly improving the real-time and accuracy of remote surgery; by setting a delay synchronization module 720 to synchronize the delay differences between different remote master ends 600 and the slave end 200, it can ensure that multi-master end instructions can be executed at the slave end 200 in the correct timing. The coordination and accuracy of the surgery can be effectively improved; by setting the trajectory prediction module 730, when any of the remote master terminals 600 has a network delay, the next action trajectory of the slave terminal 200 is predicted according to the historical operation trajectory data of the remote master terminal 600, so as to avoid the influence of network delay or fluctuation on the surgery, ensure the accuracy of command transmission, provide reliable support for multi-master collaborative control, and effectively improve the stability of the surgery; by setting the command conflict arbitration module 740 to dynamically determine the priority of each remote master terminal 600 and process the multi-master command conflict based on the priority of each remote master terminal 600, so as to generate a single valid command and transmit it to the slave terminal 200, it can support multiple operators to operate efficiently and collaboratively, solve the command conflict problem during multi-master operation, ensure that the slave terminal 200 receives a single valid command, effectively improve the coordination and safety of remote multi-master surgery scenes, and meet the needs of complex minimally invasive surgery scenes.

[0049] In some exemplary embodiments, the adaptive filtering module 710 is configured to adaptively adjust filtering parameters in real time according to the network status of each remote master end 600, and smooth the data transmitted between each remote master end 600 and the slave end 200 according to the adjusted filtering parameters.

[0050] Due to network fluctuations (packet loss, jitter, etc.), noise or interruption may occur in the command data sent by the remote master end 600 or the image or force data returned by the slave end 200, thereby causing unstable movement of the robotic arm 210 and the surgical instrument 220 or endoscope 230 mounted on the robotic arm 210. Therefore, by adaptively adjusting the filtering parameters (such as process noise covariance and measurement noise covariance) according to the network status of each remote master end 600 monitored in real time, the accuracy and stability of the data transmitted between each remote master end 600 and the slave end 200 can be ensured, and the robotic arm 210 and instruments (including surgical instruments 220 and endoscope 230) of the slave end 200 can be ensured to move accurately following the hand movement of the operator of the remote master end 600 and feedback the force to the remote master end 600.

[0051] Please continue to refer to Figure 8 , which is a flowchart of the adaptive filtering module 710 provided in one embodiment of the present invention. Figure 8 As shown, the workflow of the adaptive filtering module 710 starts with receiving network data (including command data, image data, force data, etc.); after detecting network fluctuations, adjusting the filtering parameters; then processing the data according to the adjusted filtering parameters, and finally outputting a stable signal to the slave end 200 or the remote master end 600, thereby ensuring that the robotic arm 210 and instruments (including surgical instruments 220 and endoscope 230) of the slave end 200 can accurately move according to the operator's hand movements and feedback the force to the operator.

[0052] Furthermore, the adaptive filtering module 710 can use the Kalman filtering algorithm to smooth the data transmitted between each of the remote master end 600 and the slave end 200. Specifically, the Kalman filtering algorithm is divided into a prediction phase and an update phase. The prediction phase predicts the current state based on the state estimation and state transition model of the previous moment, and calculates the predicted covariance. The update phase uses the new measurement value to correct the predicted value, and balances the weights of the predicted value and the measured value by calculating the Kalman gain. And the Kalman gain depends on the process noise covariance Q and the measurement noise covariance R. These two parameters determine the trust of the filter: a large Q indicates high uncertainty in the system model, and a large R indicates unreliable measurement data.

[0053] Specifically, the state transition model is as follows:

[0054]

[0055] Among them, X t is the state vector (e.g., position coordinates); A is the state transfer matrix (e.g., identity matrix, assuming uniform motion); w t is the process noise, which obeys the normal distribution and has a covariance of Q t .

[0056] The measurement model is as follows:

[0057]

[0058] Among them, Z t is the measurement value (the original data received by the network); H is the observation matrix (usually the unit matrix); v t is the measurement noise, with covariance R t .

[0059] The prediction stage includes state prediction and covariance prediction, where the state prediction formula is as follows:

[0060]

[0061] in, is the predicted value of the state at the current moment; is the estimated value of the state at the previous moment.

[0062] The covariance prediction formula is as follows:

[0063]

[0064] in, is the covariance matrix predicted at the current moment; is the covariance matrix of the previous moment.

[0065] The update phase includes the calculation of Kalman gain, state update and covariance update, where the Kalman gain K t The calculation formula is as follows:

[0066]

[0067] The state update formula is as follows:

[0068]

[0069] in, is the revised optimal state estimate.

[0070] The covariance update formula is as follows:

[0071]

[0072] Among them, P t is the updated covariance matrix, and I is the identity matrix.

[0073] Furthermore, the following formula can be used to adaptively adjust the process noise covariance Q according to the updated covariance matrix: t :

[0074]

[0075] Among them, Q 0 is the initial process noise covariance; α, β are adjustment coefficients (e.g. α=0.5, β=0.3); J t It is a network status indicator (such as jitter amplitude, packet loss rate, etc.).

[0076] Furthermore, the following formula can be used to adaptively adjust the measurement noise covariance R according to the updated covariance matrix: t :

[0077]

[0078] Among them, R 0 is the initial measurement noise covariance.

[0079] In some exemplary embodiments, the delay synchronization module 720 is configured to periodically measure the delay between each of the remote master ends 600 and the slave end 200 to calibrate the instruction timestamps of each of the remote master ends 600, and control the slave end 200 to execute the instructions sent by each of the remote master ends 600 in sequence according to the order of the calibrated timestamps of the instructions sent by each of the remote master ends 600.

[0080] Due to differences in network paths, bandwidth, and load, the delays from each remote master end 600 to the slave end 200 may be different, resulting in inconsistent instruction arrival times, thereby affecting the coordinated operation of the robot arm 210 and instruments (including surgical instruments 220 and endoscopes 230). The present invention periodically measures the delay between each remote master end 600 and the slave end 200, and calibrates the instruction timestamp of the remote master end 600 according to the delay between the remote master end 600 and the slave end 200 for each remote master end 600, so as to dynamically calibrate the instruction timestamp, thereby achieving delay synchronization. By controlling the slave end 200 to sequentially execute the instructions sent by each remote master end 600 according to the order of the calibrated timestamps of the instructions sent by each remote master end 600, it can ensure that the multi-master end instructions are executed in the correct timing at the slave end 200, effectively improving the coordination and accuracy of remote multi-master end surgery.

[0081] Please continue to refer to Fig. 9 , which is a flowchart of the delayed synchronization module 720 provided in one embodiment of the present invention. Fig. 9As shown, for each remote master end 600, the round-trip delay between the remote master end 600 and the slave end 200 (the time difference between the remote master end 600 sending the detection packet and receiving the detection packet returned by the slave end 200) is periodically measured; based on the round-trip delay between the remote master end 600 and the slave end 200, the one-way delay between the remote master end 600 and the slave end 200 is calculated, and based on the local clock, the instruction timestamp of the remote master end 600 is calibrated according to the one-way delay between the remote master end 600 and the slave end 200; finally, the instructions of each remote master end 600 are stored in the cache queue in the order of the calibrated timestamps to adjust the execution timing of the slave end 200, thereby achieving delay synchronization.

[0082] Specifically, for each remote master end 600, the round-trip delay RTT between the remote master end 600 and the slave end 200 can be calculated according to the following formula: k :

[0083]

[0084] Among them, T send,k The time when the remote master 600 sends the detection packet; T recv,k It is the time when the remote master end 600 receives the detection packet returned by the slave end 200.

[0085] For each remote master end 600, the one-way delay D between the remote master end 600 and the slave end 200 can be calculated according to the following formula: k :

[0086]

[0087] For each remote master end 600, the command timestamp of the remote master end 600 can be calibrated according to the following formula:

[0088]

[0089] Among them, T ’ k,t is the calibrated instruction timestamp; T slave It is the timestamp of the instruction sent by the remote master end 600 being received locally.

[0090] It should be noted that, since the remote multi-master cooperative control system 700 provided by the present invention is deployed locally, the delay between the remote multi-master cooperative control system 700 and the slave 200 can be ignored, that is, T slave It is the timestamp when the remote multi-master cooperative control system 700 receives the instruction sent by the remote master 600.

[0091] In some exemplary embodiments, the delay synchronization module 720 is further configured to, for each remote master end 600, remeasure the delay between the remote master end 600 and the slave end 200 if the absolute value of the deviation between the delay between the remote master end 600 and the slave end 200 measured in the current cycle and the delay between the remote master end 600 and the slave end 200 measured in the previous cycle is greater than a preset deviation threshold.

[0092] Therefore, by triggering re-measurement when the delay fluctuation exceeds a preset deviation threshold (for example, 50ms), the accuracy of subsequent instruction timestamp calibration can be effectively guaranteed by re-measuring the delay between the remote master end 600 and the slave end 200 when the network suddenly fluctuates (for example, an instantaneous high packet loss rate).

[0093] In some exemplary embodiments, the trajectory prediction module 730 is configured to predict the next action trajectory of the remote master end 600 based on a trajectory prediction model trained based on the historical operation trajectory data of the remote master end 600 and the operation trajectory data of the remote master end 600 in a previous preset time period when a network delay occurs in any of the remote master ends 600, and predict the next action trajectory of the slave end 200 based on the next action trajectory of the remote master end 600 and the master-slave mapping relationship between the remote master end 600 and the slave end 200.

[0094] Therefore, for each remote master end 600, a corresponding trajectory prediction model is trained based on the historical operation trajectory data of the remote master end 600, which can optimize the prediction ability of the trajectory prediction model and further ensure the accuracy of the predicted next action trajectory of the slave end 200, so as to effectively overcome network delay.

[0095] Please continue to refer to Fig.10 , which is a workflow diagram of the trajectory prediction module 730 provided in one embodiment of the present invention. Fig.10 As shown, for each remote master end 600, the historical operation trajectory data of the remote master end 600 is first collected and divided into multiple training samples, each training sample X includes the trajectory sequence of the remote master end 600, that is, X={x t-n , …, x t-1}, using the training samples obtained based on the historical operation trajectory data of the remote master end 600 to train the LSTM model (Long Short-Term Memory Network Model); in actual surgery, input the operation trajectory data (current operation data) of the remote master end 600 in the previous preset time period {x' t-n , …, x' t-1}, forward propagation through the trained LSTM model to predict the next action trajectory of the remote master end 600 It should be noted that, as can be understood by those skilled in the art, the present invention does not limit the specific value of n.

[0096] Specifically, the LSTM model retains long-term dependencies and updates short-term memory through the forget gate, input gate, and output gate mechanisms, and is suitable for continuous action prediction in surgical operations.

[0097] The expression of the forget gate is as follows:

[0098]

[0099] Among them, f t is the output of the forget gate; σ is the sigmoid function; W f is the weight matrix; h t-1 is the hidden state of the previous moment; x t is the input at the current moment; b f is the bias term.

[0100] The expression of the input gate is as follows:

[0101]

[0102] Among them, i t is the input gate output; W i is the weight matrix; b i is the bias term.

[0103] The cell state update formula is as follows:

[0104]

[0105]

[0106] in, is the candidate cell state; W C is the weight matrix; b C is the bias term; C t-1 is the cell state at the previous moment.

[0107] The expression of the output gate is as follows:

[0108]

[0109]

[0110] Among them, t Output gate output; W o is the weight matrix; b o is the bias term; C t is the current cell state.

[0111] The trajectory prediction formula is as follows:

[0112]

[0113] Among them, W h is the weight matrix; b h is the bias term.

[0114] The loss function used in the LSTM model training process is as follows:

[0115]

[0116] Among them, L is the loss value; m is the number of samples; is the predicted value of the i-th sample; x t,i is the true value of the i-th sample.

[0117] In some exemplary embodiments, the instruction conflict arbitration module 740 is configured to dynamically determine the priority of each of the remote masters 600 according to the authority of each of the remote masters 600 and a pre-acquired priority rule table.

[0118] Thus, the accuracy of the determined priorities of the remote master terminals 600 can be effectively guaranteed by using the authority of each remote master terminal 600 and the pre-acquired priority rule table. Specifically, the priority corresponding to each remote master terminal 600 with different authority is recorded in the priority rule table, for example, the priority of the remote master terminal 600 of the primary surgeon is higher than that of the auxiliary remote master terminal 600.

[0119] Please continue to refer to Fig.11 , which is a flowchart of the instruction conflict arbitration module 740 provided in one embodiment of the present invention. Fig.11 As shown, taking the first remote master end 600A (main knife) and the second remote master end 600B (auxiliary) as an example, the first remote master end 600A (main knife) and the second remote master end 600B (auxiliary) simultaneously send instructions to the instruction conflict arbitration module 740, wherein the first remote master end 600A (main knife) issues a cutting instruction, and the second remote master end 600B (auxiliary) issues a lens adjustment instruction. The instruction conflict arbitration module 740 processes multi-master end instruction conflicts based on a priority rule table (e.g., the priority of the main knife is higher than that of the auxiliary), and finally outputs the arbitrated instruction (i.e., the instruction of the remote master end 600 with the highest priority, such as the instruction of the first remote master end 600A) to the slave end 200.

[0120] Please continue to refer to Figure 7 ,like Figure 7As shown, in some exemplary embodiments, the remote multi-master collaborative control system 700 provided by the present invention also includes a multi-master access management module 750, which is configured to receive access requests sent by each of the remote masters 600, and verify whether the identity of each remote master 600 requesting access is legal through a digital signature, and dynamically allocate permissions to each remote master 600 with a legal identity based on surgical requirements and the network quality of each of the remote masters 600.

[0121] Therefore, by using digital signatures to verify the identity of each remote master terminal 600 requesting access, access security can be effectively ensured. In addition, by dynamically allocating permissions to each remote master terminal 600 with a legitimate identity according to surgical requirements and the network quality of each remote master terminal 600, it is possible to support multiple masters (such as the main surgeon and the assistant) to simultaneously control the surgical instrument 220 and the endoscope 230, divide the work and cooperate, so as to break through the limitation of a single operation, avoid command conflicts, improve the efficiency of complex surgeries, optimize multi-master collaboration, and provide a safe and efficient management mechanism for remote minimally invasive surgery and teaching.

[0122] Please continue to refer to Fig.12 , which is a flowchart of the multi-host access management module 750 provided in one embodiment of the present invention. Fig.12 As shown, first, the remote master end 600 sends an access request; then, the multi-master end access management module 750 verifies the identity of the remote master end 600 through a digital signature; after confirming the identity of the remote master end 600, the multi-master end access management module 750 allocates the authority of the remote master end 600 (such as master, auxiliary); finally, the integrated instructions are sent to the slave end 200.

[0123] Please continue to refer to Figure 7 ,like Figure 7 As shown, in some exemplary embodiments, the remote multi-master collaborative control system 700 provided by the present invention also includes a fault safety response module 760, which is configured to detect the status of each of the remote master ends 600 in real time and take response measures when any of the remote master ends 600 fails.

[0124] Therefore, by setting up the fail-safe response module 760, when any remote master end 600 fails, timely response measures can be taken, thereby effectively improving the safety of the operation. Specifically, the fail-safe response module 760 can detect the status of each remote master end 600 in real time based on a heartbeat signal (a periodic status confirmation signal, generally actively sent by the remote master end 600). For example, for each remote master end 600, if the heartbeat signal of the remote master end 600 is not received within a preset time window, it is determined that the remote master end 600 has failed.

[0125] In some exemplary embodiments, the fault safety response module 760 is configured to synchronize the data of each of the remote master ends 600 and the slave ends 200 to the backup master end 800 in real time, and trigger switching upon detecting a failure of any of the remote master ends 600, so as to switch the failed remote master end 600 to the backup master end 800.

[0126] Thus, by synchronizing the data of each remote master end 600 and the slave end 200 to the standby master end 800 in real time, the standby master end 800 can be kept in a hot standby state, so that when any remote master end 600 fails, the failed remote master end 600 can be switched to the standby master end 800 preloaded with the relevant data of the remote master end 600 with zero delay, thereby ensuring that the surgery is carried out seamlessly in the event of a failure, effectively improving the safety of the surgery, solving the problem of switching delay in the prior art, and providing reliable redundancy protection for remote minimally invasive surgery. It should be noted that, as can be understood by those skilled in the art, the standby master end 800 can be deployed locally (that is, the standby master end 800 can be a local master end 100), and of course the standby master end 800 can also be deployed remotely.

[0127] Please continue to refer to Fig.13 , which is a flowchart of the fail-safe response module 760 provided in one embodiment of the present invention. Fig.13 As shown, the fail-safe response module 760 synchronizes data (including relevant data of the remote master end 600 and the slave end 200) to the backup master end 800 in real time; monitors the status of each remote master end 600 through a heartbeat signal; triggers switching when a failure is detected in any remote master end 600; the backup master end 800 takes over the operation so that the master-slave control of the main operator 120 and the robot arm 210 can be continued, and the robot arm 210 performs tasks according to the surgical movements of the operator of the backup master end 800 and feeds back the force to the operator.

[0128] In other exemplary embodiments, the fault safety response module 760 is configured to control the slave end 200 to enter a safety lock state when a fault occurs in any of the remote master ends 600, and display fault prompt information on the operation page, wherein the fault prompt information includes the fault type and a list of takeover master ends.

[0129] Thus, by controlling the slave end 200 to enter a safety lock state when any remote master end 600 fails, the safety of the operation can be effectively guaranteed. By displaying fault prompt information including the fault type and a list of takeover master ends on the operation page, it is convenient for the doctor to select another master end without fault to take over the relevant operations of the remote master end 600 that has failed when any remote master end 600 fails, so as to ensure that the operation can continue.

[0130] Please continue to refer to Figure 7 ,like Figure 7 As shown, in some exemplary embodiments, the remote multi-master collaborative control system 700 provided by the present invention also includes a dynamic network bandwidth allocation module 770, which is configured to determine the network bandwidth weight of each of the remote master ends 600 according to the network status and instruction requirements of each of the remote master ends 600, and dynamically adjust the network bandwidth of each of the remote master ends 600 according to the network bandwidth weight of each of the remote master ends 600.

[0131] Therefore, by setting a dynamic network bandwidth allocation module 770 to dynamically adjust the network bandwidth of each of the remote master terminals 600, the efficient transmission of data (including command data, image data, etc.) can be ensured, and the transmission bottleneck problem of multi-master collaboration when network resources are limited can be effectively solved, and the real-time transmission of data (including command data, image data, etc.) can be effectively guaranteed. In addition, the stability and accuracy of remote operation can be effectively improved by dynamically allocating bandwidth, providing reliable network support for complex minimally invasive surgery and doctor collaboration. In addition, by determining the network bandwidth weight of each of the remote master terminals 600 according to the network status and command requirements of each of the remote master terminals 600, the transmission of key data (such as surgeon commands and high-definition images) can be prioritized to avoid the impact of network congestion on surgery, and provide stable communication support for remote surgery.

[0132] Please continue to refer to Fig.14 , which is a flowchart of the dynamic network bandwidth allocation module 770 provided in one embodiment of the present invention. Fig.14 As shown, first monitor the network status of each remote master end 600 (including bandwidth, delay and other parameters); then evaluate the instruction requirements of each remote master end 600 (such as the real-time requirements of the main operator 120); then calculate the bandwidth allocation weight according to the requirements (such as the main surgeon's instruction priority); finally dynamically adjust the bandwidth allocation of each remote master end 600 to ensure the efficient transmission of instructions and endoscopic images. After the transmission is completed, feedback the network performance to optimize the subsequent allocation.

[0133] Based on the same inventive concept, the present invention also provides a surgical robot system, which includes multiple remote master terminals 600, slave terminals 200, and the remote multi-master terminal collaborative control system 700 described in any one of the above. Since the surgical robot system provided by the present invention includes the remote multi-master terminal collaborative control system 700 provided by the present invention, the surgical robot system provided by the present invention at least has all the beneficial effects of the remote multi-master terminal collaborative control system 700 provided by the present invention. For details, please refer to the relevant description above, which will not be repeated here.

[0134] In some exemplary embodiments, the surgical robot system provided by the present invention further includes a backup master terminal 800 .

[0135] Please continue to refer to Fig.15 , which is a block diagram of a remote master terminal 600 provided in one embodiment of the present invention. Fig.15 As shown, the remote master end 600 includes a main control unit 610, a doctor's observation window 620 and a remote communication unit 630. The main control unit 610 is used to sense the operator's hand movements and identify the operator's operation intentions. The doctor's observation window 620 is used to present the surgical screen (endoscopic image). The remote communication unit 630 is used to send instructions to the local end (remote multi-master end collaborative control system 700) and communicate with it.

[0136] Please continue to refer to Fig.16 , which is a schematic block diagram of the slave terminal 200 provided in one embodiment of the present invention. Fig.16 As shown, the slave end 200 includes a slave control unit 240 and a visual capture unit 250, wherein the slave control unit 240 is used to receive instructions from the main control unit 610, execute movements after processing, and control the robotic arm 210 and instruments (including surgical instruments 220 and endoscope 230) to replicate the master end operation; the visual capture unit 250 (such as endoscope 230) is used to capture surgical images.

[0137] In summary, compared with the prior art, the remote multi-master collaborative control system 700 and surgical robot system provided by the present invention have the following beneficial effects: (1) The present invention can effectively deal with network delay and fluctuation problems through the adaptive filtering module 710 and the trajectory prediction module 730, and can ensure the consistency of multi-master command timing through the delay synchronization module 720, thereby significantly improving the real-time and accuracy of remote surgery. Among them, the adaptive filtering module 710 can smooth the master operator 120 command and endoscopic image data, reducing the impact of jitter and packet loss; the trajectory prediction module 730 generates the slave end 200 (mechanical arm 210) motion instructions in advance according to historical data, thereby overcoming network delay. Compared with the single delay compensation method in the prior art, the present invention can still maintain the real-time synchronization of the slave end 200 (mechanical arm 210) and the master end (master operator 120) in a dynamic network environment (such as 5G or satellite communication), making fine operations such as cutting and suturing more accurate, and significantly reducing the surgical error rate.

[0138] (2) The present invention breaks through the limitation that the traditional remote surgery system only supports a single operator through the multi-master access management module 750 and the command conflict arbitration module 740. Multiple doctors (such as the main surgeon and the assistant surgeon) can control the surgical instruments 220 and the endoscope 230 simultaneously through different remote master terminals 600, and complete complex surgical tasks through division of labor and cooperation. For example, the main surgeon is responsible for the operation of the instruments, and the assistant surgeon adjusts the camera angle. The commands are executed in an orderly manner after arbitration to avoid conflicts. This collaborative mode can improve surgical efficiency and shorten surgical time. It is particularly suitable for high-difficulty minimally invasive surgeries (such as multi-organ joint operations) and can reduce the reliance on the skills of a single operator.

[0139] (3) By introducing the backup master terminal 800 and the fault safety response module 760, the reliability of the remote surgery system can be greatly improved, and the safety and continuity of the surgical process can be enhanced. Since the backup master terminal 800 synchronizes the data of the remote master terminal 600 in real time, when a fault (such as power outage or network disconnection) occurs, zero-delay switching is triggered to ensure that the surgery is not interrupted. Or when a fault occurs, the slave terminal 200 is controlled to enter a safe lock state, and clear fault prompt information is provided on the operation page, so that the doctor can quickly switch to other master terminals to continue the operation. Compared with the non-redundant design of the prior art, the present invention can effectively avoid surgical risks caused by equipment failure and ensure patient safety.

[0140] (4) The present invention can prioritize the transmission of key data (such as surgeon instructions and high-definition images) by setting a dynamic network bandwidth allocation module 770 to dynamically adjust the network bandwidth of each remote master terminal 600, avoid the impact of network congestion on surgery, optimize network resource utilization, and improve the stability of data transmission. Compared with traditional fixed bandwidth allocation, the present invention can still maintain real-time performance and image quality in scenarios with limited bandwidth, providing stable communication support for remote surgery.

[0141] (5) The multi-host collaboration and real-time optimization of the present invention can enable the laparoscopic surgical robot to adapt to distributed medical environments, such as remote collaboration between hospitals and expert teams in remote areas. Doctors do not need to be concentrated in the same location to complete the operation, thereby improving the coverage of high-quality medical resources. At the same time, the fail-safe response module 760 can lower the technical threshold of remote surgery, enhance the confidence of doctors and patients, and promote the popularization and application of remote minimally invasive surgery.

[0142] (6) The multi-host collaborative control design of the present invention provides new possibilities for remote medical education. Experts can guide local doctors through the remote host 600, adjust instructions in real time and receive feedback. Combined with high-definition image transmission, teaching and surgery can be carried out in parallel. This not only improves training efficiency, but also creates conditions for young doctors to improve their skills.

[0143] It should be noted that the above description is only a description of the preferred embodiment of the present invention, and is not any limitation to the scope of the present invention. Any changes and modifications made by ordinary technicians in the field of the present invention based on the above disclosure are within the scope of protection of the present invention.

Claims

1. A remote multi-master collaborative control system, characterized in that: include: An adaptive filtering module configured to use an adaptive filtering algorithm to smooth data transmitted between each remote master end and the slave end; A delay synchronization module, configured to synchronize the delay differences between different remote master ends and slave ends; A trajectory prediction module, configured to predict the next action trajectory of the slave end according to the historical operation trajectory data of the remote master end when a network delay occurs at any of the remote master ends; as well as An instruction conflict arbitration module, configured to dynamically determine the priority of each of the remote master ends and process multi-master end instruction conflicts based on the priority of each of the remote master ends, so as to generate a single valid instruction to be transmitted to the slave end; The remote multi-master collaborative control system also includes a multi-master access management module, which is configured to receive access requests sent by each of the remote masters, and to verify the legitimacy of the identity of each remote master requesting access through a digital signature, and to dynamically allocate permissions to each remote master with a legitimate identity based on surgical requirements and the network quality of each of the remote masters.

2. The remote multi-master cooperative control system according to claim 1, characterized in that: The adaptive filtering module is configured to adaptively adjust the filtering parameters in real time according to the network status of each remote master end, and perform smoothing processing on the data transmitted between each remote master end and the slave end according to the adjusted filtering parameters.

3. The remote multi-master cooperative control system according to claim 1, characterized in that: The delay synchronization module is configured to periodically measure the delay between each of the remote master ends and the slave ends to calibrate the instruction timestamps of each of the remote master ends, and control the slave ends to execute the instructions sent by each of the remote master ends in sequence according to the order of the calibrated timestamps of the instructions sent by each of the remote master ends.

4. The remote multi-host cooperative control system according to claim 3, characterized in that: The delay synchronization module is also configured to remeasure the delay between the remote master end and the slave end for each remote master end if the absolute value of the deviation between the delay between the remote master end and the slave end measured in the current cycle and the delay between the remote master end and the slave end measured in the previous cycle is greater than a preset deviation threshold.

5. The remote multi-master cooperative control system according to claim 1, characterized in that: The trajectory prediction module is configured to predict the next action trajectory of the remote master end according to a trajectory prediction model trained based on the historical operation trajectory data of the remote master end and the operation trajectory data of the remote master end in a previous preset time period when a network delay occurs at any of the remote master ends, and predict the next action trajectory of the slave end according to the next action trajectory of the remote master end and the master-slave mapping relationship between the remote master end and the slave end.

6. The remote multi-master cooperative control system according to claim 1, characterized in that: The instruction conflict arbitration module is configured to dynamically determine the priority of each remote master end according to the authority of each remote master end and a pre-acquired priority rule table.

7. The remote multi-master cooperative control system according to claim 1, characterized in that: It also includes a fault safety response module, which is configured to detect the status of each of the remote master ends in real time and take response measures when any of the remote master ends fails.

8. The remote multi-host cooperative control system according to claim 7, characterized in that: The fault safety response module is configured to control the slave end to enter a safety lock state when any of the remote master ends fails, and display fault prompt information on the operation page, wherein the fault prompt information includes the fault type and a list of takeover master ends.

9. The remote multi-host cooperative control system according to claim 7, characterized in that: The fail-safe response module is configured to synchronize the data of each of the remote master ends and the slave ends to the backup master end in real time, and trigger switching when a failure is detected in any of the remote master ends, so as to switch the failed remote master end to the backup master end.

10. The remote multi-host cooperative control system according to claim 1, characterized in that: It also includes a dynamic network bandwidth allocation module, which is configured to determine the network bandwidth weight of each remote master terminal according to the network status and instruction requirements of each remote master terminal, and dynamically adjust the network bandwidth of each remote master terminal according to the network bandwidth weight of each remote master terminal.

11. A surgical robot system, characterized in that: The invention comprises a plurality of remote master terminals, slave terminals and a remote multi-master terminal cooperative control system as described in any one of claims 1 to 10.

Images