Motion prediction system for telesurgical procedure and surgical robot system

Abstract

The invention provides an action prediction system for remote surgical operation and a surgical robot system. The action prediction system for the remote surgical operation comprises an autonomous motion trail generation unit and an instrument motion trail prediction unit, the autonomous motion trail generation units are respectively deployed at a remote end and a local end; the instrument movement track prediction unit is deployed at a remote end; when the communication delay of the remote end and the local end reaches a trigger condition, the autonomous movement track generation unit is configured to generate autonomous movement tracks at the remote end and the local end respectively based on the current movement state of the mechanical arm at the local end and a prediction model; the instrument movement track prediction unit is configured to generate a real-time operation prediction track based on the current movement state of the mechanical arm of the local end and the operation information of the remote end; the autonomous movement track of the local end is executed by a mechanical arm of the local end; the autonomous motion track and the operation prediction track of the remote end are displayed on the remote end in an overlapping mode.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a motion prediction system for remote surgical operations and a surgical robot system. Background Art

[0002] Existing technologies in the field of remote surgery mainly include remote surgical robot systems and network delay compensation methods based on simple predictions, such as estimating the trajectory of instrument motion through linear interpolation or historical data averages. However, these technologies have significant defects: they are highly dependent on stable high-speed networks. When the network quality is poor (such as delays exceeding 300ms or packet loss rates exceeding 5%), operation instruction delays will cause surgery interruptions or reduced accuracy; simple linear prediction methods cannot adapt to the dynamic changes of doctors' personalized operating habits and complex surgical scenarios, and have low prediction accuracy. It is difficult for doctors to respond quickly to potential risks, limiting the safety and reliability of remote surgery. Summary of the invention

[0003] The purpose of the present invention is to provide a motion prediction system and a surgical robot system for remote surgical operations to solve the problems of poor continuity, low security and reduced accuracy of existing surgical robot systems when encountering poor network quality during remote applications.

[0004] In order to solve the above technical problems, the present invention provides a motion prediction system for remote surgical operation, which comprises an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at a remote end and a local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches a trigger condition: The autonomous motion trajectory generating unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively using a kinematic model based on the current motion state of the robot arm at the local end and the prediction model; The instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory using a kinematic model based on the current motion state of the robot arm at the local end and the operation information of the remote end; The autonomous motion trajectory generated by the local end is used for execution by the robotic arm at the local end; the autonomous motion trajectory and the operation prediction trajectory generated by the remote end are used for superimposed display at the remote end.

[0005] Optionally, the remote surgical operation motion prediction system further includes a deep learning training unit; The deep learning training unit is configured to train the prediction model using pre-collected surgical operation data to generate the prediction model for predicting operation intention.

[0006] Optionally, the deep learning training unit adopts a convolutional neural network or a long short-term memory network combined with a linear regression model for training, and the surgical operation data includes historical operation trajectory data, operation habit data and endoscopic image data of multiple doctors.

[0007] Optionally, the deep learning training unit is also configured to update the parameters of the prediction model using current operation trajectory data when the communication delay between the remote end and the local end does not reach a trigger condition.

[0008] Optionally, the remote surgical operation motion prediction system further includes a visual information constraint unit; The visual information constraint unit is configured to constrain the autonomous motion trajectory in combination with the depth information obtained by the local end so that the autonomous motion trajectory complies with a preset safety range and is within the field of view; wherein the depth information is used to determine the spatial relationship between instruments and tissues in the surgical scene.

[0009] Optionally, the step of constraining the autonomous motion trajectory is implemented based on tissue boundary detection and / or visual field boundary detection.

[0010] Optionally, the remote surgical operation motion prediction system further includes a visual field monitoring and early warning unit; The field of vision monitoring and early warning unit is configured to monitor whether the predicted operation trajectory exceeds the field of vision, and issue a warning message if it exceeds the field of vision.

[0011] Optionally, the kinematic model includes forward kinematics and inverse kinematics; the kinematic model is used to predict the position and angle of the end effector of the robotic arm.

[0012] Optionally, the remote surgical operation motion prediction system further includes a tactile feedback unit; The tactile feedback unit is configured to generate vibration at the main operating arm of the remote end based on the predicted operation trajectory to simulate the contact force between the instrument and the tissue.

[0013] Optionally, the communication delay includes at least one of the delay, bandwidth, throughput, jitter, bit error rate and packet loss rate of communication between the remote end and the local end.

[0014] In order to solve the above technical problems, the present invention also provides a surgical robot system, which includes a remote end, a local end and the motion prediction system for remote surgical operations as described above; the remote end and the local end are configured in a master-slave control relationship.

[0015] In summary, in the remote surgical operation motion prediction system and surgical robot system provided by the present invention, the remote surgical operation motion prediction system includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is deployed at the remote end and the local end respectively; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches the trigger condition: the autonomous motion trajectory generation unit is configured to generate an autonomous motion trajectory at the remote end and the local end respectively based on the current motion state of the robotic arm at the local end and the prediction model, using a kinematic model; the instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory based on the current motion state of the robotic arm at the local end and the operation information of the remote end using a kinematic model; wherein the autonomous motion trajectory generated at the local end is used for execution by the robotic arm at the local end; the autonomous motion trajectory and the operation prediction trajectory generated at the remote end are used for superimposed display at the remote end.

[0016] With this configuration, on the one hand, the autonomous motion trajectory generation unit makes predictions based on the prediction model and integrates with multiple dimensions such as kinematics. Compared with traditional linear predictions, it can capture complex nonlinear operation modes and improve prediction accuracy. In a network delay environment, the robotic arm can perform movements according to the autonomous motion trajectory, improve the continuity, real-time and safety of surgical operations in a network delay environment, optimize the real-time and accuracy of remote surgery, reduce the doctor's need for real-time fine-tuning of surgical instruments in an unstable network environment, and reduce the doctor's operating burden. On the other hand, the superimposed display of the autonomous motion trajectory and the predicted operation trajectory at the remote end improves the transparency of the operation, and the doctor can intervene and adjust at any time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Those skilled in the art will appreciate that the drawings are provided for a better understanding of the present invention, but do not constitute any limitation on the scope of the present invention.

[0018] Figure 1 It is a schematic diagram of a master-slave teleoperated surgical robot system.

[0019] Figure 2 yes Figure 1 A schematic diagram of a master-end control device of a surgical robot system is shown.

[0020] Figure 3 Schematic diagram of the main operating arm of the master end control device.

[0021] Figure 4 Schematic diagram of communication delay of a remotely controlled surgical robot system according to an embodiment of the present invention.

[0022] Figure 5Schematic diagram of a module of a motion prediction system for remote surgical operation according to an embodiment of the present invention.

[0023] Figure 6 Schematic diagram of an autonomous motion trajectory generation unit generating an autonomous motion trajectory according to an embodiment of the present invention.

[0024] Figure 7 It is a schematic diagram of display content of a display device at a remote end according to an embodiment of the present invention.

[0025] Figure 8 It is a schematic diagram of constraining the autonomous motion trajectory to a field of view according to an embodiment of the present invention.

[0026] Fig. 9 It is a schematic diagram of a situation in which the field of vision monitoring and early warning unit of an embodiment of the present invention detects that the predicted operation trajectory exceeds the field of vision.

[0027] Fig.10 It is a schematic diagram of the overall operation step flow chart of the action prediction system for remote surgery operation according to an embodiment of the present invention.

[0028] In the attached figure: 1-autonomous motion trajectory generation unit; 2-instrument motion trajectory prediction unit; 3-deep learning training unit; 4-visual information constraint unit; 5-field of vision monitoring and early warning unit; 6-tactile feedback unit; 10-master control device; 11-master operating arm; 12-joint; 13-motor; 14-force interaction center; 15-display device; 20-slave robot device; 21-robotic arm; 30-image trolley; 40-communication device; 50-remote end; 60-local end. DETAILED DESCRIPTION

[0029] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the drawings is often a part of the actual structure. In particular, the emphasis of each drawing is different, and sometimes different scales are used.

[0030] As used in the present invention, the singular forms "a", "an", "one" and "the" include plural objects, the term "or" is usually used to include the meaning of "and / or", the term "several" is usually used to include the meaning of "at least one", and the term "at least two" is usually used to include the meaning of "two or more". In addition, the terms "first", "second" and "third" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" and "third" may explicitly or implicitly include one or at least two of the features, "one end" and "the other end" as well as "proximal end" and "distal end" generally refer to two corresponding parts, which include not only endpoints. In addition, as used in the present invention, "installed", "connected", "connected", and one element "set" on another element should be understood in a broad sense, usually only indicating that there is a connection, coupling, cooperation or transmission relationship between the two elements, and the connection, coupling, cooperation or transmission between the two elements can be direct or indirect through an intermediate element, and cannot be understood as indicating or implying the spatial position relationship between the two elements, that is, one element can be in any orientation such as inside, outside, above, below or one side of another element, unless the content clearly indicates otherwise. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, directional terms such as above, below, up, down, upward, downward, left, right, etc. are used relative to the exemplary embodiments as shown in the figures, with the upward or upper direction toward the top of the corresponding figure, and the downward or lower direction toward the bottom of the corresponding figure.

[0031] The purpose of the present invention is to provide a remote surgical operation motion prediction system and a surgical robot system to solve the problems of poor continuity, low security and reduced accuracy of the existing surgical robot system when the network quality is poor during remote application. The following is a description with reference to the accompanying drawings.

[0032] First, please refer to Figures 1 to 3, which exemplarily shows a locally controlled master-slave teleoperated surgical robot system, where the master-slave teleoperation mainly refers to the master-end control device 10 and the slave-end robot device 20 being configured in a master-slave control relationship, and the operator (such as a doctor) can realize the drive control of the slave-end robot device 20 by operating the master-end control device 10 through master-slave mapping control. Further, the surgical robot system also includes an image trolley 30, etc. In a specific example, the master-end control device 10 includes a master operating arm 11 (also called a master control arm or a master hand), which includes a plurality of joints 12 and a motor 13. The operator (such as a doctor) can directly operate the master operating arm 11, and the end of the master operating arm 11 is in direct contact with the operator's hand, and the contact point is the force interaction center 14. The interaction force is generated by the motor 13 of each joint 12 of the master operating arm 11. The slave-end robot device 20 includes a plurality of mechanical arms 21, which can be used to mount instruments (such as surgical instruments or other auxiliary instruments) and image acquisition devices (such as endoscopes). The movements of the main operating arm 11 driven by the operator are mapped to the movements of the instruments and image acquisition devices of the robotic arm 21 through master-slave mapping, thereby realizing master-slave mapping operation to perform surgery. Furthermore, the image screen collected by the image acquisition device is fed back and transmitted to the image trolley 30, and after being processed by the image trolley 30, it is transmitted to the display device 15 of the main control device 10 for display for the operator to observe. The operator can operate based on the observed image screen. In application, in addition to forward mapping the operation control information, the main control device 10 and the slave robot device 20 can also reversely map the feedback information, such as compensating the reaction force of the contact between the instrument and the tissue to the main operating arm 11, thereby forming a closed loop from operation to feedback.

[0033] After learning about the locally controlled surgical robotic system, please refer to Figure 4 , which shows a remote-controlled surgical robot system, which includes a remote master control device (hereinafter, for the convenience of description, the remote master control device is referred to as the remote end 50) and a local slave robot device (hereinafter, for the convenience of description, the local slave robot device is referred to as the local end 60). The remote end 50 and Figure 2 The master control device 10 shown can be of the same or similar structure. The remote end 50 and the local end 60 are configured to exchange data via a communication device 40 (such as a server) to achieve remote operation.

[0034] In remote control, due to the addition of the communication device 40, the communication delay between the remote end 50 and the local end 60 includes: the delay t1 from the remote end 50 outputting the control signal to the communication device 40 receiving the signal; the delay t2 from the communication device 40 outputting the control signal to the local end 60 receiving the signal; the delay t3 from the local end 60 executing the action to the local image acquisition device capturing the current image; the delay t4 from the local image trolley 30 outputting the image to the communication device 40 receiving the signal (which may involve the encoding and decoding process); the delay t5 from the communication device 40 outputting the signal to the display device of the remote end 50. It can be seen that the remote-controlled surgical robot system has more data exchange links. Compared with the locally controlled surgical robot system, its communication delay is more easily affected by various factors, resulting in a larger communication delay. In the case of poor network quality (such as high latency and low bandwidth), the real-time nature of the surgical operation is seriously affected in the prior art, which may cause delays in operation instructions, affect surgical accuracy, and even endanger patient safety.

[0035] To resolve this issue, refer to Figure 5 , an embodiment of the present invention provides a remote surgical operation motion prediction system, which is mainly used for a remotely controlled surgical robot system. The remote surgical operation motion prediction system includes an autonomous motion trajectory generation unit 1 and an instrument motion trajectory prediction unit 2; the autonomous motion trajectory generation unit 1 is respectively deployed at a remote end 50 and a local end 60; the instrument motion trajectory prediction unit 2 is deployed at the remote end 50; when the communication delay between the remote end 50 and the local end 60 reaches a trigger condition: the autonomous motion trajectory generation unit 1 is configured to generate an autonomous motion trajectory at the remote end 50 and the local end 60 respectively based on the current motion state and prediction model of the robot arm of the local end 60 using a kinematic model; the instrument motion trajectory prediction unit 2 is configured to generate a real-time operation prediction trajectory based on the current motion state of the robot arm of the local end 60 and the operation information of the remote end 50 using a kinematic model; wherein the autonomous motion trajectory generated by the local end 60 is used for execution by the robot arm of the local end 60; the autonomous motion trajectory and the operation prediction trajectory generated by the remote end 50 are used for superimposed display at the remote end 50.

[0036] Optionally, the prediction model here can be generated by AI, preferably can be obtained by deep learning algorithm training. The prediction model can also be obtained by traditional machine learning methods, or by rule-based and logical construction, and this embodiment does not limit its acquisition method. The communication delay includes at least one of the delay, bandwidth, throughput, jitter, bit error rate and packet loss rate of the communication between the remote end 50 and the local end 60. The trigger condition refers to the condition representing the poor communication condition set by the communication parameter specifically selected corresponding to the communication delay. For example, corresponding to the packet loss rate, the trigger condition can be set to be higher than 5%; corresponding to the delay, the trigger condition can be set to be greater than 300ms, etc. Those skilled in the art can set corresponding trigger conditions for different communication parameters according to the prior art. When the communication delay between the remote end 50 and the local end 60 reaches the trigger condition, it can be considered that the communication condition between the remote end 50 and the local end 60 is poor at this time, and the action prediction system of this embodiment is triggered.

[0037] For ease of understanding, the steps performed by the local end 60 and the remote end 50 are described below respectively.

[0038] Local terminal 60: The autonomous motion trajectory generation unit 1 deployed at the local terminal 60 generates the autonomous motion trajectory T (referring to the set of several expected motion points of the robotic arm after the current moment) using the kinematic model based on the current motion state of the robotic arm of the local terminal 60 and the prediction model, such as Figure 6 Optionally, the kinematic model includes forward kinematics and inverse kinematics; the kinematic model is used to predict the position and angle of the end effector of the robot arm.

[0039] Then the robotic arm executes the autonomous motion trajectory and moves. Since the communication delay reaches the trigger condition, the local end 60 cannot obtain the subsequent operation instructions in real time at this time, and the robotic arm moves according to the autonomous motion trajectory generated by the autonomous motion trajectory generation unit 1. Since the autonomous motion trajectory generation unit 1 performs prediction based on the prediction model and integrates with multiple dimensions such as kinematics, it can capture complex nonlinear operation modes and improve prediction accuracy compared to traditional linear prediction. In a network delay environment, the robotic arm can perform movements according to the autonomous motion trajectory, improve the continuity, real-time and safety of surgical operations in a network delay environment, optimize the real-time and accuracy of remote surgery, reduce the remote operator (doctor)'s need for real-time fine-tuning of surgical instruments in an unstable network environment, and reduce the operating burden of the remote operator (doctor).

[0040] Optionally, the autonomous motion trajectory includes multiple expected motion points. After the multiple expected motion points are generated, they can be smoothed, for example, by using a moving average filter to smooth the autonomous motion trajectory. The window size of the moving average filter is an adjustable parameter, and the window size can be selected as 5, for example. The smoothed autonomous motion trajectory T is as follows Figure 6 shown.

[0041] Remote terminal 50: The autonomous motion trajectory generation unit 1 deployed at the remote terminal 50 generates an autonomous motion trajectory (referring to the expected motion point of the robotic arm after the current moment) using a kinematic model based on the current motion state of the robotic arm at the local terminal 60 and the prediction model.

[0042] Based on the above description, it can be understood that before the communication delay between the remote end 50 and the local end 60 reaches the trigger condition, the communication status between the remote end 50 and the local end 60 meets the requirements. At this time, the remote end 50 and the local end 60 are in a normal master-slave teleoperation control state, and the remote operator controls the remote end 50, and the robot arm of the local end 60 follows the movement. When the communication delay between the remote end 50 and the local end 60 reaches the trigger condition, the position information of the robot arm of the local end 60 at the current moment and the operation status and other information are recorded as the current motion state of the robot arm. It can be understood that since the remote end 50 and the local end 60 are in normal communication before the communication delay reaches the trigger condition, the current motion state of the robot arm can be recorded synchronously at both the remote end 50 and the local end 60. That is, when the action prediction system is triggered, the remote end 50 and the local end 60 have the same initial constraint conditions (referring to the current motion state of the robot arm).

[0043] The autonomous motion trajectory generation unit 1 deployed at the remote end 50 and the autonomous motion trajectory generation unit 1 deployed at the local end 60 have the same initial constraints and use the same prediction model, so the autonomous motion trajectories they generate are also consistent. However, the autonomous motion trajectory generated by the autonomous motion trajectory generation unit 1 at the remote end 50 is not directly used to control the operation of the robot arm of the local end 60, but is used to be displayed on the display device of the remote end 50 for observation by the remote operator (doctor). Please refer to Figure 7 , which shows the situation displayed on the display device of the remote end 50, wherein the autonomous motion trajectory T1 reflects the trajectory of the current operation of the robotic arm of the local end 60.

[0044] The instrument motion trajectory prediction unit 2 deployed at the remote end 50 generates a real-time operation prediction trajectory using a kinematic model based on the current motion state of the robotic arm of the local end 60 and the operation information of the remote end 50. The autonomous motion trajectory and the operation prediction trajectory generated by the remote end 50 are superimposed and displayed at the remote end 50. The operation information of the remote end 50 refers to the actual operation information of the remote operator (doctor). After the communication delay reaches the trigger condition, although the operation information of the remote end 50 cannot be transmitted to the local end 60 in real time, the remote operator (doctor) still continues the action before the communication delay reaches the trigger condition, that is, the remote operator (doctor) is still operating the main operating arm of the remote end 50, thereby generating a series of actual operation information. The instrument motion trajectory prediction unit 2 can generate a real-time operation prediction trajectory based on the current motion state of the robotic arm of the local end 60 before the communication delay reaches the trigger condition, and the actual operation information of the remote operator (doctor) after the communication delay reaches the trigger condition. Please refer to Figure 7 , the operation prediction trajectory T2 reflects the predicted trajectory of the actual operation of the current remote operator (doctor). Optionally, when the autonomous motion trajectory T1 and the operation prediction trajectory T2 are superimposed and displayed, the two trajectories preferably use different colors or line shapes to distinguish them. The legacy display of the autonomous motion trajectory T1 improves the transparency of the operation. The remote operator (doctor) can intervene and correct his own operation at any time according to the displayed autonomous motion trajectory T1 and the operation prediction trajectory T2, so that the two trajectories are as close or overlapped as possible. If the remote operator is an experienced doctor, the overlap between his operation prediction trajectory T2 and the autonomous motion trajectory T1 itself may be very high.

[0045] Preferably, the remote surgical operation motion prediction system further includes a deep learning training unit 3; the deep learning training unit 3 is configured to train the prediction model using pre-collected surgical operation data to generate the prediction model for predicting the operation intention. In some embodiments, the prediction model used by the autonomous motion trajectory generation unit 1 can be trained externally and directly input into the motion prediction system of this embodiment. Preferably, the deep learning training unit 3 can be integrated into the motion prediction system, and the prediction model used by the autonomous motion trajectory generation unit 1 is trained by the motion prediction system.

[0046] Optionally, the deep learning training unit 3 uses a convolutional neural network (CNN) or a long short-term memory network (LSTM) in combination with a linear regression model for training, and the surgical operation data includes historical operation trajectory data, operation habit data, and endoscopic image data of multiple (senior) doctors. The following is an example to illustrate the process of training the prediction model.

[0047] The input parameters of the prediction model can optionally include historical operation trajectory data and endoscopic image data. Historical operation trajectory data: defined as P hist ={P1, P2, …, P n}, where P i =[Pos i ,Ori i ] represents the coordinate point in three-dimensional space, n=50 (sampling frequency is 10 Hz, time span is 5 seconds), P hist Includes historical operation trajectory data of multiple (senior) doctors. Endoscopic image data is recorded as I t , represents the two-dimensional image frame at time t. Furthermore, the historical operation trajectory data is processed by range normalization: P i '=(P i -P min ) / (P max -P min ) Among them, P min and P max They are respectively the minimum and maximum coordinate values ​​in the historical operation trajectory data.

[0048] Taking the long short-term memory network (LSTM) as an example, the deep learning training unit 3 uses the historical operation trajectory data P hist Generate the trajectory P for the next j seconds pred ={P n+1 , P n+2 ,…,P n+m}, where j and m are optional parameters, for example j=2, m=20.

[0049] The state update formula of the long short-term memory network (LSTM) is as follows: i t =σ(W i ·[h t-1 , P t ]+b i ) (Input Gate) f t =σ(Wf·[ h t-1 , P t ]+b f ) (Forget Gate) o t =σ(Wo·[ h t-1 , P t ]+b o ) (Output Gate) c t =f t ⊙c t-1 +i t ⊙tanh(W c ·[ht-1 , P t ]+b c ) (memory unit) h t =o t ⊙tanh(c t ) (Hidden state) Among them, W and b are the weight matrix and bias vector respectively, σ is the sigmoid activation function, ⊙ represents element-by-element multiplication, and h t is the predicted value of the output trajectory point. For other principles of the long short-term memory network (LSTM), please refer to the existing technology and will not be elaborated here.

[0050] Preferably, the deep learning training unit 3 is also configured to update the parameters of the prediction model using the current operation trajectory data when the communication delay between the remote end 50 and the local end 60 does not reach the trigger condition. In one embodiment, the input parameters of the prediction model may include the operation trajectory data of the current remote operator (i.e., the current doctor) in addition to the pre-collected surgical operation data. When the communication delay does not reach the trigger condition, the operation trajectory data of the current remote operator can also be used as a training input parameter of the prediction model and included in the training process. That is, the prediction model can be updated in real time, which can better adapt to and predict the operation intention of the current remote operator.

[0051] Preferably, the remote surgical action prediction system further includes a visual information constraint unit 4; the visual information constraint unit 4 is configured to constrain the autonomous motion trajectory in combination with the depth information obtained by the local terminal 60, so that the autonomous motion trajectory conforms to a preset safety range and is within the field of view; wherein the depth information is used to determine the spatial relationship between the instrument and the tissue in the surgical scene. The depth information can be obtained by a binocular vision acquisition device (such as a binocular endoscope) or a depth sensor. The specific principle can refer to the prior art.

[0052] Please refer to Figure 8, the field of view refers to the visible area of ​​the image acquisition device (such as an endoscope) of the local end 60. In some embodiments, the expected movement point of a part of the unconstrained autonomous motion trajectory T3 may be outside the field of view, and it will be difficult to ensure the safety of the operation. In addition, when the local end 60 starts autonomous movement, the unconstrained autonomous motion trajectory T3 may have unexpected contact with the tissue and cause harm. Therefore, it is necessary to impose certain constraints on the autonomous motion trajectory T3. In this embodiment, through the setting of the visual information constraint unit 4, on the one hand, it combines the endoscopic image data, and on the other hand, it combines the depth information to constrain the autonomous motion trajectory T3. The constrained autonomous motion trajectory T4 can always maintain a certain safety distance from the tissue, and the expected movement point of the autonomous motion trajectory T4 is always within the field of view, which effectively improves the safety of the autonomous movement of the local end 60.

[0053] Optionally, the step of constraining the autonomous motion trajectory is implemented based on tissue boundary detection and / or visual field boundary detection. Tissue boundary detection and visual field boundary detection are described below by way of example.

[0054] Tissue boundary detection: using U-Net network for endoscopic image data I t Perform semantic segmentation and generate tissue mask M t , where M t = 1 indicates tissue area, where M t =0 indicates background.

[0055] Calculate the predicted trajectory point P n+k Projection coordinates on the image (uk, vk) Among them, K is the camera intrinsic parameter matrix and T is the extrinsic parameter matrix.

[0056] Taking the requirement that the distance between the trajectory point and the tissue boundary is dk ≥ 3 mm as an example, if the predicted trajectory point P n+k If the distance to the tissue boundary dk is less than 3 mm, it is adjusted by gradient descent:

[0057] Among them, the distance penalty loss function L=Σk(3-dk) 2 , η is the learning rate. The optimized trajectory P pred =[P n+1 , P n+2 ,…,P n+k ], and then the optimized trajectory P pred 'Converted into instrument control instructions, the instrument moves at an adaptive speed V k Execution: V k = Vmax·exp(-αk k ).

[0058] Vmax is the maximum speed, α is the gain, k k is the trajectory curvature, and all of the above are adjustable parameters.

[0059] Field of view boundary detection: using U-Net for endoscope image data I t Segmentation is performed to generate the field of view boundary mask B t , where B t =1 indicates the area within the boundary, where B t =0 indicates the area outside the boundary.

[0060] Projection calculation, predicting the trajectory point P n+k Projected onto the image plane,

[0061] Among them, K is the camera intrinsic parameter matrix and T is the extrinsic parameter matrix.

[0062] Calculate the distance dk from the projection point (uk, vk) to the edge of the field of view:

[0063] Among them, b is a point on the boundary mask Bt.

[0064] Constraint adjustment: If dk < dsafe (the safety distance dsafe can be set to 5 mm, for example), adjust the trajectory point to keep it within the field of view:

[0065] Among them, bnearest is the nearest point on the boundary, and η is the adjustment step size (such as 0.1).

[0066] If the predicted trajectory point P n+k If the distance to the tissue boundary dk is less than 3 mm, it is adjusted by gradient descent:

[0067] Among them, the distance penalty loss function L=Σk(dsafe-dk) 2 , η is the learning rate. The speed V during trajectory execution k Adjust based on the distance to the edge of the field of view: V k =Vmax·min (1, dk / dsafe). If dk≥dsafe, execute at the maximum speed Vmax. If dk<dsafe, the speed V k Decreases linearly with distance.

[0068] Consider both tissue boundary detection and field of view boundary detection: Tissue Boundary Detection Using U-Net on Endoscopic Images It Perform semantic segmentation to generate the tissue mask M t , where M t (x, y) = 1 represents the tissue area, where M t (x, y) = 0 represents the background.

[0069] Calculate the projected coordinates (uk, vk) of the predicted trajectory point P n+k on the image Define the distance from the tissue boundary:

[0070] where m is a point on the tissue mask M t on it.

[0071] If dtissus,k < safety distance (the safety distance can be set to 3mm, for example), adjust the trajectory point to keep it within the field of view:

[0072] where mnearest is the nearest point on the tissue boundary and η is the adjustment step size (which can be set to 0.1, for example).

[0073] For field of view boundary detection, use U-Net to segment the endoscopic image data I t to generate the field of view boundary mask B t , where B t (x, y) = 1 represents the area inside the boundary, where B t (x, y) = 0 represents the area outside the boundary.

[0074] Calculate the distance from the field of view boundary:

[0075] Constraint adjustment: If dvision,k < dsafe (the safety distance dsafe can be set to 5mm, for example), adjust the trajectory point:

[0076] where bnearest is the nearest point on the tissue boundary and η2 is the adjustment step size (which can be set to 0.1, for example).

[0077] Comprehensively optimize the tissue and field of view constraints, and the loss function is:

[0078] where W1 = 1.0, W2 = 1.5 (the field of view constraint has a higher priority), I is the indicator function, and the corresponding term is 0 if the distance meets the requirements. The speed V during trajectory execution k Comprehensively consider the tissue and field of view distances:

[0079] If dtissue,k ≥ 3 and dvision,k ≥ dsafe, then V k =Vmax; if any distance is less than the threshold, the speed decreases linearly and the minimum value is 0.

[0080] Taking both tissue boundary detection and field of view boundary detection into consideration can ensure that the instrument does not exceed the field of view of the endoscope and does not damage the tissue. This is suitable for endoscopic surgery scenarios that require both safety and visibility.

[0081] Optionally, the remote surgery operation motion prediction system further includes a visual field monitoring and warning unit 5; the visual field monitoring and warning unit 5 is configured to monitor whether the operation prediction trajectory exceeds the visual field range, and issue a warning message if it exceeds the visual field range. Fig. 9 The visual field monitoring and early warning unit 5 is preferably deployed at the remote end 50, and is used to monitor whether the operation prediction trajectory T5 of the remote operator (doctor) exceeds the visual field, and issue a warning message if it exceeds the visual field. The warning message can be issued in the form of text, sound or light. Furthermore, when issuing the warning message, the visual field monitoring and early warning unit 5 also preferably automatically adjusts the movement path of the instrument to maintain the instrument within the visual field.

[0082] Optionally, the remote surgical operation motion prediction system further includes a tactile feedback unit 6, which is configured to generate vibrations at the main operating arm of the remote end 50 based on the operation prediction trajectory to simulate the contact force between the instrument and the tissue. The tactile feedback unit 6 is preferably deployed at the remote end 50, which can apply simulated feedback force to the main operating arm to enhance the operation experience of the remote operator. The specific setting principle can be referred to the prior art and will not be expanded here.

[0083] Finally, please refer to Fig.10 , which shows the overall operation step flow chart of the action prediction system for remote surgery operation of this embodiment. It specifically includes: Step S1: Using pre-collected surgical operation data, the prediction model is trained to generate a prediction model for predicting operation intention.

[0084] Step S2: Determine whether the communication delay between the remote end and the local end reaches the trigger condition; if so, execute step S3, if not, execute step S8.

[0085] Step S3: Based on the current motion state of the robot arm at the local end and the prediction model, the kinematic model is used to generate the autonomous motion trajectory at the remote end and the local end respectively.

[0086] Step S4: In combination with the depth information obtained by the local end, the autonomous motion trajectory is constrained so that the autonomous motion trajectory conforms to a preset safety range and is within the field of view.

[0087] Step S8: Use the current operation trajectory data to update the parameters of the prediction model.

[0088] Here, steps S1 to S4 and S8 are common steps, wherein steps S1, S2 and S8 can be performed at either the remote end or the local end, or they can be performed simultaneously. Steps S3 and S4 are performed at the remote end and the local end, respectively. Subsequent steps S5A to S7A are performed only at the remote end, and step S5B is performed only at the local end.

[0089] Step S5A: Based on the current motion state of the local robot arm and the operation information of the remote end, a real-time operation prediction trajectory is generated using a kinematic model.

[0090] Step S6A: The autonomous motion trajectory and the operation prediction trajectory are superimposed and displayed on the remote end.

[0091] Step S7A: monitor whether the predicted trajectory of the operation exceeds the field of view, and issue a warning message if it exceeds the field of view.

[0092] Step S5B: The local robotic arm executes the autonomous motion trajectory.

[0093] The specific implementation of the above-mentioned operation steps can refer to the previous description of the various functional units of the motion prediction system for remote surgical operations, which will not be repeated here.

[0094] Optionally, each functional unit of the remote surgical operation motion prediction system of this embodiment can be configured as a physical module, which is installed and deployed on the remote end 50 or the local end 60, or can be configured as a non-physical program, which is deployed on the remote end 50 or the local end 60 in the form of software, or can be deployed in the form of a readable storage medium storing the program, and the present invention is not limited to this. Those skilled in the art can deploy the various functions of the remote surgical operation motion prediction system of this embodiment according to the prior art.

[0095] The embodiment of the present invention further provides a surgical robot system, which includes a remote terminal 50, a local terminal 60, and the above-mentioned remote surgical operation motion prediction system; the remote terminal 50 and the local terminal 60 are configured in a master-slave control relationship. The structure and principle of other components of the surgical robot system can be understood by referring to the above and combining with the prior art, and will not be described in detail here.

[0096] In summary, in the remote surgical action prediction system and surgical robot system provided by the present invention, the remote surgical action prediction system includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at the remote end and the local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches the trigger condition: the autonomous motion trajectory generation unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively based on the current motion state of the robot arm at the local end and the prediction model, using the kinematic model; the instrument motion trajectory prediction unit is configured to generate real-time operation prediction trajectories based on the current motion state of the robot arm at the local end and the operation information of the remote end using the kinematic model; wherein the autonomous motion trajectory generated by the local end is used for execution by the robot arm at the local end; the autonomous motion trajectory and the operation prediction trajectory generated by the remote end are used for superimposed display at the remote end. With such a configuration, on the one hand, the autonomous motion trajectory generation unit performs prediction based on the prediction model, and integrates with multiple dimensions such as kinematics, which can capture complex nonlinear operation modes and improve prediction accuracy compared with traditional linear prediction. In a network delay environment, the robotic arm can perform movements according to the autonomous motion trajectory, improving the continuity, real-time and safety of surgical operations in a network delay environment, optimizing the real-time and accuracy of remote surgery, reducing the doctor's need for real-time fine-tuning of surgical instruments in an unstable network environment, and reducing the doctor's operating burden. On the other hand, the superimposed display of the autonomous motion trajectory and the predicted operation trajectory at the remote end improves the transparency of the operation, and the doctor can intervene and adjust at any time.

[0097] It should be noted that the above embodiments can be combined with each other. The above description is only a description of the preferred embodiments of the present invention, and does not limit the scope of the present invention. Any changes and modifications made by ordinary technicians in the field of the present invention based on the above disclosure are within the protection scope of the present invention.

Claims

1. A motion prediction system for remote surgery, characterized in that: It includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at a remote end and a local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches a trigger condition: The autonomous motion trajectory generating unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively using a kinematic model based on the current motion state of the robot arm at the local end and the prediction model; The instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory using a kinematic model based on the current motion state of the robot arm at the local end and the operation information of the remote end; The autonomous motion trajectory generated by the local end is used for execution by the robotic arm at the local end; the autonomous motion trajectory and the operation prediction trajectory generated by the remote end are used for superimposed display at the remote end.

2. The remote surgery motion prediction system according to claim 1, characterized in that: It also includes a deep learning training unit; The deep learning training unit is configured to train the prediction model using pre-collected surgical operation data to generate the prediction model for predicting operation intention.

3. According to the motion prediction system for remote surgical operations according to claim 2, the deep learning training unit adopts a convolutional neural network or a long short-term memory network combined with a linear regression model for training, and the surgical operation data includes historical operation trajectory data, operation habit data and endoscopic image data of multiple doctors.

4. According to the motion prediction system for remote surgical operations according to claim 2, the deep learning training unit is also configured to update the parameters of the prediction model using current operation trajectory data when the communication delay between the remote end and the local end does not reach the trigger condition.

5. The remote surgery motion prediction system according to claim 1, characterized in that: Also included is a visual information constraint unit; The visual information constraint unit is configured to constrain the autonomous motion trajectory in combination with the depth information obtained by the local end so that the autonomous motion trajectory complies with a preset safety range and is within the field of view; wherein the depth information is used to determine the spatial relationship between instruments and tissues in the surgical scene.

6. The remote surgery motion prediction system according to claim 5, characterized in that: The step of constraining the autonomous motion trajectory is implemented based on tissue boundary detection and / or visual field boundary detection.

7. The remote surgery motion prediction system according to claim 1, characterized in that: It also includes a visual field monitoring and early warning unit; The field of vision monitoring and early warning unit is configured to monitor whether the predicted operation trajectory exceeds the field of vision, and issue a warning message if it exceeds the field of vision.

8. The remote surgery motion prediction system according to claim 1, characterized in that: The kinematic model includes forward kinematics and inverse kinematics; the kinematic model is used to predict the position and angle of the end effector of the robotic arm.

9. The remote surgery motion prediction system according to claim 1, characterized in that: Also included is a tactile feedback unit; The tactile feedback unit is configured to generate vibration at the main operating arm of the remote end based on the predicted operation trajectory to simulate the contact force between the instrument and the tissue.

10. The remote surgery motion prediction system according to claim 1, characterized in that: The communication delay includes at least one of a delay, bandwidth, throughput, jitter, bit error rate, and packet loss rate of communication between the remote end and the local end.

11. A surgical robot system, characterized in that: It comprises a remote end, a local end and a motion prediction system for remote surgical operation according to any one of claims 1 to 10; the remote end and the local end are configured in a master-slave control relationship.

Images