Action Prediction System for Remote Surgery Operations and Surgical Robot System

Abstract

The present invention provides a motion prediction system for remote surgical operation and a surgical robot system. The motion prediction system for remote surgical operation includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit. The autonomous motion trajectory generation unit is respectively deployed at the remote end and the local end. The instrument motion trajectory prediction unit is deployed at the remote end. When the communication delay between the remote end and the local end reaches a triggering condition: The autonomous motion trajectory generation unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively based on the current motion state of the robotic arm at the local end and a prediction model. The instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory based on the current motion state of the robotic arm at the local end and the operation information at the remote end. The autonomous motion trajectory at the local end is used for the robotic arm at the local end to execute. The autonomous motion trajectory and the operation prediction trajectory at the remote end are superimposed and displayed at the remote end.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and particularly relates to a motion prediction system for remote surgical operations and a surgical robot system. Background Art

[0002] Existing technologies in the field of remote surgery mainly include remote surgical robot systems and network latency compensation methods based on simple prediction, such as estimating the instrument motion trajectory through linear interpolation or the average value of historical data. However, these technologies have significant defects: they highly rely on a stable high-speed network. When the network quality is poor (such as the latency exceeds 300 ms or the packet loss rate is higher than 5%), the delay of operation instructions will cause the surgical operation to be interrupted or the accuracy to decrease; simple linear prediction methods cannot adapt to the personalized operation habits of doctors and the dynamic changes in complex surgical scenarios, with low prediction accuracy, making it difficult for doctors to quickly respond to potential risks, which limits the safety and reliability of remote surgery. Summary of the Invention

[0003] The purpose of the present invention is to provide a motion prediction system for remote surgical operations and a surgical robot system, so as to solve the problems of poor continuity, low safety, and decreased accuracy that occur when the existing surgical robot system is remotely applied and the network quality is poor.

[0004] To solve the above technical problems, the present invention provides a motion prediction system for remote surgical operations, which includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at the remote end and the local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches a trigger condition:

[0005] The autonomous motion trajectory generation unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively by using a kinematic model based on the current motion state of the robotic arm at the local end and a prediction model.

[0006] The instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory by using a kinematic model based on the current motion state of the robotic arm at the local end and the operation information at the remote end.

[0007] Among them, the autonomous motion trajectory generated at the local end is used for the robotic arm at the local end to execute; the autonomous motion trajectory and the operation prediction trajectory generated at the remote end are used for superimposed display at the remote end.

[0008] Optionally, the motion prediction system for remote surgical operations further includes a deep learning training unit.

[0009] The deep learning training unit is configured to train the prediction model by using pre-collected surgical operation data, so as to generate the prediction model for predicting the operation intention.

[0010] Optionally, the deep learning training unit adopts a convolutional neural network or a long short-term memory network and combines with a linear regression model for training. The surgical operation data includes historical operation trajectory data, operation habit data and endoscopic image data of multiple doctors.

[0011] Optionally, the deep learning training unit is further configured to update the parameters of the prediction model by using the current operation trajectory data when the communication delay between the remote end and the local end does not reach the trigger condition.

[0012] Optionally, the action prediction system for remote surgical operation further includes a visual information constraint unit;

[0013] The visual information constraint unit is configured to constrain the autonomous motion trajectory in combination with the depth information obtained by the local end, so that the autonomous motion trajectory conforms to a preset safety range and is within the visual field; wherein the depth information is used to determine the spatial relationship between the instrument and the tissue in the surgical scene.

[0014] Optionally, the step of constraining the autonomous motion trajectory is implemented based on tissue boundary detection and / or visual field boundary detection.

[0015] Optionally, the action prediction system for remote surgical operation further includes a visual field monitoring and warning unit;

[0016] The visual field monitoring and warning unit is configured to monitor whether the operation prediction trajectory exceeds the visual field range, and if it exceeds the visual field range, a warning message is sent.

[0017] Optionally, the kinematic model includes forward kinematics and inverse kinematics; the kinematic model is used to predict the position and angle of the end effector of the robotic arm.

[0018] Optionally, the action prediction system for remote surgical operation further includes a haptic feedback unit;

[0019] The haptic feedback unit is configured to generate vibration at the master operating arm of the remote end based on the operation prediction trajectory to simulate the contact force between the instrument and the tissue.

[0020] Optionally, the communication delay includes at least one of the delay, bandwidth, throughput, jitter, bit error rate and packet loss rate of the communication between the remote end and the local end.

[0021] To solve the above technical problems, the present invention further provides a surgical robot system, which includes a remote end, a local end, and the action prediction system for remote surgical operation as described above; the remote end and the local end are configured in a master-slave control relationship.

[0022] In summary, in the action prediction system for remote surgical operation and the surgical robot system provided by the present invention, the action prediction system for remote surgical operation includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at the remote end and the local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches a trigger condition: the autonomous motion trajectory generation unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively by using a kinematic model based on the current motion state of the robotic arm at the local end and a prediction model; the instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory by using a kinematic model based on the current motion state of the robotic arm at the local end and the operation information at the remote end; wherein, the autonomous motion trajectory generated at the local end is used for the robotic arm at the local end to execute; the autonomous motion trajectory and the operation prediction trajectory generated at the remote end are used for superimposed display at the remote end.

[0023] With such a configuration, on the one hand, the autonomous motion trajectory generation unit makes predictions based on the prediction model and fuses with multiple dimensions such as kinematics. Compared with traditional linear prediction, it can capture complex non-linear operation modes and improve the prediction accuracy. In a network delay environment, the robotic arm can execute motions according to the autonomous motion trajectory, enhancing the continuity, real-time performance, and safety of surgical operations in a network delay environment, optimizing the real-time performance and precision of remote surgery, reducing the need for doctors to perform real-time fine-tuning of surgical instruments in an unstable network environment, and reducing the operation burden on doctors. On the other hand, the autonomous motion trajectory and the operation prediction trajectory are superimposed and displayed at the remote end, improving the operation transparency and allowing doctors to intervene and adjust at any time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Those of ordinary skill in the art will understand that the provided drawings are used to better understand the present invention and do not constitute any limitation to the scope of the present invention.

[0025] Figure 1 is a schematic diagram of a master-slave teleoperation surgical robot system.

[0026] Figure 2 is Figure 1 a schematic diagram of the master control device of the surgical robot system shown.

[0027] Figure 3 is a schematic diagram of the master operating arm of the master control device.

[0028] Figure 4 It is a schematic diagram of the communication delay of the remotely controlled surgical robot system according to an embodiment of the present invention.

[0029] Figure 5 It is a schematic diagram of the modules of the motion prediction system for remote surgical operations according to an embodiment of the present invention.

[0030] Figure 6 It is a schematic diagram of the autonomous motion trajectory generated by the autonomous motion trajectory generation unit according to an embodiment of the present invention.

[0031] Figure 7 It is a schematic diagram of the display content of the display device at the remote end according to an embodiment of the present invention.

[0032] Figure 8 It is a schematic diagram of constraining the autonomous motion trajectory within the field of view according to an embodiment of the present invention.

[0033] Figure 9 It is a schematic diagram of the field of view monitoring and warning unit detecting that the operation prediction trajectory exceeds the field of view according to an embodiment of the present invention.

[0034] Figure 10 It is a schematic diagram of the overall operation step flow chart of the motion prediction system for remote surgical operations according to an embodiment of the present invention.

[0035] In the drawings: 1 - autonomous motion trajectory generation unit; 2 - instrument motion trajectory prediction unit; 3 - deep learning training unit; 4 - visual information constraint unit; 5 - field of view monitoring and warning unit; 6 - tactile feedback unit; 10 - master control device; 11 - master operating arm; 12 - joint; 13 - motor; 14 - force interaction center; 15 - display device; 20 - slave robot device; 21 - robotic arm; 30 - image trolley; 40 - communication device; 50 - remote end; 60 - local end. Detailed implementation manners

[0036] To make the objectives, advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are in very simplified forms and are not drawn to scale, and are only used to facilitate and clearly assist in explaining the objectives of the embodiments of the present invention. In addition, the structures shown in the accompanying drawings are often part of the actual structures. In particular, the accompanying drawings need to show different emphases and sometimes different scales are used.

[0037] As used in the present invention, the singular forms "a", "an", "one" and "the" include plural referents, the term "or" is generally used in the sense of including "and / or", the term "several" is generally used in the sense of including "at least one", the term "at least two" is generally used in the sense of including "two or more", in addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", "third" may explicitly or implicitly include one or at least two of such features, "one end" and "the other end" and "proximal end" and "distal end" generally refer to two corresponding parts, which include not only the endpoints. In addition, as used in the present invention, "mounted", "connected", "coupled", an element "disposed" on another element should be understood in a broad sense, generally only indicating a connection, coupling, cooperation or transmission relationship between the two elements, and the two elements may be directly or indirectly connected, coupled, cooperated or transmitted through an intermediate element, and cannot be construed as indicating or implying the spatial position relationship between the two elements, that is, an element may be in any orientation such as inside, outside, above, below or on one side of another element, unless otherwise expressly specified. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, directional terms such as above, below, up, down, upward, downward, left, right, etc. are used relative to the exemplary embodiments as shown in the figures, the upward or upward direction is towards the top of the corresponding figure, and the downward or downward direction is towards the bottom of the corresponding figure.

[0038] The object of the present invention is to provide a motion prediction system and a surgical robot system for remote surgical operation, so as to solve the problems of poor continuity, low safety and reduced accuracy of the existing surgical robot system when applied remotely in case of poor network quality. The following will be described with reference to the accompanying drawings.

[0039] First, please refer to Figures 1 to 3, which exemplarily shows a locally controlled master-slave teleoperation surgical robot system. Here, the master-slave teleoperation mainly means that the master control device 10 and the slave robot device 20 it includes are configured in a master-slave control relationship. An operator (such as a doctor) can drive and control the slave robot device 20 through operating the master control device 10 and master-slave mapping control. Further, the surgical robot system also includes an image cart 30, etc. In a specific exemplary case, the master control device 10 includes a master operating arm 11 (also called a master control arm or a master hand), which includes several joints 12 and motors 13. An operator (such as a doctor) can directly operate the master operating arm 11. The end of the master operating arm 11 is in direct contact with the operator's hand, and this contact point is the force interaction center 14. The interaction force is generated by the motors 13 of the respective joints 12 of the master operating arm 11. The slave robot device 20 includes several robotic arms 21, which can be used to mount instruments (such as surgical instruments or other auxiliary instruments) and image acquisition devices (such as endoscopes). The actions of the master operating arm 11 driven by the operator are mapped to the actions of the instruments and image acquisition devices of the robotic arms 21 through master-slave mapping, so as to achieve master-slave mapping operations and perform surgeries. Further, the image screen collected by the image acquisition device is fed back and transmitted to the image cart 30, and after being processed by the image cart 30, it is transmitted to the display device 15 of the master control device 10 for display, for the operator to observe. The operator can operate based on the observed image screen. In applications, in addition to forward mapping operation control information, the master control device 10 and the slave robot device 20 can also reverse map feedback information, such as compensating the reaction force of the contact between the instrument and the tissue to the master operating arm 11, so as to form a closed loop from operation to feedback.

[0040] After understanding the locally controlled surgical robot system, please refer to Figure 4 , which shows a remotely controlled surgical robot system, which includes a remote master control device (hereinafter, for the convenience of narration, the remote master control device is simply referred to as the remote end 50) and a local slave robot device (hereinafter, for the convenience of narration, the local slave robot device is simply referred to as the local end 60). The remote end 50 and Figure 2 The master control device 10 shown in may have the same or similar structure. The remote end 50 and the local end 60 are configured to exchange data through a communication device 40 (such as a server) to achieve remote operation.

[0041] In remote control, due to the addition of the communication device 40, the communication delay between the remote end 50 and the local end 60 includes: the delay t1 from the remote end 50 outputting a control signal to the communication device 40 receiving the signal; the delay t2 from the communication device 40 outputting a control signal to the local end 60 receiving the signal; the delay t3 from the local end 60 performing an action to the local image acquisition device capturing the current image; the delay t4 from the local image trolley 30 outputting an image to the communication device 40 receiving the signal (which may involve an encoding and decoding process); the delay t5 from the communication device 40 outputting a signal to the display device at the remote end 50 for imaging. It can be seen that there are more data exchange links in the remotely controlled surgical robot system. Compared with the locally controlled surgical robot system, its communication delay is more easily affected by various factors, resulting in a larger communication delay. In the prior art, when the network quality is poor (such as high latency and low bandwidth), the real-time performance of surgical operations is severely affected, which may cause operation instructions to be delayed, affecting surgical accuracy and even endangering the safety of patients.

[0042] To solve this problem, please refer to Figure 5 , an embodiment of the present invention provides an action prediction system for remote surgical operations, which is mainly used for a remotely controlled surgical robot system. The action prediction system for remote surgical operations includes an autonomous motion trajectory generation unit 1 and an instrument motion trajectory prediction unit 2; the autonomous motion trajectory generation unit 1 is respectively deployed at the remote end 50 and the local end 60; the instrument motion trajectory prediction unit 2 is deployed at the remote end 50; when the communication delay between the remote end 50 and the local end 60 reaches a trigger condition: the autonomous motion trajectory generation unit 1 is configured to generate autonomous motion trajectories at the remote end 50 and the local end 60 respectively based on the current motion state of the robotic arm at the local end 60 and a prediction model, using a kinematic model; the instrument motion trajectory prediction unit 2 is configured to generate a real-time operation prediction trajectory based on the current motion state of the robotic arm at the local end 60 and the operation information at the remote end 50, using a kinematic model; wherein, the autonomous motion trajectory generated by the local end 60 is used for the robotic arm at the local end 60 to execute; the autonomous motion trajectory and the operation prediction trajectory generated by the remote end 50 are used for superimposed display at the remote end 50.

[0043] Optionally, the prediction model here can be generated by AI, preferably trained by a deep learning algorithm. The prediction model can also be obtained by means such as traditional machine learning, or constructed based on rules and logic. This embodiment places no restrictions on its obtaining method. The communication delay includes at least one of the delay, bandwidth, throughput, jitter, bit error rate, and packet loss rate of the communication between the remote end 50 and the local end 60. The trigger condition refers to the condition representing poor communication status set for the specific communication parameters corresponding to the communication delay. For example, corresponding to the packet loss rate, the trigger condition can be set to be higher than 5%; corresponding to the delay, the trigger condition can be set to be greater than 300 ms, etc. Those skilled in the art can set corresponding trigger conditions for different communication parameters according to the prior art. When the communication delay between the remote end 50 and the local end 60 reaches the trigger condition, it can be considered that the communication status between the remote end 50 and the local end 60 is poor at this time, and the action prediction system of this embodiment is triggered.

[0044] For ease of understanding, the steps executed by the local end 60 and the remote end 50 will be described separately below.

[0045] Local end 60: The autonomous motion trajectory generation unit 1 deployed on the local end 60 generates an autonomous motion trajectory T (referring to a set of several expected motion positions of the robotic arm after the current moment) based on the current motion state of the robotic arm of the local end 60 and the prediction model, using the kinematic model, as Figure 6 shown. Optionally, the kinematic model includes forward kinematics and inverse kinematics; the kinematic model is used to predict the position and angle of the end effector of the robotic arm.

[0046] Then the robotic arm executes this autonomous motion trajectory and moves. Since the communication delay reaches the trigger condition, the local end 60 cannot obtain real-time subsequent operation instructions at this time, and the robotic arm moves according to the autonomous motion trajectory generated by the autonomous motion trajectory generation unit 1. Since the autonomous motion trajectory generation unit 1 makes predictions based on the prediction model and integrates multiple dimensions such as kinematics, compared with traditional linear prediction, it can capture complex non-linear operation patterns and improve the prediction accuracy. In a network delay environment, the robotic arm can execute the motion according to the autonomous motion trajectory, improving the continuity, real-time performance, and safety of surgical operations in a network delay environment, optimizing the real-time performance and precision of remote surgery, reducing the need for real-time fine-tuning of surgical instruments by the distal operator (doctor) in an unstable network environment, and reducing the operation burden on the distal operator (doctor).

[0047] Optionally, the autonomous motion trajectory includes multiple desired motion points. After generating multiple desired motion points, they can be smoothed. For example, moving average filtering can be used to smooth the autonomous motion trajectory. The window size of the moving average filtering is an adjustable parameter, and the window size can be selected as 5, for example. The smoothed autonomous motion trajectory T is as Figure 6 shown.

[0048] Remote end 50: The autonomous motion trajectory generation unit 1 deployed at the remote end 50 generates an autonomous motion trajectory (referring to the desired motion points of the robotic arm after the current moment) based on the current motion state of the robotic arm at the local end 60 and the prediction model, using the kinematic model.

[0049] Based on the foregoing description, it can be understood that before the communication delay between the remote end 50 and the local end 60 reaches the trigger condition, the communication status between the remote end 50 and the local end 60 meets the requirements. At this time, the remote end 50 and the local end 60 are in the normal master-slave teleoperation control state. The remote operator operates at the remote end 50, and the robotic arm at the local end 60 moves accordingly. When the communication delay between the remote end 50 and the local end 60 reaches the trigger condition, the position information of the robotic arm at the local end 60 at the current moment and information such as the surgical situation are recorded as the current motion state of the robotic arm. It can be understood that since the remote end 50 and the local end 60 are in normal communication before the communication delay reaches the trigger condition, the current motion state of the robotic arm can be synchronously recorded at both the remote end 50 and the local end 60. That is, when the action prediction system is triggered, the remote end 50 and the local end 60 have the same initial constraint conditions (referring to the current motion state of the robotic arm).

[0050] The autonomous motion trajectory generation unit 1 deployed at the remote end 50 and the autonomous motion trajectory generation unit 1 deployed at the local end 60 have the same initial constraint conditions and use the same prediction model. Therefore, the autonomous motion trajectories they generate are also the same. However, the autonomous motion trajectory generated by the autonomous motion trajectory generation unit 1 at the remote end 50 is not directly used to control the operation of the robotic arm at the local end 60, but is used to be displayed on the display device at the remote end 50 for the remote operator (doctor) to observe. Please refer to Figure 7 , which shows the situation displayed on the display device at the remote end 50, where the autonomous motion trajectory T1 reflects the trajectory that the robotic arm at the current local end 60 is running.

[0051] The instrument motion trajectory prediction unit 2 deployed at the remote end 50 generates a real-time operation prediction trajectory using a kinematic model based on the current motion state of the robotic arm of the local end 60 and the operation information of the remote end 50. The autonomous motion trajectory and the operation prediction trajectory generated by the remote end 50 are superimposed and displayed at the remote end 50. The operation information of the remote end 50 refers to the actual operation information of the remote operator (doctor). After the communication delay reaches the trigger condition, although the operation information of the remote end 50 cannot be transmitted to the local end 60 in real time, the remote operator (doctor) still continues the action before the communication delay reaches the trigger condition, that is, the remote operator (doctor) is still operating the main operating arm of the remote end 50, thereby generating a series of actual operation information. The instrument motion trajectory prediction unit 2 can generate a real-time operation prediction trajectory based on the current motion state of the robotic arm of the local end 60 before the communication delay reaches the trigger condition, and the actual operation information of the remote operator (doctor) after the communication delay reaches the trigger condition. Please refer to Figure 7 , the operation prediction trajectory T2 reflects the predicted trajectory of the actual operation of the current remote operator (doctor). Optionally, when the autonomous motion trajectory T1 and the operation prediction trajectory T2 are superimposed and displayed, the two trajectories preferably use different colors or line shapes to distinguish them. The legacy display of the autonomous motion trajectory T1 improves the transparency of the operation. The remote operator (doctor) can intervene and correct his own operation at any time according to the displayed autonomous motion trajectory T1 and the operation prediction trajectory T2, so that the two trajectories are as close or overlapped as possible. If the remote operator is an experienced doctor, the overlap between his operation prediction trajectory T2 and the autonomous motion trajectory T1 itself may be very high.

[0052] Preferably, the remote surgical operation motion prediction system further includes a deep learning training unit 3; the deep learning training unit 3 is configured to train the prediction model using pre-collected surgical operation data to generate the prediction model for predicting the operation intention. In some embodiments, the prediction model used by the autonomous motion trajectory generation unit 1 can be trained externally and directly input into the motion prediction system of this embodiment. Preferably, the deep learning training unit 3 can be integrated into the motion prediction system, and the prediction model used by the autonomous motion trajectory generation unit 1 is trained by the motion prediction system.

[0053] Optionally, the deep learning training unit 3 uses a convolutional neural network (CNN) or a long short-term memory network (LSTM) in combination with a linear regression model for training, and the surgical operation data includes historical operation trajectory data, operation habit data, and endoscopic image data of multiple (senior) doctors. The following is an example to illustrate the process of training the prediction model.

[0054] The input parameters of the prediction model may optionally include historical operation trajectory data and endoscopic image data. Historical operation trajectory data: defined as P hist ={P 1 ,P 2 ,…,P n}, where P i =[Pos i ,Ori i represents a coordinate point in three-dimensional space, n = 50 (sampling frequency is 10Hz, time span is 5 seconds), and P hist includes the historical operation trajectory data of multiple (senior) doctors. The endoscopic image data is denoted as I t , representing the two-dimensional image frame at time t. Further, the historical operation trajectory data is processed by range normalization: P i '=(P i -P min ) / (P max -P min ), where P min and P max are the minimum and maximum coordinate values in the historical operation trajectory data respectively.

[0055] Taking the long short-term memory network (LSTM) adopted by the deep learning training unit 3 as an example, the trajectory P hist is used to generate the trajectory P pred ={P n+1 ,P n+2 ,…,P n+m} for the next j seconds. Here, both j and m are optional parameters. For example, j = 2 and m = 20.

[0056] The state update formula of the long short-term memory network (LSTM) is as follows:

[0057] i t =σ(W i ·[h t-1 ,P t +b i ) (input gate)

[0058] f t =σ(Wf·[ h t-1 ,P t +b f ) (forget gate)

[0059] o t =σ(Wo·[ h t-1 ,P t +b o ) (output gate)

[0060] c t =ft ⊙c t-1 +i t ⊙tanh(W c ·[ h t-1 ,P t +b c ) (memory cell)

[0061] h t =o t ⊙tanh(c t ) (hidden state)

[0062] Wherein, W and b are the weight matrix and the bias vector respectively, σ is the sigmoid activation function, ⊙ represents element-wise multiplication, and h t is the predicted value of the output trajectory point. For other principles of the long short-term memory network (LSTM), reference can be made to the prior art, which will not be elaborated here.

[0063] Preferably, the deep learning training unit 3 is further configured to update the parameters of the prediction model by using the current operation trajectory data when the communication delay between the remote end 50 and the local end 60 does not reach the trigger condition. In one embodiment, the input parameters of the prediction model may include, in addition to the pre-collected surgical operation data, the operation trajectory data of the current remote operator (i.e., the current doctor). When the communication delay does not reach the trigger condition, the operation trajectory data of the current remote operator can also be used as the training input parameters of the prediction model and incorporated into the training process. That is, the prediction model can be updated in real time, and it can better adapt to and predict the operation intention of the current remote operator.

[0064] Preferably, the action prediction system for remote surgical operation further includes a visual information constraint unit 4; the visual information constraint unit 4 is configured to constrain the autonomous motion trajectory in combination with the depth information obtained by the local end 60, so that the autonomous motion trajectory conforms to a preset safety range and is within the field of view; wherein the depth information is used to determine the spatial relationship between the instrument and the tissue in the surgical scene. The depth information can be obtained by a binocular vision acquisition device (such as a binocular endoscope) or a depth sensor. Its specific principle can be referred to the prior art.

[0065] Please refer to Figure 8, The field of view refers to the visible area of the image acquisition device (such as an endoscope) at the local end 60. In some embodiments, some of the desired motion points on an unconstrained autonomous motion trajectory T3 may be outside the field of view, making it difficult to ensure surgical safety at this time. In addition, when the local end 60 starts autonomous motion, the unconstrained autonomous motion trajectory T3 may have unexpected contact with tissues, causing harm. Therefore, it is necessary to impose certain constraints on the autonomous motion trajectory T3. In this embodiment, through the setting of the visual information constraint unit 4, on the one hand, combined with the endoscope image data, and on the other hand, combined with the depth information, the autonomous motion trajectory T3 is constrained. The constrained autonomous motion trajectory T4 can always maintain a certain safety distance from the tissues, and the desired motion points on the autonomous motion trajectory T4 are always within the field of view, effectively improving the safety of the autonomous motion of the local end 60.

[0066] Optionally, the step of constraining the autonomous motion trajectory is implemented based on tissue boundary detection and / or field of view boundary detection. The tissue boundary detection and the field of view boundary detection are demonstrated below.

[0067] Tissue boundary detection: Use a U-Net network to perform semantic segmentation on the endoscope image data I t to generate a tissue mask M t , where M t = 1 indicates the tissue area, and where M t = 0 indicates the background.

[0068] Calculate the projected coordinates (uk, vk) of the predicted trajectory point P n+k on the image

[0069]

[0070] where K is the camera intrinsic matrix and T is the extrinsic matrix.

[0071] Taking the requirement that the distance dk between the trajectory point and the tissue boundary is ≥ 3 mm as an example, if the distance dk between the predicted trajectory point P n+k and the tissue boundary is < 3 mm, then it is adjusted by gradient descent:

[0072]

[0073] where the distance penalty loss function L = Σk(3 - dk) 2 , and η is the learning rate. The optimized trajectory P pred ' = [P n+1 , P n+2 , …, P n+k , and then the optimized trajectory P pred ' is converted into an instrument control instruction, and the instrument executes at an adaptive speed V k : Vk = Vmax·exp(-αk k )。

[0074] Vmax is the maximum speed, α is the gain, and k k is the trajectory curvature. All of the above are adjustable parameters.

[0075] Field of view boundary detection: Use U-Net to segment the endoscopic image data I t to generate a field of view boundary mask B t , where B t = 1 represents the area inside the boundary, and B t = 0 represents the area outside the boundary.

[0076] Projection calculation: Project the predicted trajectory point P n+k onto the image plane.

[0077]

[0078] where K is the camera internal parameter matrix and T is the external parameter matrix.

[0079] Calculate the distance dk from the projected point (uk, vk) to the field of view boundary:

[0080]

[0081] where b is the point on the boundary mask Bt.

[0082] Constraint adjustment: If dk < dsafe (the safety distance dsafe can be set to 5mm, for example), adjust the trajectory point to keep it within the field of view:

[0083]

[0084] where bnearest is the nearest point on the boundary and η is the adjustment step size (which can be set to 0.1, for example).

[0085] If the distance dk between the predicted trajectory point P n+k and the tissue boundary is < 3mm, adjust it by gradient descent:

[0086]

[0087] where the distance penalty loss function L = Σk (dsafe - dk 2 ), and η is the learning rate. When the trajectory is executed, the speed V k is adjusted based on the field of view boundary distance: V k = Vmax·min (1, dk / dsafe). If dk ≥ dsafe, execute at the maximum speed Vmax. If dk < dsafe, the speed V kDecreases linearly with the decrease of distance.

[0088] Consider both tissue boundary detection and field of view boundary detection:

[0089] Tissue boundary detection uses U-Net to perform semantic segmentation on the endoscopic image I t to generate a tissue mask M t , where M t M(x, y) = 1 represents the tissue area, where M t M(x, y) = 0 represents the background.

[0090] Calculate the projected coordinates (uk, vk) of the predicted trajectory point P n+k on the image

[0091]

[0092] Define the distance from the tissue boundary:

[0093]

[0094] where m is a point on the tissue mask M t on it.

[0095] If dtissus,k < safety distance (the safety distance can be set to 3mm, for example), adjust the trajectory point to keep it within the field of view:

[0096]

[0097] where mnearest is the nearest point on the tissue boundary and η is the adjustment step (which can be set to 0.1, for example).

[0098] Field of view boundary detection uses U-Net to segment the endoscopic image data I t to generate a field of view boundary mask B t , where B t B(x, y) = 1 represents the area inside the boundary, where B t B(x, y) = 0 represents the area outside the boundary.

[0099] Calculate the distance from the field of view boundary:

[0100]

[0101] Constraint adjustment: If dvision,k < dsafe (the safety distance dsafe can be set to 5mm, for example), adjust the trajectory point:

[0102]

[0103] where bnearest is the nearest point on the tissue boundary and η 2For adjusting the step size (which can be set to 0.1, for example).

[0104] Comprehensively optimize the tissue and field-of-view constraints, and the loss function is as follows:

[0105]

[0106] Among them, W1 = 1.0, W2 = 1.5 (the priority of the field-of-view constraint is higher), I is the indicator function, and the corresponding term is 0 if the distance meets the requirements. The speed V during the trajectory execution k Comprehensive tissue and field-of-view distance:

[0107]

[0108] If dtissue,k≥3 and dvision,k≥dsafe, then V k = Vmax; if any of the distances is less than the threshold, the speed linearly decreases to a minimum of 0.

[0109] Considering both tissue boundary detection and field-of-view boundary detection simultaneously can ensure that the instrument does not exceed the field of view of the endoscope and does not damage the tissue, which is applicable to endoscopic surgery scenarios that require both safety and visibility.

[0110] Optionally, the motion prediction system for remote surgical operations further includes a field-of-view monitoring and warning unit 5; the field-of-view monitoring and warning unit 5 is configured to monitor whether the operation prediction trajectory exceeds the field of view, and if it exceeds the field of view, a warning message is issued. Please refer to Figure 9 , the field-of-view monitoring and warning unit 5 is preferably deployed at the remote end 50, which is used to monitor whether the operation prediction trajectory T5 of the distal operator (doctor) exceeds the field of view, and if it will exceed the field of view, a warning message is issued. The warning message can be issued in the form of, for example, text, sound, or light. Further, while issuing the warning message, the field-of-view monitoring and warning unit 5 also preferably automatically adjusts the motion path of the instrument to keep the instrument within the field of view.

[0111] Optionally, the motion prediction system for remote surgical operations further includes a haptic feedback unit 6, and the haptic feedback unit 6 is configured to generate vibrations at the main operating arm of the remote end 50 based on the operation prediction trajectory to simulate the contact force between the instrument and the tissue. The haptic feedback unit 6 is preferably deployed at the remote end 50, which can apply a simulated feedback force to the main operating arm to enhance the operation experience of the distal operator. The specific setting principle can refer to the prior art and will not be elaborated here.

[0112] Finally, please refer to Figure 10 , which shows the overall operation step flowchart of the motion prediction system for remote surgical operations in this embodiment. It specifically includes:

[0113] Step S1: Use the pre - collected surgical operation data to train a prediction model to generate a prediction model for predicting operation intentions.

[0114] Step S2: Determine whether the communication delay between the remote end and the local end reaches the trigger condition; if yes, execute Step S3, if no, execute Step S8.

[0115] Step S3: Based on the current motion state of the robotic arm at the local end and the prediction model, use the kinematic model to generate autonomous motion trajectories at the remote end and the local end respectively.

[0116] Step S4: Combine the depth information obtained at the local end to constrain the autonomous motion trajectory so that the autonomous motion trajectory conforms to the preset safety range and is within the field of view.

[0117] Step S8: Update the parameters of the prediction model using the current operation trajectory data.

[0118] Steps S1 - S4 and Step S8 here are general steps. Among them, Steps S1, S2, and S8 can be executed either at the remote end or the local end, or can be executed simultaneously respectively. Steps S3 and S4 are executed at the remote end and the local end respectively. The subsequent Steps S5A - S7A are only executed at the remote end, and Step S5B is only executed at the local end.

[0119] Step S5A: Based on the current motion state of the robotic arm at the local end and the operation information at the remote end, use the kinematic model to generate a real - time operation prediction trajectory.

[0120] Step S6A: Overlay and display the autonomous motion trajectory and the operation prediction trajectory at the remote end.

[0121] Step S7A: Monitor whether the operation prediction trajectory exceeds the field of view. If it exceeds the field of view, send a warning message.

[0122] Step S5B: The robotic arm at the local end executes the autonomous motion trajectory.

[0123] For the specific implementation methods of the above operation steps, reference can be made to the descriptions of the functional units of the action prediction system for remote surgical operations in the previous text, which will not be repeated here.

[0124] Optionally, each functional unit of the action prediction system for remote surgical operation in this embodiment can be configured as a physical module and installed and deployed at the remote end 50 or the local end 60 respectively, or can be configured as a program without a physical entity, which is deployed at the remote end 50 or the local end 60 in the form of software, or can also be deployed in the form of a readable storage medium storing the program. The present invention is not limited thereto. Those skilled in the art can deploy each function of the action prediction system for remote surgical operation in this embodiment according to the prior art.

[0125] An embodiment of the present invention further provides a surgical robot system, which includes a remote end 50, a local end 60, and the action prediction system for remote surgical operation as described above; the remote end 50 and the local end 60 are configured in a master-slave control relationship. The structures and principles of other components of this surgical robot system can be understood by referring to the above and in combination with the prior art, and will not be elaborated here.

[0126] In summary, in the action prediction system for remote surgical operation and the surgical robot system provided by the present invention, the action prediction system for remote surgical operation includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at the remote end and the local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches the trigger condition: the autonomous motion trajectory generation unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively based on the current motion state of the robotic arm at the local end and the prediction model by using the kinematic model; the instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory based on the current motion state of the robotic arm at the local end and the operation information at the remote end by using the kinematic model; wherein, the autonomous motion trajectory generated at the local end is used for the robotic arm at the local end to execute; the autonomous motion trajectory and the operation prediction trajectory generated at the remote end are used for superimposed display at the remote end. With such a configuration, on the one hand, the autonomous motion trajectory generation unit makes predictions based on the prediction model and fuses with multiple dimensions such as kinematics. Compared with traditional linear prediction, it can capture complex non-linear operation modes and improve the prediction accuracy. In a network delay environment, the robotic arm can execute motions according to the autonomous motion trajectory, enhancing the continuity, real-time performance, and safety of surgical operations in a network delay environment, optimizing the real-time performance and precision of remote surgery, reducing the need for doctors to perform real-time fine-tuning of surgical instruments in an unstable network environment, and reducing the doctor's operation burden. On the other hand, the autonomous motion trajectory and the operation prediction trajectory are superimposed and displayed at the remote end, improving the operation transparency, and doctors can intervene and adjust at any time.

[0127] It should be noted that the above-mentioned several embodiments can be combined with each other. The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A motion prediction system for remote surgery, characterized in that: It includes an autonomous motion trajectory generation unit and an instrument motion trajectory prediction unit; the autonomous motion trajectory generation unit is respectively deployed at a remote end and a local end; the instrument motion trajectory prediction unit is deployed at the remote end; when the communication delay between the remote end and the local end reaches a trigger condition: The autonomous motion trajectory generating unit is configured to generate autonomous motion trajectories at the remote end and the local end respectively using a kinematic model based on the current motion state of the robot arm at the local end and the prediction model; The instrument motion trajectory prediction unit is configured to generate a real-time operation prediction trajectory using a kinematic model based on the current motion state of the robot arm at the local end and the operation information of the remote end; The autonomous motion trajectory generated by the local end is used for execution by the robotic arm at the local end; the autonomous motion trajectory and the operation prediction trajectory generated by the remote end are used for superimposed display at the remote end.

2. The remote surgery motion prediction system according to claim 1, characterized in that: It also includes a deep learning training unit; The deep learning training unit is configured to train the prediction model using pre-collected surgical operation data to generate the prediction model for predicting operation intention.

3. According to the motion prediction system for remote surgical operations according to claim 2, the deep learning training unit adopts a convolutional neural network or a long short-term memory network combined with a linear regression model for training, and the surgical operation data includes historical operation trajectory data, operation habit data and endoscopic image data of multiple doctors.

4. According to the motion prediction system for remote surgical operations according to claim 2, the deep learning training unit is also configured to update the parameters of the prediction model using current operation trajectory data when the communication delay between the remote end and the local end does not reach the trigger condition.

5. The remote surgery motion prediction system according to claim 1, characterized in that: Also included is a visual information constraint unit; The visual information constraint unit is configured to constrain the autonomous motion trajectory in combination with the depth information obtained by the local end so that the autonomous motion trajectory complies with a preset safety range and is within the field of view; wherein the depth information is used to determine the spatial relationship between instruments and tissues in the surgical scene.

6. The remote surgery motion prediction system according to claim 5, characterized in that: The step of constraining the autonomous motion trajectory is implemented based on tissue boundary detection and / or visual field boundary detection.

7. The remote surgery motion prediction system according to claim 1, characterized in that: It also includes a visual field monitoring and early warning unit; The field of vision monitoring and early warning unit is configured to monitor whether the predicted operation trajectory exceeds the field of vision, and issue a warning message if it exceeds the field of vision.

8. The remote surgery motion prediction system according to claim 1, characterized in that: The kinematic model includes forward kinematics and inverse kinematics; the kinematic model is used to predict the position and angle of the end effector of the robotic arm.

9. The remote surgery motion prediction system according to claim 1, characterized in that: Also included is a tactile feedback unit; The tactile feedback unit is configured to generate vibration at the main operating arm of the remote end based on the predicted operation trajectory to simulate the contact force between the instrument and the tissue.

10. The remote surgery motion prediction system according to claim 1, characterized in that: The communication delay includes at least one of a delay, bandwidth, throughput, jitter, bit error rate, and packet loss rate of communication between the remote end and the local end.

11. A surgical robot system, characterized in that: It comprises a remote end, a local end and a motion prediction system for remote surgical operation according to any one of claims 1 to 10; the remote end and the local end are configured in a master-slave control relationship.

Images