An adaptive assistance method and system for teleoperated surgery

Abstract

The application belongs to the technical field of medical instrument control, and particularly discloses a self-adaptive assisting method and system applied to teleoperation surgery. The method comprises the following steps: generating a generalization standard trajectory set according to an expert operation trajectory and a target point position; constructing a mapping relationship of target point speed-position hybrid teleoperation based on a dynamic bubble control method, and realizing control of a robot pose according to the mapping relationship; acquiring the position of the robot end, predicting the target in real time and calculating the credibility of the target; exerting an auxiliary force on the master hand according to the real-time position of the robot end and the standard trajectory of the predicted target, so as to complete the assistance of target point position control; and synchronously adjusting the attitude under speed control according to the distance between the robot end and the predicted target position, so as to complete the assistance of target point attitude control. The application improves the continuity of master hand operation, and can accurately and adaptively complete the assistance of slave hand position and pose control.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical instrument control, and more particularly, to a self-adaptive assistance method and system applied to teleoperation surgery. BACKGROUND

[0002] In recent years, robot-assisted minimally invasive surgery has been increasingly applied due to its small trauma to patients, low infection rate, and short recovery time. There is an increasing demand for doctors with rich experience in robot-assisted minimally invasive surgery. Robot-assisted minimally invasive surgery involves inserting a laparoscope and surgical instruments from an end-effector robot into the abdominal cavity through a small hole in the patient's abdomen. The doctor controls the end-effector robot by operating the master hand, thereby completing the surgery. However, there are several problems in actual surgery: (1) poor operation continuity. Doctors often need to adjust the master-slave scaling ratio for small-scale fine operations, and often need to press the clutch multiple times for large-scale movements, reducing the continuity of operation. (2) laparoscope view causes operation deviation. Due to the perspective effect of the laparoscope, it is difficult for the operator to accurately judge the position and posture of the operation target, resulting in operation deviation or even errors. (3) long training time. Doctors need to undergo a long training period to approach the professional level in skills.

[0003] To solve the problem of poor operation continuity, the paper A self-adaptive motion scaling framework for surgical robot remote control proposes a self-adaptive scaling method that dynamically adjusts the scaling ratio between the master and slave considering multiple factors. However, the changing scaling ratio can cause non-intuitive speed fluctuations, increase cognitive load, and amplify meaningless hand shaking during master operation, which can be risky. To eliminate the interference of the laparoscope view, US20230131431(A1) uses a method to register the end-effector reference frame of the surgical instrument and the laparoscope view reference frame. Although this method compensates for the lack of intuition in the short term, the perspective effect of the laparoscope still exists. US20230112592(A1) proposes that by obtaining information from multiple non-robotic system sensors, guidance for the doctor's remote operation of the robotic system is generated. However, this method is mainly for the Da Vinci surgical robot system and does not consider the doctor's operation information during teleoperation. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the application provides a self-adaptive assistance method and system applied to teleoperation surgery, which generates a generalized standard trajectory set according to an expert operation trajectory and a target point position; a mapping relationship of target point speed-position hybrid teleoperation is constructed based on a dynamic bubble control method, and the mapping relationship is used to realize control of a robot pose; a position of a robot end is acquired, a target is predicted in real time and a credibility of the target is calculated; an auxiliary force is applied to a master hand according to a real-time position of the robot end and a standard trajectory of the predicted target, so that assistance for target point position control is completed; a pose under speed control is adjusted synchronously according to a distance between the robot end and the predicted target position, so that assistance for target point pose control is completed, which can effectively solve the problems that the master hand operation continuity is poor and the position and pose of the operation target are difficult to accurately judge in the prior art, and meanwhile, the pedaling of the clutch can be reduced when moving in a large range, and the operation continuity is improved. In addition, the application also realizes real-time and accurate prediction of the operator's intention, ensures implementation of the assistance strategy and self-adaptive pose adjustment of the hand.

[0005] To achieve the above object, according to one aspect of the application, a self-adaptive assistance method applied to teleoperation surgery is provided, comprising the following steps:

[0006] S100: generating a generalized standard trajectory set according to an expert operation trajectory and a target point position;

[0007] S200: constructing a mapping relationship of target point speed-position hybrid teleoperation based on a dynamic bubble control method, and realizing control of a robot pose according to the mapping relationship;

[0008] S300: acquiring a position of a robot end, predicting a target in real time and calculating a credibility of the target;

[0009] S400: applying an auxiliary force to a master hand according to a real-time position of the robot end and a standard trajectory of the predicted target, so that assistance for target point position control is completed;

[0010] S500: adjusting a pose under speed control synchronously according to a distance between the robot end and the predicted target position, so that assistance for target point pose control is completed.

[0011] As a further optimization, in step S100, an expert trajectory is acquired according to an operation trajectory of a doctor, a target point is acquired as a position and a pose according to a video stream, and then shape parameters of the expert trajectory in x, y and z dimensions are learned based on a dynamic motion primitive method, a terminal point in the dynamic motion primitive is replaced by a position of each target point, and thus a generalized standard trajectory based on each target is generated.

[0012] As a further optimization, step S200 comprises the following steps:

[0013] S201 obtaining an initial position of the master hand and setting an initial bubble radius, at this time, the initial position of the master hand coincides with the center position of the bubble;

[0014] S202 calculating the position of the master hand and judging whether the position of the master hand is within the initial bubble, if yes, the control mode is position control, directly calculating the expected position of the slave under velocity control, if not, the control mode is velocity control, calculating the equivalent velocity of the slave under control by introducing a factor for perceiving the scene, and then calculating the expected position of the slave under velocity control;

[0015] S203 updating the bubble radius and the center position of the bubble according to the actual control mode.

[0016] As a further preferred, in step S202, the factor for perceiving the scene includes:

[0017]

[0018]

[0019] wherein k goal (t k )∈(0,1] is a basic velocity parameter, k dis (t k )∈(0,1] is a distance parameter, and k cur (t k )∈(0,1] is a curvature parameter. is the position of the predicted target k at time t , s start is the position of the starting point, is the position of the i th target point, and d s (t k ) is the distance between the slave and the predicted target is the first derivative on the reference trajectory corresponding to the predicted target at the nearest point from the slave, is the corresponding second derivative, and ρ and μ are parameters of the steepness and offset of the control sigmoid function.

[0020] Preferably, the calculation formula of the equivalent velocity includes:

[0021]

[0022] wherein v(t k+1 ) is the expected equivalent velocity of the slave under velocity control, wherein is the position of the master hand after the clutch pedal is closed, and s c is the position of the bubble center. For along s b The unit vector of direction, through s b / ||s b ||Calculation yields R b (t k () represents the radius of the bubble;

[0023] Preferably, the calculation model for the desired location at the slave end includes:

[0024]

[0025]

[0026] Wherein, D(t) k ) represents the weight of speed control. The sum of the weights of speed control and position control is 1, and its value range is (0,1). sd For the desired position of the hand, k pos A fixed scaling factor for position control, typically k. pos ∈(0,1), v(t) i ) for t i The speed of time, It is a parameter that controls the steepness of the sigmoid function;

[0027] Preferably, the update model for bubble radius and bubble center position includes:

[0028] R b (t k )=k c (t k )R b0

[0029]

[0030]

[0031] in, As a control mode factor, when in velocity control, the bubble radius R b The bubble radius gradually shrinks as the speed control duration increases, and when returning to position control mode, the bubble radius gradually increases back to the initial radius R. b0 R b (t k ) for t k Bubble radius at time k c (t k ) is the scaling factor for the bubble radius, c(t) k ) represents the count of the current mode. When in position control mode, c(t) k+1 )=c(t k) = c(t k+1 ) + 1, c0 is a given constant for controlling the steepness of the sigmoid function, k and and respectively represent the upper and lower bounds of the control mode factor, and k up + k down = 1, s c (t k+1 ) is the position of the bubble center.

[0032] As a further preferred, the target prediction in step S300 comprises:

[0033] S301 obtaining the pose of the slave in the world coordinate system, searching for the nearest point on each reference trajectory;

[0034] S302 calculating the probability density of the slave falling on each trajectory by using a Gaussian distribution, obtaining the transition probability of each target;

[0035] S303 introducing a forgetting factor, updating the transition probability of each target by Bayesian inference, obtaining the probability of each target, and normalizing the probability;

[0036] S304 taking the target corresponding to the maximum probability as the predicted target, and calculating the credibility of the predicted target based on information entropy.

[0037] As a further preferred, in step S301, first obtain the state vector of the slave where s s is the position of the slave, is the speed of the slave, and a set contains all targets, when observing k trajectory points, the probability of reaching the target can be expressed as P(G|h 0:k ), calculate the nearest point on each trajectory to the current slave position

[0038] Preferably, in step S302, it is assumed that the predicted k+1th point is on a certain reference trajectory, and the probability satisfies a Gaussian distribution N(0,σ 2 ), the probability is expressed by the probability density corresponding to the distance between the nth point and the reference trajectory, and the distance rate of change is introduced into the calculation of the conditional probability, then the transition probability f(h k+1 |G,h k ) is:

[0039]

[0040] When a new observation point is obtained, the predicted probability of reaching each target is updated as P(G|h0:k+1 ), based on the Bayesian theory, the predicted probability of the n+1th point to each target is:

[0041]

[0042] wherein, may be expressed as f(h k+1 |G,h 0:k ), which is the probability of reaching each target at the next time, is obtained by multiplying the probability P(G|h 0:k ) at the previous time and the conditional probability f(h k+1 |G,h 0:k ), the Markov assumption is introduced, so that f(h k+1 |G,h 0:k ) = f(h k+1 |G,h k ), that is, the probability of the k+1th point is only related to the state of the kth point, and is irrelevant to the previous historical trajectory, and the predicted probability of the n+1th point to each target is expressed as:

[0043] P(G|h 0:k+1 ) ∝ P(G|h 0:k )f(h k+1 |G,h k );

[0044] Preferably, in step S303, a forgetting factor ξ∈(0, 1] is introduced to weaken the influence of early trajectory data, the probability is taken logarithm and multiplied by different power forgetting factors according to the proximity of the sampling time and the current time, to obtain the following equation:

[0045]

[0046] The probability of each target is obtained by normalization calculation:

[0047]

[0048] Finally, the probabilities of each target are compared, and the target with the largest probability is taken as the predicted target, that is:

[0049]

[0050] Preferably, in step S304, the credibility of the predicted target is:

[0051]

[0052]

[0053] wherein, is the predicted target, and α(t k+1wherein, α(t) is the confidence of the target, m is the total number of possible targets to reach.

[0054] As a further preferred, the step S400 comprises the following steps:

[0055] S401 obtaining the slave end trajectory point, and obtaining the nearest point on the predicted target standard trajectory;

[0056] S402 calculating the auxiliary elastic force according to the position difference between the current slave end position and the nearest point on the predicted target standard trajectory;

[0057] S403 calculating the damping force according to the projection of the current slave end velocity on the standard trajectory velocity method plane;

[0058] S404 calculating the slave end inertial force according to the master end velocity;

[0059] S405 applying the auxiliary force to the master hand according to the auxiliary elastic force, the damping force, the inertial force and the bubble boundary force;

[0060] Preferably, the auxiliary force applied to the master hand is:

[0061] F m (t k+1 ) = F s (t k+1 ) + F b (t k+1 )

[0062] wherein,

[0063] In the formula, F s (t k+1 ) is the auxiliary force of the virtual clamp, F b (t k+1 ) is the feedback force of the bubble boundary, α(t k ) is the confidence of the target, k cur (t k ) is the curvature factor calculated above, M s is the inertial parameter, D s is the damping parameter, K s is the stiffness parameter, s s (t k ) is the actual position of the slave hand, is the nearest point on the reference trajectory corresponding to the predicted target.

[0064] As a further preferred, the step S500 comprises the following steps:

[0065] S501 obtaining the slave end current position and posture, and the position and posture of the predicted target;

[0066] S502 obtaining a current speed of the slave end, and predicting a projection of a displacement of the slave end at a next time instant in a direction towards the target point;

[0067] S503 calculating a ratio between a projection size and a distance between the slave end and the target point;

[0068] S504 calculating a rotation axis and a rotation angle between a current pose of the slave end and a target pose;

[0069] S505 calculating a new rotation matrix according to the ratio in step S503 and the rotation axis and the rotation angle, so as to obtain a pose of the slave end at the next time instant.

[0070] As a further preferred, in step S502, the projection of the slave hand speed in the direction towards the target point is:

[0071]

[0072] In the formula, is the projection of the slave hand speed in the direction towards the target point;

[0073] Preferably, in step S503, the ratio between the projection size and the distance between the slave end and the target point is:

[0074]

[0075] In the formula, ε(t k ) is the ratio between the projection size and the distance between the slave end and the target point, is a position of the slave hand in the direction towards the target point at the next time instant, and is calculated as:

[0076] Preferably, in step S505, the pose of the slave end at the next time instant is:

[0077]

[0078] In the formula, R s0 (t k+1 ) is the pose of the slave end at the next time instant, is a rotation matrix of a pose R m0 (t k ) of the master hand end recorded when the pedal is closed to a current master hand pose R m (t k ).

[0079] According to another aspect of the present application, there is also provided an adaptive assistance system applied to teleoperation surgery, comprising:

[0080] a first control module, configured to generate a generalized standard trajectory set according to an expert operation trajectory and a target point position;

[0081] The first control module is configured to construct a mapping relationship of target point speed-position hybrid remote operation based on a dynamic bubble control method, and control the pose of the robot according to the mapping relationship.

[0082] The second control module is configured to obtain the position of the robot end, predict the target in real time and calculate the credibility of the target.

[0083] The third control module is configured to apply an auxiliary force to the master hand according to the real-time position of the robot end and the standard trajectory of the predicted target, so as to complete the auxiliary control of the target point position.

[0084] The fourth control module is configured to adjust the pose under the speed control according to the distance between the robot end and the predicted target position, so as to complete the auxiliary control of the target point pose.

[0085] Overall, compared with the prior art, the above technical scheme of the present application mainly has the following technical advantages:

[0086] 1. The present application generates a general standard trajectory set based on the trajectory of expert operation and all target point positions, and realizes the speed-position hybrid remote operation mapping scheme through a dynamic bubble, so as to realize the pose control of the slave robot by the master hand. According to the position of the robot end, the target is predicted in real time, the credibility of the target is calculated, and the real-time position of the robot end and the standard trajectory of the predicted target are calculated, so as to apply an auxiliary force to the master hand, complete the auxiliary control of the position, and adjust the pose under the speed control according to the distance between the robot end and the predicted target position, so as to complete the auxiliary control of the pose, thereby reducing the off-on pedal in a wide range of movement and improving the continuity of the operation process. In addition, the present application also realizes real-time and accurate prediction of the operator's intention, ensures the implementation of the auxiliary strategy and the adaptive pose adjustment of the slave hand.

[0087] 2. The present application proposes a dynamic bubble type master-slave remote operation control scheme, which realizes the speed-position hybrid control of the slave hand of the surgical robot by the master hand. The continuity of the operator in the fine operation area and the large range movement stage is improved, and the operation time is greatly shortened.

[0088] 3. The present application proposes a target prediction strategy based on a standard trajectory and introducing a forgetting factor, which realizes real-time and accurate prediction of the operator's intention and ensures the implementation of the auxiliary strategy.

[0089] 4. The present application proposes a remote operation force auxiliary strategy based on a virtual clamp, which assists the operator in controlling the position of the slave hand in the speed mode through force guidance. On the one hand, the operator's surgical performance is improved, and on the other hand, the time for the operator to re-adjust the position is reduced, thereby shortening the operation time.

[0090] 5. The application proposes an adaptive adjustment method from the hand posture, which combines the change of the posture with the proximity of the position to realize the adaptive posture adjustment of the slave hand under the speed control, and can still continuously adjust when the predicted target changes. The laparoscopic perspective view effectively avoids the misleading of the operator posture adjustment, and ensures that the slave hand posture has been adjusted to the optimal state before entering the fine operation. BRIEF DESCRIPTION OF DRAWINGS

[0091] Figure 1 is a flowchart of an adaptive assistance method applied to teleoperation surgery related to an embodiment of the application;

[0092] Figure 2 is a dynamic bubble control flowchart related to an embodiment of the application;

[0093] Figure 3 is a target prediction flowchart related to an embodiment of the application;

[0094] Figure 4 is a teleoperation force assistance flowchart related to an embodiment of the application;

[0095] Figure 5 is a teleoperation posture adaptive adjustment flowchart related to an embodiment of the application. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical scheme and advantages of the application clearer, further detailed description of the application will be made below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0097] The adaptive assistance method applied to teleoperation surgery of the application first generates a generalized standard trajectory set according to the expert operation trajectory and the target point position, then constructs the mapping relationship of the target point speed-position hybrid teleoperation based on the dynamic bubble control method, and realizes the control of the robot pose according to the mapping relationship, then acquires the position of the robot end, predicts the target in real time and calculates the credibility of the target; according to the real-time position of the robot end and the standard trajectory of the predicted target, the master hand is applied with auxiliary force to complete the assistance of the target point position control; the posture under the speed control is adjusted synchronously according to the distance between the robot end and the predicted target position to complete the assistance of the target point posture control.

[0098] More specifically, as shown in Figure 1 the teleoperation assistance flowchart of the embodiment of the application takes the widely used peg transfer surgery training scene as an example, which includes the following steps:

[0099] (1)Firstly, the expert trajectory can be obtained by collecting the operation trajectory of experienced doctors who have completed the nail column transfer training. At the same time, the position and attitude of the target point need to be obtained in advance through vision or other methods. The above process is not discussed in this patent. Then, the shape parameters of the expert trajectory in x, y, and z dimensions are learned based on the dynamic motion primitive (DMP) method. The termination point in DMP is replaced by the position of each target point to generate a general standard trajectory based on each target.

[0100] (2) In order to enable the doctor to switch between small range, accurate, slow control and large range, less accurate, fast control, a dynamic bubble mixed control scheme is proposed. As shown in Figure 2 , the specific process of dynamic bubble mixed control.

[0101] Firstly, the position of the master hand after the clutch pedal is stepped down and the initial radius R of the bubble is set b = R b0 At this time, the bubble center s c position coincides with the master hand position.

[0102] By calculating and comparing with R b , it is judged whether the master hand position is in the bubble. If it is outside the bubble, the control mode is speed control, and the equivalent speed v of the slave hand needs to be calculated. In order to optimize the teleoperation process, the change of speed introduces multiple factors for sensing the experimental scene. As shown below:

[0103]

[0104]

[0105] In the formula, k goal ∈(0, 1] is the basic speed parameter, the farther the starting point is from the target, the faster the basic speed is. k dis ∈(0, 1] is the distance parameter, which decreases to 0 when reaching the vicinity of the target position. k cur ∈(0, 1] is the curvature parameter, which is determined by the curvature of the reference trajectory . When the curvature of the desired trajectory is large, the speed should be reduced to make it easier for the operator to correct the trajectory. Then the equivalent speed of the slave hand under speed control is calculated:

[0106]

[0107] Whether the master hand is in the bubble or outside the bubble, it is put into the same control framework and the Sigmoid function is used to realize smooth switching, so the desired position of the slave end can be calculated as:

[0108]

[0109]

[0110] where D ranges from (0, 1) and is used to set the weight of the two control modes. D gradually changes to 1 when outside the bubble and gradually changes to 0 when inside the bubble control.

[0111] Finally, the bubble radius and the center position of the bubble are updated, both of which need to be dynamically scaled according to the actual control mode, and the calculation formula is as follows:

[0112] R b (t k )=k c (t k )R b0

[0113]

[0114]

[0115] where, denotes the control mode factor. When in speed control, the bubble radius R b will gradually decrease with the extension of the speed control duration, improving the flexibility of steering. When returning to the position control mode, the bubble radius gradually increases to the initial radius R b0 , ensuring a large enough fine operation range.

[0116] (3) In order to provide targeted assistance, the operator's desired goal needs to be known in advance. As shown in FIG. 6, the method of goal prediction includes: Figure 3

[0117] First, the state vector of the hand h including position and velocity is obtained. Assuming a set G contains all goals, when observing k trajectory points, the probability of reaching the goal can be represented as P(G|h 0:k ). The nearest point to the current hand position on each trajectory is calculated.

[0118] Assuming that the probability of the predicted k+1 point on a certain reference trajectory satisfies a Gaussian distribution N(0, σ 2 ), the probability is represented by the probability density corresponding to the distance between the nth point and the reference trajectory. At the same time, the distance change rate is introduced into the calculation of the conditional probability, improving the gain of the "approaching" and "moving away" intentions. The transition probability f(h k+1 |G, h k ) can be calculated as:

[0119] ​

[0120] Then when a new observation point is acquired, the predicted probability of reaching each target will be updated as P(G | h 0:k+1 ), based on the Bayes theory, the predicted probability of the n+1 point reaching each target is:

[0121]

[0122] where, which can be expressed as f(h k+1 | G, h 0:k ), the meaning is that the probability of reaching each target at the next moment can be obtained by multiplying the probability P(G | h 0:k ) at the previous moment and the conditional probability f(h k+1 | G, h 0:k ). By introducing the Markov assumption, f(h k+1 | G, h 0:k ) = f(h k+1 | G, h k ), that is, the probability of the k+1 point is only related to the state of the k point, and has nothing to do with the previous historical trajectory, then formula (8) can be changed to:

[0123] P(G | h 0:k+1 ) ∝ P(G | h 0:k )f(h k+1 | G, h k )(9)

[0124] Then a forgetting factor ξ ∈ (0, 1] is introduced to weaken the influence of early trajectory data. Taking the logarithm of the probability and multiplying the forgetting factor of different powers according to the distance between the sampling time and the current time, the following equation can be obtained:

[0125]

[0126] Since the operator will eventually reach a certain target, it can be considered that the sum of the probabilities of reaching each target is equal to 1, and the probability of each target is calculated by normalization:

[0127]

[0128] Finally, the probabilities of each target are compared, and the target with the largest probability is taken as the predicted target, that is:

[0129]

[0130] In addition, in order to measure the credibility of the predicted target, the information entropy is introduced as the measure of credibility:

[0131]

[0132]

[0133] in Let H(t) represent the predicted target, and α represent the confidence level of the target. A higher information entropy H(t) indicates that the probabilities of multiple targets are similar, and the greater the uncertainty of the target. The confidence level of the predicted target directly affects the degree of assistance.

[0134] (4) The auxiliary force is generated based on an attraction-type virtual fixture, using a linear time-invariant system and a reference trajectory as the guiding reference. The specific process is shown in Figure 4.

[0135] First, select the corresponding standard trajectory based on the predicted target above, and then calculate the point on the standard trajectory that is closest to the current position.

[0136] To avoid placing additional cognitive stress on the physician performing delicate procedures, the assistive force is only applied during speed control, at which point the primary operator's differential system is activated. The assistive force F of the virtual fixture is applied... s It is divided into three parts: elastic force F P Damping force F D Inertial force F M .

[0137] F P Always along Direction, and with Positive correlation. F D Based on the current slave speed Velocity vector at the closest point on the trajectory Projection on the normal plane The calculations show that the direction is... The directions are opposite. It can be calculated as follows:

[0138]

[0139] To ensure operational stability, the force applied by the main operator must be supplemented by the main operator's damping force, the direction of which is related to the main operator's speed. The opposite direction, mapped to the operator's hand, is the inertial force F. M Furthermore, the curvature gain calculated above needs to be introduced. The auxiliary force based on the standard trajectory can then be calculated as:

[0140]

[0141] In the formula and These represent the stiffness and damping of the virtual fixture, respectively. This represents virtual quality. α is the target credibility calculated above. The total assist force between the master hand and the user should include not only the assist force from the slave virtual gripper but also the feedback force from the bubble boundary. The total master hand-side assist force is calculated as follows:

[0142] F m (t k+1 ) = F s (t k+1 )+F b (t k+1 (17)

[0143]

[0144] (5) Figure 5 As shown, the adaptive attitude adjustment method automatically adjusts the slave attitude according to the predicted target during the control of a wide range of movements.

[0145] This invention addresses the lack of degrees of freedom in remote attitude control during large-scale movements. First, the predicted target attitude is obtained. Its corresponding rotation matrix and the position of the surgical instrument tip s s and posture o s .

[0146] Then obtain the projection of the hand speed in the direction towards the target:

[0147]

[0148] If the time step from the current moment to the next moment is T, then the predicted position at the next moment in the direction pointing to the target is: Within this time step, the proximity in the direction pointing to the target, i.e., the ratio between the projected size and the distance from the end to the target, can be calculated as follows:

[0149]

[0150] The physical meaning of ε is how much closer the predicted target position is to the desired target in the next time step.

[0151] The change in target pose at each moment is synchronized with the degree of proximity in position, so that when the hand reaches the target point, its pose is the same as the target pose. The rotation axis vector from the current pose to the target pose is calculated. and rotation angle The attitude at the next moment will be rotated by an angle εθ from the original attitude towards the target attitude, starting from the current time t. k At the next moment t k+1 Rotation matrix It can be calculated using the Rodriguez formula:

[0152]

[0153] wherein denotes the skew-symmetric matrix of the rotation axis vector. Then the next pose can be calculated as:

[0154]

[0155] In the velocity-position hybrid control, it is necessary to switch between the position control and the velocity control. In order to ensure the continuity of the pose when switching from the large-range movement mode to the fine operation mode, the stored master pose R s0 will be updated as:

[0156]

[0157] wherein, denotes the pose R m0 (t k ) recorded by the master end when the pedal is closed to the rotation matrix of the current master pose R m (t k ).

[0158] In addition, according to another aspect of the present application, there is also provided an adaptive assistance system applied to the teleoperation surgery, which is used to implement the method of any embodiment or combination of multiple embodiments described above, comprising:

[0159] The first control module is used to generate a set of generalized standard trajectories according to the expert operation trajectory and the target point position;

[0160] The first control module is used to construct the mapping relationship of the target point velocity-position hybrid teleoperation based on the dynamic bubble control method, and to realize the control of the robot pose according to the mapping relationship;

[0161] The second control module is used to obtain the position of the robot end, to predict the target in real time and to calculate the credibility of the target;

[0162] The third control module is used to apply an auxiliary force to the master according to the real-time position of the robot end and the standard trajectory of the predicted target, so as to complete the assistance of the target point position control;

[0163] The fourth control module is used to adjust the pose under the velocity control according to the distance between the robot end and the predicted target position, so as to complete the assistance of the target point pose control.

[0164] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An adaptive assistance system for application in teleoperated surgery, characterized in that, The application relates to a teleoperation control method and device. The application comprises: A first control module for generating a generalized standard trajectory set according to an expert operation trajectory and a target point position; A second control module for constructing a mapping relationship of target point speed-position hybrid teleoperation based on a dynamic bubble control method, and realizing control of a robot pose according to the mapping relationship; A third control module for acquiring a position of a robot end, predicting a target in real time and calculating a credibility of the target; The second control module is used for: Acquiring an initial position of a master hand and setting an initial bubble radius, at this time, the initial position of the master hand is coincided with a bubble center position; Calculating a master hand position and judging whether the master hand position is in the initial bubble, if yes, the control mode is position control, an expected position of a slave end under speed control is directly calculated, if not, the control mode is speed control, an equivalent speed of the slave end is calculated by introducing a factor for sensing a scene, and then an expected position of the slave end under speed control is calculated; According to an actual control mode, a bubble radius and a bubble center position are updated; where k goal (t k ) is a base speed parameter, k dis (t k ) is a distance parameter, k cur (t k ) is a curvature parameter, is the position of the predicted target k at time t start , s s is the position of the start point, k is the position of the i-th target point, d k+1 (t c ) is the distance from the end and the predicted target, b is the first derivative on the reference trajectory corresponding to the predicted target at the point closest to the hand, b is the corresponding second derivative, and p and m are parameters controlling the steepness and offset of the sigmoid function. The factor for sensing the scene comprises: A fourth control module for applying an auxiliary force to the master hand to complete auxiliary control of the target point position according to a real-time position of the robot end and a standard trajectory of a predicted target; 2. The adaptive assistant system for use in teleoperated surgery of claim 1, wherein, A fifth control module for synchronously adjusting a pose under speed control according to a distance between the robot end and the predicted target position to complete auxiliary control of the target point pose.

3. A self-adapting assistant system for teleoperated surgery according to claim 2, characterized in that, The first control module acquires an expert trajectory according to a doctor's operation trajectory, acquires a target point as a position and a pose according to a video stream, and then learns shape parameters of the expert trajectory in x, y and z three dimensions based on a dynamic motion primitive method, replaces a terminal point in the dynamic motion primitive with a position of each target point to generate generalized standard trajectories based on each target. where v(t k+1 ) is the desired equivalent speed under speed control from the end, where, is the position of the master hand after the clutch pedal is closed, s c is the position of the bubble center, is the unit vector in the direction of s b , R b (t k ) is the radius of the bubble.

4. The adaptive assistant system for teleoperated surgery of claim 3, wherein, The calculation formula of the equivalent speed comprises: where D(t k ) is the weight of velocity control, the sum of the weights of velocity control and position control is 1, s sd is the desired position of the hand, k pos is the fixed scaling factor of position control, generally taken k pos ∈(0, 1), v(t i ) is the velocity at t i time, is the parameter of controlling the steepness of the sigmoid function.

5. A self-adapting assistant system for teleoperated surgery according to claim 4, characterized in that, The calculation model of the expected position of the slave end comprises: R b (t k )=k c (t k )R b0 wherein, is the control mode factor, when in speed control, the bubble radius R b is gradually reduced as the speed control duration extends, and when back to position control mode, the bubble radius gradually increases to the initial radius R b0 , R b (t k ) is the bubble radius at time t k , k c (t k ) is the scaling factor of the bubble radius, c(t k ) is the count of the mode it is in, c0 is a given constant, and are the upper and lower bounds of the control mode factor respectively, s c (t k+1 ) is the position of the bubble center.

6. The adaptive assistant system for teleoperated surgery of claim 1, wherein, The update model of the bubble radius and the bubble center position comprises: In the third control module, target prediction comprises: Acquiring a pose of the slave end in a world coordinate system and searching for a nearest point on each reference trajectory; Calculating a probability density of the slave end falling on each trajectory by adopting a Gaussian distribution to obtain a transfer probability of each target; Introducing a forgetting factor, updating the transfer probability of each target by Bayesian inference to obtain a probability of each target, and normalizing the probability; 7. A self-adapting assistant system for teleoperated surgery according to claim 6, characterized in that, The searching for the nearest point on each reference trajectory comprises: firstly obtaining the state vector of the end where s s is the position of the end, is the velocity of the end, assuming a set contains all targets, when observing k trajectory points, the probability of reaching the target is represented as P(G | h 0:k , calculating the nearest point on each trajectory to the current end position 8. The adaptive assistant system for teleoperated surgery of claim 1, wherein, Taking a target corresponding to a maximum probability as a predicted target, and calculating a credibility of the predicted target based on information entropy. The fourth control module is used for: Acquiring a slave end trajectory point and acquiring a nearest point on a standard trajectory of a predicted target; Calculating an auxiliary force elastic force according to a position difference between a current slave end position and the nearest point on the standard trajectory of the predicted target; Calculating a damping force according to a projection of a current slave end speed on a standard trajectory speed plane; Calculating a slave end inertial force according to a master end speed; 9. A self-adapting assistant system for teleoperated surgery according to claim 8, characterized in that, Applying an auxiliary force to the master hand according to the auxiliary force elastic force, the damping force, the inertial force and a bubble boundary force. F m (t k+1 )=F s (t k+1 )+F b (t k+1 ) wherein, where F s (t k+1 ) is the assistive force of the virtual clamp, F b (t k+1 ) is the feedback force of the bubble boundary, a(t k ) is the confidence of the target, k cur (t k ) is the curvature factor calculated above, M s is the inertia parameter, D s is the damping parameter, K s is the stiffness parameter, s s (t k ) is the actual position of the hand, is the closest point on the reference trajectory corresponding to the predicted target.

10. The adaptive assistant system for teleoperated surgery of claim 1, wherein, The auxiliary force applied to the master hand is: The fifth control module is used for: Acquiring a current position and a pose of the slave end and a position and a pose of the predicted target; Acquiring a current speed of the slave end and projecting a displacement of a next time on a direction towards the target point; calculating a ratio between a projection size and a distance from the slave to the target; calculating a rotation axis and a rotation angle between a current pose of the slave and a target pose; calculating a new rotation matrix according to the ratio and the rotation axis and the rotation angle, so as to obtain a next-time pose of the slave.

11. The adaptive assistant system for use in teleoperated surgery of claim 10, wherein, a projection of the slave velocity in the direction towards the target is: wherein is the projection of the hand velocity in the direction towards the target, s s (t k ) is the actual position of the hand.

12. The adaptive assistant system for use in teleoperated surgery of claim 11, wherein, a ratio between a projection size and a distance from the slave to the target is: where ε(t) is the size of the projection and the ratio between the distance from the end and the target, k is the predicted position of the hand at the next time instant in the direction of the target, calculated as:​ where s s (t k ) is the actual position of the hand, T is the time step.

13. The adaptive assistant system for use in teleoperated surgery of claim 12, wherein, the next-time pose of the slave is: In the formula, R s0 (t k+1 (This refers to the posture at the next moment from the end.) The posture R recorded by the master hand when the pedal is closed m0 (t k ) to the current main hand stance R m (t k The rotation matrix of ).

Images