A motion estimation method for the end effector of a cable-driven minimally invasive surgical instrument

Abstract

The present invention belongs to the field of robot kinematics, and discloses a motion estimation method for an end effector of a flexible cable-driven minimally invasive surgical instrument. The method is based on training a motion estimation model of the end effector of the flexible cable-driven minimally invasive surgical instrument. First, training samples are intercepted for a time series containing motor motion state information and input signals, and then data preprocessing is performed, including maximum and minimum value normalization processing and feature sequence reshaping; then, these processed training samples are sent to a three-layer bidirectional gated unit recurrent neural network with an attention mechanism, and an optimal neural network model is trained in combination with training sample labels; the preprocessed sample time series containing motor motion state information and input signals is input into the optimal deep neural network obtained by the above training, and finally the predicted joint position of the sample corresponding to the time point is output.

Description

Technical Field

[0001] The invention belongs to the field of robot kinematics, and in particular relates to a motion estimation method for an end effector of a flexible cable-driven minimally invasive surgical instrument. Background Art

[0002] Minimally invasive surgery (MIS) is a type of surgery that uses long, thin laparoscopic instruments through small incisions. Its advantages include accurate positioning, ease of implementation, increased success rate, reduced pain, and reduced recovery time. In recent years, more and more researchers, companies, and hospitals have begun to pay attention to the development, commercialization, and application of minimally invasive surgical robotic technology. The cable-driven robot can install the actuator on the base, away from the driven axis. It has the advantages of low inertia, simple structure, and remote drive. Therefore, it is widely used in rehabilitation and medical robots. Minimally invasive surgical instruments, as the end effectors of minimally invasive surgical robots, are key equipment for minimally invasive surgery. By driving the end effectors with cables, surgical operations such as cutting and suturing can be effectively completed in the complex surgical environment of minimally invasive surgery, thereby improving the success rate and completion of the surgery.

[0003] However, the cable-driven robot may be affected by nonlinear delay factors such as cable flexibility, fatigue, mechanical transmission and external interference. These factors often appear in complex surgical scenes, bringing new challenges to the control of the end effector in the robot manipulator. At the same time, since the end of the minimally invasive surgical instrument cannot integrate various sensors, that is, it is impossible to directly detect various information when the end effector contacts the human body, such as joint angle, clamping force, etc., the current common problem of minimally invasive surgical instruments is that the state information is difficult to estimate accurately, which will affect the accuracy of the motion control of the end joint of the surgical robot. In addition, most of the existing research focuses on modeling the predictor using prior knowledge, such as the trajectory planning method of the end effector of the target tracking performance test device with the publication number CN114675542A. The modeling method disclosed in the prior art will lead to insufficient adaptability to complex conditions in the surgical environment, resulting in a loss of accuracy in the motion control of the end joint. Summary of the invention

[0004] Considering that cable-driven robots are susceptible to nonlinear delay factors such as cable flexibility, fatigue, mechanical transmission and external interference, this paper proposes a data-driven end joint position estimation method in view of the fact that most existing studies use prior knowledge to estimate and model the end joint position.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] In the first aspect, the present invention provides a motion estimation model training method for an end effector of a flexible cable-driven minimally invasive surgical instrument. First, a flexible cable-driven end effector motion data set is established. Under a single motor state feature sequence, a training set and a test set are divided. The motor state feature sequence, i.e., sample features, is constructed using the time series of motor position, motor speed, and periodic sinusoidal signals input to the system. The motor state feature sequence training set samples and the motor state feature sequence test samples both include motor position, motor speed, and system input signals. The label sequence is the actual angle of the end effector joint. Using the motor state feature sequence, 49 time points are overlapped between the front and rear samples to construct training samples and test samples with a length of T = 50 continuous time points (0.5s). The label of each sample is the actual angle of the end effector joint at the last time point corresponding to the sample, i.e., the actual position information of the end joint.

[0007] Specifically, the motion estimation method of the cable-driven minimally invasive surgical instrument end effector of the present invention comprises the following sequential execution steps:

[0008] Step 1: The sample characteristics of all training samples in the training set can be expressed as The sample labels of all training samples in the training set are expressed as Where N represents the number of samples in the training set; then the sample features are processed using maximum and minimum value normalization;

[0009] Step 2: Reshape each sample feature sequence of the N training samples in step 1 into That is, the feature sequence corresponding to each d time point is used as the 3d-dimensional feature corresponding to a time point, where the integer d ≥ 1 represents the reshaping scale; the training set of the input network is obtained

[0010] Step 3: Construct a bidirectional gated recurrent neural network structure based on the attention mechanism as a deep neural network model for end joint position estimation; use the training set after shape reshaping in step 2 And use the corresponding sample label γ to train the deep neural network f(·) to obtain the terminal joint position estimator

[0011] Preferably, the motion estimation model training method for the end effector of a flexible cable-driven minimally invasive surgical instrument further comprises step 4, wherein the specific operation of step 4 comprises: for any motor state feature sequence test sample x in the test set, firstly, the maximum and minimum value normalization processing in step 1 and the sample feature sequence reshaping in step 2 are performed to obtain the processed test sample feature sequence x. And input it into the end joint position estimator trained in step 3 Make a prediction for each test sample and obtain the predicted position corresponding to the test sample The accuracy of the model is verified by comparing it with the sample labels in the test set.

[0012] Furthermore, the deep neural network in step 3 is a bidirectional gated recurrent neural network with a self-attention mechanism, and its network structure is as follows: the first layer is the input layer, and the input is reshaped into The number of time steps of the sample is The feature dimension of each time step corresponding to the time point is 3d; the second layer is the self-attention layer, which uses the scaled dot product attention mechanism, and its mapping dimension is set to m; the third to fifth layers are all bidirectional gated recurrent layers, and the number of bidirectional gated unit nodes in each layer is respectively The sixth layer is a fully connected output layer with 1 output node. The network uses the minimum mean square error (MSE) loss function and the adaptive moment estimation (Adam) optimizer.

[0013] Further preferably, in the deep neural network f(·) constructed in step 3, the implementation process of the scaled dot product self-attention mechanism of the second self-attention layer is:

[0014] Suppose a batch of N after maximum and minimum value normalization and feature sequence reshaping (0) samples, and the i-th sample Input into the self-attention layer, the final self-attention layer output corresponding to the sample is:

[0015]

[0016] Among them, softmax(·) represents the Softmax operation; After the mapping matrix W (Q) , W (K) , W (V) , the query, key and value are:

[0017]

[0018] The learnable mapping matrix W (Q) ,W (K) , m is the mapping dimension of the self-attention layer.

[0019] Further preferably, in the deep neural network f(·) constructed in step 3, the implementation process of the third to fifth bidirectional gated recurrent layers is:

[0020] Suppose the output of the second layer of the network at time t is

[0021]

[0022] Then the output of the fifth layer of the network at time t is

[0023]

[0024] in They represent the mapping of the third to fifth layers of the network, and the output dimensions of their bidirectional gating units are n (1) 、n (2) 、n (3) . Assume that the three bidirectional gated recurrent layers corresponding to the third to fifth layers are numbered k=1, 2, and 3 respectively, then the kth bidirectional gated recurrent layer The output corresponding to time t can be expressed as

[0025]

[0026] in and They represent the backward and forward hidden outputs of the bidirectional gated recurrent layer, respectively.

[0027] Further preferably, in the third to fifth bidirectional gated recurrent layers in the deep neural network f(·) in step 3, the backward and forward hidden output implementation process is:

[0028] Assume that the backward and forward hidden output and Any output of H can be expressed as t ,but:

[0029]

[0030] Where t+1 and t-1 correspond to the backward and forward hidden outputs respectively, and the backward and forward hidden outputs are calculated separately. ⊙ represents the bitwise multiplication operation of the matrix. represents a matrix of all 1s. The candidate hidden states in the formula are for:

[0031]

[0032] The linear weight of the network Bias Update Gate U t and reset gate R t They are:

[0033]

[0034] Including network weights and bias b (u) , b (r) , the activation function g(·) is taken as the sigmoid function.

[0035] Further preferably, in the deep neural network f(·) constructed in step 3, the implementation process of the sixth fully connected output layer is:

[0036] Let the output of the fifth layer be O (3) , then the output of the fully connected output layer of the current batch of samples is:

[0037]

[0038] Including network weights and bias

[0039] In the second aspect, the present invention provides a motion estimation method for the end effector of a flexible cable-driven minimally invasive surgical instrument, the estimation method specifically comprising the following steps: obtaining a sample of a state feature sequence of any motor to be tested, firstly subjecting the sample to be tested to maximum and minimum value normalization processing in step one and sample feature sequence reshaping in step two, obtaining the processed sample feature to be tested, and inputting it into the end joint position estimator trained in step three Predict each sample to be tested and obtain the predicted position corresponding to the sample to be tested.

[0040] Beneficial effects: Figure 1 As shown, the present invention provides a method for estimating the motion of the distal joint of a flexible cable-driven minimally invasive surgical instrument, preprocesses the training samples with motor motion state information, and reshapes the feature sequence of the sample features to model the sample feature sequence on a suitable time scale, combined with the labels corresponding to the training samples; then these training samples after feature sequence reshaping are sent to the deep neural network constructed by the present invention, that is, the bidirectional gated recurrent neural network structure based on the attention mechanism, to solve the optimal deep neural network joint position estimator. In the test phase, the optimal position estimator is used to input the test sample with motor motion state information and estimate its joint position.

[0041] Most of the existing studies on motion estimation of the end joints of cable-driven minimally invasive surgical instruments focus on modeling the predictor using prior knowledge. However, due to the interference of multiple factors in the surgical operation environment, it may lead to insufficient adaptability to the complex conditions of the surgical environment during the operation, resulting in loss of accuracy in the motion control of the end joints and inapplicability to complex environments. Therefore, a data-driven method for estimating the motion of the end effector of a minimally invasive surgical instrument is adopted in a method for estimating the motion of the end joints of a cable-driven minimally invasive surgical instrument in the present invention. Experimental results show that compared with the traditional regression method, this method has better performance and can accurately estimate the actual position of the end joint. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a method for estimating the motion of a distal joint of a cable-driven minimally invasive surgical instrument according to the present invention;

[0043] Figure 2 The system input signal time series with the maximum rotation angle of 27.5deg in the HFUT-CDEM dataset;

[0044] Figure 3 It is the actual angle time series of the end effector joint with the maximum rotation angle of 27.5 degrees in the HFUT-CDEM dataset;

[0045] Figure 4 It is the motor position time series of the maximum rotation angle of 27.5deg in the HFUT-CDEM dataset;

[0046] Figure 5 This is the motor speed time series at the maximum rotation angle of 27.5 degrees in the HFUT-CDEM dataset. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0048] like Figure 1 As shown, the method of the present invention first performs maximum and minimum value normalization processing on samples with motor motion state information, and reshapes the feature sequence of the processed sample features, and then inputs them into the constructed deep neural network predictor to predict the corresponding joint motion position. In the training stage, the samples containing motor motion state information are first normalized to the maximum and minimum values, and the feature sequence of the processed sample features is reshaped, and then combined with the corresponding training sample labels, the optimal terminal joint position estimator is solved; in the testing stage, the optimal terminal joint position estimator is used to estimate the terminal joint position of the test sample in combination with the test sample containing motor motion state information.

[0049] The method of the present invention is compared with the traditional regression method in terms of mean square error (MSE) and Pearson correlation coefficient (PCC) by experimental methods.

[0050] The following test example uses the HFUT-CDEM dataset to verify the effectiveness of the method of the embodiment of the present invention. The HFUT-CDEM dataset consists of 23 time series of periodic sinusoidal signals, corresponding to the maximum rotation angles of 2.5 degrees to 57.5 degrees, with a fixed interval of 2.5 degrees. Figure 2 , Figure 3 , Figure 4 and Figure 5 The figure shows the time series of the system input signal, the actual angle of the end effector joint, the motor position and the motor speed for the maximum rotation angle of 27.5deg in the above HFUT-CDEM dataset.

[0051] Each sequence contains 200 seconds, the sampling rate is 100Hz (20000 sampling points), and there are 11 detected time series, which come from two grating scale positions (mm) and speeds (mm / s) (4 sequences), four tension sensors (N) (4 sequences), motor positions (deg) and speeds (deg / s) (2 sequences), and actuator positions (deg) (1 sequence), where the actuator position is set as the target motion estimation. It should be noted that in the present invention, for each sequence corresponding to the maximum rotation angle, the motor position (deg) and speed (deg / s) and the time series of the input periodic sinusoidal signal (3 sequences in total) are used to form features.

[0052] For each rotation angle, the 20,000 sampling points are divided into 12,000 and 8,000 points as training and test sets, respectively. Then, every 50 time points are defined as a sample, and a 150-dimensional feature (50×3) is obtained. Considering the head-tail interference, a total of 11,250 training samples and 7,250 test samples are selected for each angle, so this experiment finally conducts 23 training and testing trials. Before further processing, the minimum and maximum normalization is first used for each feature. For the parameter setting of the proposed method, the number of nodes of the three-layer bidirectional gated recurrent network layer is set to 16, 16, and 8, respectively, while the mapping dimension m of the self-attention layer is set to the range of {5, 10, 15}. The sequence reshaping process is set to be reduced by a ratio of d=2 to obtain a 25×6 shape. The batch size is further set to 256, the maximum number of training rounds is 30, and the initial learning rate is {0.0001, 0.001, 0.01, 0.1}, and the best results are recorded.

[0053] Test Example 1

[0054] In the experiment, in order to reflect the effect of the method of the present invention, the methods used for comparison are: SVR (Support Vector Regression), LSTM (Long Short-Term Memory) / GRU (Gated Recurrent Unit), LSTM / GRU based on attention mechanism, Bi-LSTM (Bi-directional Long Short-Term Memory) / Bi-GRU (Bi-directional Gated Recurrent Unit) and Bi-LSTM based on attention mechanism (Attention-based Bi-LSTM). The method proposed in the present invention is Bi-GRU (Attention-based Bi-GRU) based on attention mechanism, which is a preferred embodiment in the invention content;

[0055] The LSTM and GRU based networks used in the comparison both adopt the same three recurrent layers as the proposed method. The comparison metrics include MSE and PCC. Table 1 shows the average MSE and PCC results of the training and testing trials for 23 angles. It should be noted that the results present the average and best average results within the range of the mapping dimension m of the attention layer. As can be seen from Table 1, the proposed method achieves better performance in most cases.

[0056]

[0057] Test Example 2

[0058] The focus is on studying the effects of different settings, namely the reshaping scale d and the position of the self-attention layer, on the proposed method. In the following tests, d is set in the range of 2, 5, and 10, and the position of the self-attention layer is set in the first, second, and third bidirectional gated recurrent layers, which correspond to the third, fourth, and fifth layers of the deep neural network f(·), respectively. A total of 9 different settings are generated, as shown in Table 2 below. The performance of the end-effector motion estimation methods under these 9 different settings is compared, and the corresponding MSE and PCC are shown in Table 2.

[0059] Experimental results show that the current method settings contribute to the performance of the proposed method when applied to the motion estimation of the cable-driven end effector. Among them, Attention is in the first bidirectional gated recurrent layer, and d=2 is the optimal setting of the present invention.

[0060] Table 2

[0061]

[0062] In summary, the bidirectional gated recurrent neural network based on the attention mechanism used in this embodiment first preprocesses and reshapes the input signal and short-term data containing the motor motion state, then trains and learns the deep neural network in combination with the training sample labels, and finally estimates the end joint position of the test sample. Experimental results show that compared with the traditional regression method, this method achieves better performance.

[0063] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A motion estimation model training method for a cable-driven minimally invasive surgical instrument end effector, characterized in that: The method comprises: firstly establishing a flexible cable driven end effector motion data set, dividing a training set and a test set under a single motor state feature sequence, using a time series of a motor position, a motor speed and a periodic sinusoidal signal input by a system to form a motor state feature sequence, i.e., a sample feature, wherein both the motor state feature sequence training set sample and the motor state feature sequence test sample include the motor position, the motor speed and the system input signal; the label sequence is the actual angle of the end effector joint; using the motor state feature sequence, overlapping 49 time points between the front and rear samples, constructing training samples and test samples of continuous time points with a length of 0.5 s, wherein each sample label is the actual angle of the end effector joint corresponding to the last time point of the sample, i.e., the actual position information of the end joint; wherein the sample features of all training samples in the training set can be expressed as , where the number of consecutive time points ; The sample labels of all training samples in the training set are expressed as ,in Represents the number of samples in the training set; then the sample features are processed using maximum and minimum value normalization; Then, for the training set Each sample feature sequence of training samples is reshaped into , where the number of consecutive time points , that is, every The feature sequence corresponding to a time point is used as the feature sequence corresponding to a time point dimensional feature, where integer Represents the reshaping scale; get the training set of the input network ; Then, a bidirectional gated recurrent neural network structure based on the attention mechanism is constructed as a deep neural network model for end joint position estimation. The deep neural network model includes an input layer, a self-attention layer, a three-layer bidirectional gated recurrent layer, and a fully connected output layer in sequence; the training set after shape reshaping is used , and use its corresponding sample label , training deep neural networks , get the end joint position estimator .

2. The motion estimation model training method for the end effector of a cable-driven minimally invasive surgical instrument according to claim 1 is characterized in that: The input layer of the deep neural network model is the first layer, and the input is reshaped into The number of time steps of the sample is , the feature dimension of each time step corresponding to the time point is ; The second layer is the self-attention layer, which uses the scaled dot product attention mechanism, and the mapping dimension of the self-attention layer is set to ; The third to fifth layers are bidirectional gated loop layers, and the number of bidirectional gated unit nodes in each layer is , , ; The sixth layer is the fully connected output layer, and the number of output nodes is 1; the network uses the minimum mean square error MSE loss function and the adaptive moment estimation Adam optimizer.

3. The motion estimation model training method for the end effector of a cable-driven minimally invasive surgical instrument according to claim 2 is characterized in that: Constructing a deep neural network In , the implementation process of the scaled dot product attention mechanism of the second self-attention layer is: Suppose a batch of common samples, and the Samples Input into the self-attention layer, the final self-attention layer output corresponding to the sample is: （1） in, Represents Softmax operation; After mapping matrix , , , get Query 、Key and Value They are: （2） The learnable mapping matrix , is the mapping dimension of the self-attention layer.

4. The motion estimation method of the end effector of a cable-driven minimally invasive surgical instrument according to claim 3, characterized in that: Constructing a deep neural network In the implementation process of the third to fifth bidirectional gated loop layers, the process is as follows: Assume that the second layer of the network is The output corresponding to time is ， （3） The fifth layer of the network is The output corresponding to time is ， （4） in , , They represent the mappings from the third to the fifth layer of the network, and the output dimensions of their bidirectional gating units are , , ; Let the three bidirectional gated loop layers corresponding to the third to fifth layers correspond to the numbers , then Bidirectional gated recurrent layer Corresponding time The output can be expressed as ， （5） in and They represent the backward and forward hidden outputs of the bidirectional gated recurrent layer, respectively.

5. The motion estimation model training method for the end effector of a cable-driven minimally invasive surgical instrument according to claim 4 is characterized in that: Deep Neural Networks The third to fifth bidirectional gated recurrent layers in , where the implementation process of backward and forward hidden output is: Assume that the backward and forward hidden output and Any output of can be expressed as ,but: ， （6） in , They correspond to the backward and forward hidden outputs respectively. The backward and forward hidden outputs are calculated separately. represents the bitwise matrix multiplication operation, represents a full 1 matrix; the candidate hidden state in the formula for: ， （7） The linear weight of the network , hour =m, , bias , update gate and reset gate They are: ， （8） Including network weights , , , and bias , , activation function Take it as the sigmoid function.

6. The motion estimation model training method for the end effector of a cable-driven minimally invasive surgical instrument according to claim 5 is characterized in that: Constructing a deep neural network In , the implementation process of the sixth fully connected output layer is: Suppose the output of the fifth layer is , then the output of the fully connected output layer of the current batch of samples is: ， （9） Including network weights and bias ,in Indicates the fifth layer of the network at time point Output.

7. The motion estimation model training method for the end effector of a cable-driven minimally invasive surgical instrument according to claim 1 is characterized in that: The motion estimation model training method further comprises the following steps: firstly testing a sample of any motor state feature sequence in a test set , perform maximum and minimum value normalization and sample feature sequence reshaping to obtain the processed test sample features , and input it into the trained end joint position estimator , predict each test sample and obtain the predicted position corresponding to the test sample Compare with the sample labels in the test set.

8. A motion estimation method for a cable-driven minimally invasive surgical instrument end effector, characterized in that: The estimation method specifically comprises the following steps: obtaining any motor state feature sequence sample to be tested, performing maximum and minimum value normalization processing and sample feature sequence reshaping on the sample by executing the motion estimation model training method of the end effector of a flexible cable-driven minimally invasive surgical instrument as claimed in claim 1, obtaining the processed sample feature to be tested, and inputting it into the trained end joint position estimator , predict each sample to be tested and obtain the predicted position corresponding to the sample to be tested.

Images