A method and device for predicting feedback force during virtual surgery

Abstract

This invention discloses a method and apparatus for predicting feedback force during virtual surgery, belonging to the technical field of feedback force prediction during virtual surgery. The feedback force prediction method includes: in response to collision detection during virtual surgery, acquiring data required for prediction and data required for updating; based on the data required for updating, dynamically updating a trained TCN-Transformer parallel enhanced temporal fusion model using a lightweight online learning strategy; inputting the data required for prediction into the model to obtain predicted tissue deformation values ​​and predicted feedback force values; calculating the tissue deformation prediction error based on the predicted tissue deformation values ​​and the calculated tissue deformation values; if the tissue deformation prediction error is less than a preset decision threshold, then the predicted feedback force value is used as the final prediction result. This invention significantly reduces the feedback force output delay by replacing calculation with prediction, and maintains prediction accuracy by dynamically updating to adapt to sudden operational changes.

Description

Technical Field

[0001] This invention relates to a method and apparatus for predicting feedback force during virtual surgery, belonging to the field of feedback force prediction technology during virtual surgery. Background Technology

[0002] Virtual reality is an interdisciplinary science and technology that primarily achieves its goals by creating virtual environments that combine the virtual and the real. High-quality virtual environments need to possess immersion, interactivity, and imaginative elements. In virtual surgical simulation systems, force feedback latency is a core bottleneck restricting the immersion and safety of operations, especially in liver cancer resection surgery training scenarios where operational precision requirements are extremely stringent. Therefore, overcoming latency limitations has become an urgent need for current surgical simulation systems to support precision medicine training.

[0003] Currently, the closed-loop force feedback process in traditional virtual surgical simulation systems is: user operation → collision detection → deformation calculation → feedback force calculation → force output. Among these, deformation calculation and feedback force calculation (such as the finite element method, FEM) account for over 83% of the computation time due to the nonlinear and large deformation characteristics of liver tissue. Traditional virtual surgical simulation system optimization schemes are typically based on historical data interpolation, which can only compensate for device transmission latency, or benefit from hardware and communication technology optimization, GPU (Graphics Processing Unit) acceleration, and haptic feedback algorithm optimization. These methods offer limited improvement to virtual surgery and do not help reduce latency in feedback force prediction. Furthermore, the frequent sudden changes in operation during virtual surgery can cause a surge in model prediction errors.

[0004] In summary, the latency of feedback force prediction for virtual surgery still needs improvement, and there is no method to cope with the surge in model prediction error caused by sudden changes in operation. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for predicting feedback force during virtual surgery, optimizing the delay in feedback force prediction, and maintaining prediction accuracy in the face of sudden operational changes.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] In a first aspect, the present invention provides a method for predicting feedback force during virtual surgery, comprising:

[0008] In response to collision detection during virtual surgery, the system acquires user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, calculated tissue deformation value, real-time tissue deformation, and real-time feedback force.

[0009] Based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the trained TCN-Transformer parallel enhanced temporal fusion model.

[0010] User operation parameters, biomechanical parameters, historical tissue deformation, and historical feedback force are input into the dynamically updated TCN-Transformer parallel enhanced temporal fusion model to obtain predicted values ​​of tissue deformation and feedback force.

[0011] The tissue deformation prediction error is calculated based on the predicted and calculated tissue deformation values.

[0012] If the tissue deformation prediction error is less than the preset decision threshold, the feedback force prediction value will be used as the final prediction result.

[0013] Furthermore, the TCN-Transformer parallel enhanced temporal fusion model includes parallel TCN branches and Transformer branches, as well as a fusion gating layer for fusing the outputs of the TCN branches and Transformer branches.

[0014] The TCN branch includes an input layer, an L-layer dilated convolutional block, and a first output layer connected in sequence. There are also residual connections between each layer of dilated convolutional blocks. The data frames that the L-layer dilated convolutional block can cover are greater than or equal to the average duration of the virtual surgery. When the virtual surgery is a virtual liver cancer resection surgery, L is greater than or equal to 5.

[0015] The Transformer branch includes an input layer, a multi-head attention layer, and a second output layer connected in sequence. The multi-head attention layer includes H parallel self-attention heads. When the virtual surgery is a virtual liver cancer resection surgery, H=8.

[0016] Furthermore, the step of dynamically updating the trained TCN-Transformer parallel enhanced temporal fusion model based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force using a lightweight online learning strategy includes:

[0017] Online feature vectors are constructed by using user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force as features.

[0018] Feature importance analysis is performed on the online feature vectors to obtain the feature importance score of each feature;

[0019] The weights in the trained TCN-Transformer parallel enhanced temporal fusion model are divided into a key weight set and a basic weight set. The key weight set includes the weights of the fusion gate layer, the weights of the first output layer, the weights of the second output layer, and the high-level convolutional weights in the TCN branch. The basic weight set includes the projection matrix in the residual connection, the attention projection matrix in the Transformer branch, and the low-level convolutional weights in the TCN branch. The high-level convolutional weights are the weights from the [L / 2]th dilated convolutional block to the Lth dilated convolutional layer in the TCN branch, and the low-level convolutional weights are the weights from the first dilated convolutional block to the [L / 2]-1th dilated convolutional block in the TCN branch. [ ] represents the floor function.

[0020] The decision function value of each feature is calculated using its feature importance score.

[0021] The weights in the key weight set and the basic weight set are selectively updated based on the decision function values ​​of each feature.

[0022] Furthermore, the feature importance analysis is performed on the online feature vectors to obtain the feature importance score of each feature, using the following formula:

[0023] ;

[0024] in, In the online feature vector, the first... Feature importance scores for each feature This represents the index of a feature in an online feature vector. Indicates from A random small batch of data samples from the middle. Represents a dynamic feature window. Indicates the first Online feature vector at time step, Indicates the first The first eigenvector in the online feature vector at time t Dimensional features, express Single-sample prediction loss, express about The partial derivatives, This indicates a search for the L2 norm;

[0025] in, The expression is: , Indicates the first The acceleration of the machine at any moment, Indicates the first The speed of the instrument at any given moment, Indicates the first The posture of the equipment at all times, Indicates the first The contact area between the instrument and the tissue at any given time. Indicates the first Organizational stiffness at any given moment Indicates the first The tissue viscosity coefficient at time t, Indicates the first Real-time organizational deformation at any moment. Indicates the first Real-time feedback capability Indicates a time index. Indicates the size of the window used for real-time feature monitoring;

[0026] The expression for the single-sample prediction loss is:

[0027] ;

[0028] in, This represents the predicted value of tissue deformation. This represents the true value of tissue deformation. This represents the predicted value of the feedback force. Indicates the true value of the feedback force. It represents the absolute value.

[0029] Furthermore, the decision function value of each feature is calculated based on its feature importance score using the following formula:

[0030] ;

[0031] in, In the online feature vector, the first... The decision function value of each feature. In the online feature vector, the first... Feature importance scores for each feature This represents the index of a feature in an online feature vector. In the online feature vector, the first... The feature importance score of each feature.

[0032] Furthermore, the selective updating of weights in the key weight set and the basic weight set based on the decision function values ​​of each feature is performed using the following formula:

[0033] ;

[0034] in, Indicates weight No. The value of the next iteration. Indicates weight No. The value of the next iteration. Indicates the learning rate of the weights. Indicates weight The weight gradient, Represents the online loss function. This represents the base layer update decay factor. Represents the key weight set, In the online feature vector, the first... The decision function value of each feature. This represents the index of a feature in an online feature vector. Represents the basic weight set, Indicates existence;

[0035] The expression for the online loss function is:

[0036] ;

[0037] in, Indicates from A random small batch of data samples from the middle. Represents a dynamic feature window. Indicates the first Online feature vector at time step, express The single-sample prediction loss.

[0038] Furthermore, the calculation of the tissue deformation prediction error based on the predicted and calculated tissue deformation values ​​is performed using the following formula:

[0039] ;

[0040] in, Indicates the first The error in predicting tissue deformation at time t. Indicates the first Predicted values ​​of tissue deformation at time t. Indicates the first Calculated values ​​of tissue deformation at time t. This indicates the search for the L2 norm.

[0041] Furthermore, the expression for the decision threshold is:

[0042] ;

[0043] in, Indicates the decision threshold. This represents the first balance coefficient. This represents the second balance coefficient, and , This indicates the preset maximum allowable deformation error. Indicates force-deformation sensitivity.

[0044] Furthermore, it also includes: dynamically updating the decision threshold simultaneously when dynamically updating the trained TCN-Transformer parallel enhanced temporal fusion model;

[0045] The decision threshold is dynamically updated using the following formula:

[0046] ;

[0047] in, Indicates the first The decision threshold for the next iteration. Indicates the first The decision threshold for the next iteration. Indicates the threshold learning rate. Represents the security loss function. Indicates the decision threshold. express for The partial derivatives;

[0048] The expression for the security loss function is as follows:

[0049] ;

[0050] in, Indicates the first Online feature vector at time step, Indicates from A random small batch of data samples from the middle. Represents a dynamic feature window. Indicates an indicator function, Indicates the first The error in predicting tissue deformation at time t. Indicates the first The calculated value of the feedback force at any given moment. Indicates the first The predicted value of the feedback force at any given time. This indicates that the absolute value is being calculated.

[0051] In a second aspect, the present invention provides a feedback force prediction device for virtual surgery, comprising:

[0052] The data acquisition module is configured to: in response to a collision detected during virtual surgery, acquire user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, calculated tissue deformation value, real-time tissue deformation, and real-time feedback force;

[0053] The model dynamic update module is configured to dynamically update the trained TCN-Transformer parallel enhanced temporal fusion model based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, using a lightweight online learning strategy.

[0054] The force-deformation prediction module is configured to input user operation parameters, biomechanical parameters, historical tissue deformation and historical feedback force into the dynamically updated TCN-Transformer parallel enhanced temporal fusion model to obtain predicted values ​​of tissue deformation and feedback force.

[0055] The error calculation module is configured to calculate the tissue deformation prediction error based on the predicted tissue deformation value and the calculated tissue deformation value.

[0056] The prediction result determination module is configured to: if the tissue deformation prediction error is less than the preset decision threshold, then the feedback force prediction value is taken as the final prediction result.

[0057] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0058] This invention provides a method and apparatus for predicting feedback force during virtual surgery. By designing a TCN-Transformer parallel enhanced temporal fusion model, it innovatively removes the time-consuming feedback force calculation from the critical path in the traditional force-deformation coupling calculation process through prediction-based computation, and directly predicts the feedback force using the model, significantly reducing force feedback latency. This model integrates the global attention mechanism of Transformer with the local feature extraction capability of TCN, thereby improving prediction accuracy. Based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the model to adapt to sudden operational changes during virtual surgery, improving the robustness of the model and maintaining high prediction accuracy in the face of sudden operational changes. Attached Figure Description

[0059] Figure 1 This is a flowchart of a feedback force prediction method during a virtual surgery provided in Example 1;

[0060] Figure 2 This is a flowchart of the liver cancer resection surgery simulation delay optimization method based on the feedback force prediction method in a virtual surgery process provided by the present invention, provided in Example 2;

[0061] Figure 3 This is a schematic diagram illustrating the execution of the lightweight online learning strategy provided in Example 2. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of the present invention more clearly, and should not be used to limit the scope of protection of the present invention.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides a method for predicting feedback force during virtual surgery, including:

[0065] In response to collision detection during virtual surgery, the system acquires user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, calculated tissue deformation values, real-time tissue deformation, and real-time feedback force.

[0066] Based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the trained TCN-Transformer parallel enhanced temporal fusion model.

[0067] User operation parameters, biomechanical parameters, historical tissue deformation, and historical feedback force are input into the dynamically updated TCN-Transformer parallel enhanced temporal fusion model to obtain predicted values ​​of tissue deformation and feedback force.

[0068] The tissue deformation prediction error is calculated based on the predicted and calculated tissue deformation values.

[0069] If the tissue deformation prediction error is less than the preset decision threshold, the feedback force prediction value will be used as the final prediction result.

[0070] This invention innovatively separates the time-consuming feedback force calculation from the critical path by designing a TCN-Transformer parallel enhanced temporal fusion model. By replacing computation with prediction, the model directly predicts the feedback force, significantly reducing the force feedback delay. This model integrates the global attention mechanism of Transformer with the local feature extraction capability of TCN, thereby improving prediction accuracy. Based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the model to adapt to sudden operational changes during virtual surgery, improving the model's robustness and maintaining high prediction accuracy in the face of sudden operational changes.

[0071] Example 2

[0072] This embodiment provides a method for predicting feedback force during virtual surgery, including the following steps:

[0073] S1. In response to a collision detected during the virtual surgery, acquire the data required for the prediction process.

[0074] In this embodiment, the aforementioned collision is achieved through optical guide dual-box flow-type hierarchical collision detection. Once a collision is detected, user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, calculated tissue deformation value, calculated feedback force value, real-time tissue deformation, and real-time feedback force are obtained in the prediction thread.

[0075] In this embodiment, the inputs to the TCN-Transformer parallel enhanced temporal fusion model include: user operation parameters, biomechanical parameters, historical tissue deformation states, and historical feedback forces. These inputs are selected from multi-source surgical parameters using the following method.

[0076] Pearson correlation analysis was used to calculate the correlation between multi-source surgical parameters and output parameters (including predicted tissue deformation and predicted feedback force), with the correlation represented by the Pearson correlation coefficient. Using preprocessed sample data (containing sample data for each parameter), Pearson correlation coefficients were calculated and analyzed between operational parameters, biomechanical parameters, historical parameters, and output parameters. Since the Pearson correlation coefficient ranges from -1 to 1, for cross-category correlation, variables strongly positively correlated with current tissue deformation and current feedback force were selected, while irrelevant variables were eliminated. For correlation within the same category of parameters, the absolute value of the correlation coefficient was considered; a value close to 1 indicated nearly identical parameters with collinearity redundancy, and one of these variables was eliminated. After analysis, the user operational parameters selected in this embodiment include: instrument acceleration, instrument speed, instrument posture, and the contact area between the instrument and tissue. The biomechanical parameters selected in this embodiment include tissue stiffness and tissue viscosity coefficient. The historical parameters selected in this embodiment include historical tissue deformation and historical feedback force.

[0077] S2. Based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the trained TCN-Transformer parallel enhanced temporal fusion model.

[0078] The inputs to the TCN-Transformer parallel enhanced temporal fusion model include user operation parameters, biomechanical parameters, historical tissue deformation states, and historical feedback forces. The input features are constructed using the following methods:

[0079] At the beginning of each training loop, the input features for the current time step are constructed based on the historical organizational deformation and historical feedback force from the previous time step, specifically including:

[0080] The first is constructed using the following formula. Input vector at time step:

[0081] ;

[0082] in, Indicates the first The input vector at time t, Indicates a time index. Indicates the first The acceleration of the machine at any moment, Indicates the first The speed of the instrument at any given moment, Indicates the first The posture of the equipment at all times, Indicates the first The contact area between the instrument and the tissue at any given time. Indicates the first Organizational stiffness at any given moment Indicates the first The tissue viscosity coefficient at time t, Indicates the first The state of organizational transformation at any given moment in history. Indicates the first The historical feedback force of every moment.

[0083] The first is constructed using the following formula. Input characteristics at time step:

[0084] ;

[0085] in, Indicates the first Input characteristics at time step Indicates the first The input vector at time t, Indicates the first The input vector at time t, Indicates the first The input vector at time t, Indicates the length of the history window.

[0086] The TCN-Transformer parallel enhanced temporal fusion model includes parallel TCN branches and Transformer branches, as well as a fusion gating layer for fusing the outputs of the TCN branches and the Transformer branches.

[0087] After inputting the input features into the TCN-Transformer parallel enhanced temporal fusion model, the processing of the input features includes: TCN branch processing, Transformer branch processing, and gated fusion.

[0088] TCN branch handling:

[0089] The core of the TCN branch is local feature extraction, which reduces the number of layers required for gradient propagation by expanding the receptive field. In this embodiment, an input layer, five dilated convolutional blocks, and a first output layer are used as the network structure of the TCN branch, which can cover 80 frames of historical data, meeting the average duration of liver cancer resection surgery. Each dilated convolutional block contains one dilated convolutional layer, an activation function, and a weight normalization operation, and has a residual connection to another dilated convolutional block at the end.

[0090] The TCN branch synchronously and in parallel receives input features and calculates dilated convolutional block features layer by layer. The dilated convolutional block features of layer 0 (input layer) are then processed. Input features ,Right now , No. Features of dilated convolution blocks The expression is:

[0091] ;

[0092] in, Indicates the first Features of dilated convolutional blocks in layered dilated convolutional blocks. The value of is an integer from 1 to 5. For activation function, Indicates the expansion rate The dilated convolution operation, The expansion rate of exponential growth This is the main reason for the expansion of the receptive field. Indicates the first Features of dilated convolutional blocks in layered dilated convolutional blocks. Indicates the first The kernel weights of dilated convolutional blocks, Indicates the first Bias term of layered void convolution block.

[0093] The features of the dilated convolutional block in the last dilated convolutional layer are residually concatenated with the input features to obtain the output features of the TCN branch. The residual concatenation is specifically performed using the following formula:

[0094] ;

[0095] in, This represents the output characteristics of the TCN branch. Indicates the first Features of dilated convolutional blocks in layered dilated convolutional blocks. , This represents the residual connection weight.

[0096] Transformer branch handling:

[0097] The core of the Transformer branch is to capture global dependencies through a self-attention mechanism. In this embodiment, an input layer, a single multi-head attention layer, and a second output layer are used as the network structure of the Transformer branch. The single multi-head attention layer contains eight self-attention heads, which precisely match the eight anatomical segments of the liver (Couinaud segmentation method), achieving a balance between computational efficiency and surgical safety. The multi-head attention layer executes the self-attention mechanism multiple times in parallel (multiple heads), with each head learning a different attention pattern, and the results are finally concatenated. The Transformer branch receives input features synchronously and in parallel.

[0098] Transformers themselves lack recurrent or convolutional structures and cannot directly perceive sequence order; therefore, positional information needs to be explicitly injected, i.e., positional encoding is required. For the first... The position encoding vector value corresponding to the given time is:

[0099] ;

[0100] in, Indicates the first The position encoding vector corresponding to the position at time 1 Values ​​of each dimension Indicates the first The position encoding vector corresponding to the position at time 1 Values ​​of each dimension Auxiliary variable representing dimension index.

[0101] By combining the position encoding vectors of all dimensions, we obtain the position encoding matrix. Then the position encoding matrix Add to input features In the process, the position-encoded input features are obtained. .

[0102] Position encoding matrix Add to input features In this case, it is performed using the following formula:

[0103] .

[0104] Each head will Three pre-trained projection matrices of query, key, and value , , Performing a cubic linear transformation yields three matrices: the query matrix. Key matrix Sum matrix :

[0105] .

[0106] query matrix Key matrix Sum matrix The input is fed into the multi-head attention layer, where the following calculations are performed:

[0107] ;

[0108] in, Indicates the first Output of size Indicates the attention head index, For activation function, Indicates the scaling factor. , This represents the total number of heads in the multi-head attention layer. .

[0109] The outputs of the eight heads are concatenated to obtain the output of the multi-head attention layer. :

[0110] ;

[0111] in, This indicates the output of the first header. Indicates the first Output of size This indicates a splicing operation. This indicates the output projection matrix.

[0112] After passing through a feedforward neural network and layer normalization, the output of the Transformer branch is obtained. :

[0113] ;

[0114] ;

[0115] in, Presentation layer normalization operation, It is a feedforward neural network. and This represents the weight matrix of the feedforward neural network. and This represents the bias term of the feedforward neural network.

[0116] Local temporal features of the input TCN branch output and global context features output by the Transformer branch The output of the TCN branch and the output of the Transformer branch The fused feature matrix is ​​formed by concatenating features along the feature dimensions; this is known as gated fusion. The gated fusion process is performed using the following formula:

[0117] ;

[0118] ;

[0119] in, Represents the gated signal matrix. It is the Sigmoid activation function. Represents the gate weight matrix. Indicates the gating bias term. Represents the fused feature matrix. Represents the Hadamard product. Represents a matrix of all ones. This indicates a splicing operation.

[0120] The fused feature matrix obtained above is input into the output layer to obtain the predicted output parameters:

[0121] ;

[0122] in, Indicates the first Predicted values ​​of tissue deformation and feedback force at time points. Indicates the first Predicted values ​​of tissue deformation at time t. Indicates the first The predicted value of the feedback force at any given time. Indicates the output layer weights. This represents the output layer bias term.

[0123] In this embodiment, a composite loss function is used for model training. Adjustments were made to the prediction accuracy of tissue deformation and feedback force, using a composite loss function. The calculation formula is as follows:

[0124] ;

[0125] in, Indicates the first Predicted values ​​of tissue deformation at time t. Indicates the first The predicted value of the feedback force at any given time. Indicates the first The true value of tissue deformation at time t. Indicates the first The true value of the feedback force at any moment. The weighting coefficients represent the deformation prediction error. This indicates the search for the L2 norm. The weighting coefficient represents the feedback force error.

[0126] After the above steps, this embodiment obtains the trained TCN-Transformer parallel enhanced temporal fusion model.

[0127] like Figure 3 As shown, a lightweight online learning strategy is used to dynamically update the trained TCN-Transformer parallel enhanced temporal fusion model:

[0128] Step 1: Construct an online feature vector by using user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force as features.

[0129] A dynamic feature window is constructed, through which multiple features (real-time surgical operation data stream) during the virtual surgical process are received in real time, resulting in the following set:

[0130] ;

[0131] in, Represents a dynamic feature window. Indicates the first Online feature vector at time step, Indicates the first The acceleration of the machine at any moment, Indicates the first The speed of the instrument at any given moment, Indicates the first The posture of the equipment at all times, Indicates the first The contact area between the instrument and the tissue at any given time. Indicates the first Organizational stiffness at any given moment Indicates the first The tissue viscosity coefficient at time t, Indicates the first Real-time organizational deformation at any moment. Indicates the first Real-time feedback capability Indicates a time index. This indicates the size of the window used for real-time feature monitoring.

[0132] Step 2: Perform feature importance analysis on the online feature vectors to obtain the feature importance score of each feature, that is, calculate the degree of influence of each feature on the prediction loss.

[0133] ;

[0134] in, In the online feature vector, the first... Feature importance scores for each feature This represents the index of a feature in an online feature vector. Indicates from The set of random small-batch data samples (random sampling is random sampling at fixed periods, randomly sampling small batches of data from within a window at fixed periods). , Represents a dynamic feature window. Indicates the first Online feature vector at time step, Indicates the first The first eigenvector in the online feature vector at time t Dimensional features, express Single-sample prediction loss, express about The partial derivatives, This indicates a search for the L2 norm;

[0135] The expression for the single-sample prediction loss is:

[0136] ;

[0137] in, This represents the predicted value of tissue deformation. This represents the true value of tissue deformation. This represents the predicted value of the feedback force. Indicates the true value of the feedback force. This indicates that the absolute value is being calculated.

[0138] Step 3: Divide the weights in the trained TCN-Transformer parallel enhanced temporal fusion model into key weight sets. and basic weight set The key weight set includes the weights of the fusion gate layer, the weights of the first output layer, the weights of the second output layer, and the high-level convolution weights in the TCN branch. The basic weight set includes the projection matrix in the residual connection, the attention projection matrix in the Transformer branch, and the low-level convolution weights in the TCN branch. The high-level convolution weights are the weights from the [L]th dilated convolution block to the Lth dilated convolution layer in the TCN branch, and the low-level convolution weights are the weights from the first dilated convolution block to the [L]-1th dilated convolution block in the TCN branch. Here, [ ] represents the floor function.

[0139] ;

[0140] ;

[0141] in, This represents the gate weight matrix of the fusion gate layer. This indicates the gate bias term of the fusion gate layer. Indicates the output layer weights. Indicates the output layer bias term. Indicates the first The kernel weights of dilated convolutional blocks, Indicates the residual connection weights. This represents the projection matrix of the query. The projection matrix representing the bond. The projection matrix representing the values.

[0142] Step 4: Calculate the decision function value of each feature based on its feature importance score.

[0143] ;

[0144] in, In the online feature vector, the first... The decision function value of each feature. In the online feature vector, the first... Feature importance scores for each feature This represents the index of a feature in an online feature vector. In the online feature vector, the first... The feature importance score of each feature.

[0145] Step 5: Selectively update the weights in the key weight set and the basic weight set based on the decision function values ​​of each feature.

[0146] ;

[0147] in, Indicates weight No. The value of the next iteration. Indicates weight No. The value of the next iteration. Indicates the learning rate of the weights. Indicates weight The weight gradient, Represents the online loss function. This represents the base layer update decay factor. Represents the key weight set, In the online feature vector, the first... The decision function value of each feature. This represents the index of a feature in an online feature vector. Represents the basic weight set, Indicates existence;

[0148] The expression for the online loss function is as follows:

[0149] ;

[0150] in, Indicates from A random small batch of data samples from the middle. , This represents a collection of user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback forces. Indicates the first Online feature vector at time step, express The single-sample prediction loss.

[0151] When dynamically updating the trained TCN-Transformer parallel enhanced temporal fusion model, the decision threshold is also dynamically updated simultaneously. The decision threshold is dynamically updated using the following formula:

[0152] ;

[0153] in, Indicates the first The decision threshold for the next iteration. Indicates the first The decision threshold for the next iteration. Indicates the threshold learning rate. Represents the security loss function. Indicates the decision threshold. express for The partial derivatives;

[0154] The expression for the safety loss function is as follows:

[0155] ;

[0156] in, Indicates the first Online feature vector at time step, Indicates from A random small batch of data samples from the middle. , Represents a dynamic feature window. Indicates an indicator function, Indicates the first The error in predicting tissue deformation at time t. Indicates the first The calculated value of the feedback force at any given moment. Indicates the first The predicted value of the feedback force at any given time. This indicates that the absolute value is being calculated.

[0157] S3. Input the user operation parameters, biomechanical parameters, historical tissue deformation and historical feedback force into the dynamically updated TCN-Transformer parallel enhanced temporal fusion model to obtain the predicted values ​​of tissue deformation and feedback force.

[0158] S4. Calculate the tissue deformation prediction error based on the predicted and calculated tissue deformation values.

[0159] ;

[0160] in, Indicates the first The error in predicting tissue deformation at time t. Indicates the first Predicted values ​​of tissue deformation at time t. Indicates the first Calculated values ​​of tissue deformation at time t. This indicates the search for the L2 norm.

[0161] S5. If the tissue deformation prediction error is less than the preset decision threshold, the feedback force prediction value will be used as the final prediction result (almost eliminating the delay of waiting for feedback force calculation); otherwise, the feedback force calculation value will be used as the final prediction result.

[0162] The expression for the decision threshold is:

[0163] ;

[0164] in, Indicates the decision threshold. This represents the first balance coefficient. This represents the second balance coefficient, and , This indicates the preset maximum allowable deformation error. Indicates force-deformation sensitivity; for highly sensitive tissues (such as tumors), a reduction in force-deformation sensitivity is necessary. Use strict decision thresholds; for tissues with lower sensitivity (such as normal tissues), increase... Use a lenient decision threshold.

[0165] In this embodiment, the above steps significantly reduce the delay of the output feedback force while ensuring the robustness of the model.

[0166] In this embodiment, the following is given: Figure 2 The method for delay optimization in liver cancer resection surgery simulation shown is based on a feedback force prediction method during virtual surgery provided in this embodiment, and specifically includes the following steps:

[0167] User actions.

[0168] Collision detection is performed using a light-guided dual-box flow-level collision detection system. If a collision is detected, the following two parallel threads are executed.

[0169] The two parallel threads are a calculation thread and a prediction thread. The prediction thread includes S1, S2 and S3 in this embodiment. It uses the TCN-Transformer parallel enhanced temporal fusion model to make predictions and obtain the predicted values ​​of feedback force and tissue deformation. The calculation thread adopts the traditional force feedback loop scheme, namely user operation → collision detection → tissue deformation calculation → feedback force calculation → feedback force output.

[0170] The predicted and calculated values ​​of tissue deformation are compared, and the error between them is calculated.

[0171] ;

[0172] in, Indicates the first The error in predicting tissue deformation at time t. Indicates the first Predicted values ​​of tissue deformation at time t. Indicates the first Calculated values ​​of tissue deformation at time t. This indicates the search for the L2 norm.

[0173] If the tissue deformation prediction error is less than the preset decision threshold, the predicted feedback force value is output as the final result to the force feedback device (almost eliminating the delay of waiting for feedback force calculation); otherwise, the calculated feedback force value is output as the final result to the force feedback device, and the TCN-Transformer parallel enhanced temporal fusion model is reset.

[0174] Example 3

[0175] The difference between this embodiment and embodiment 2 lies in the difference between S1 and S5. Specifically, in S1, the feedback force calculation value is not obtained. In S5, after judging the tissue deformation prediction error, if the tissue deformation prediction error is less than the preset decision threshold, the feedback force prediction value is used as the final prediction result (almost eliminating the delay of waiting for feedback force calculation).

[0176] By not obtaining the feedback force calculation value in S1 and instead using prediction to replace the calculation, the time-consuming feedback force calculation can be innovatively separated from the critical path in the traditional force-deformation coupling calculation process. The feedback force can be directly predicted by the model, which significantly reduces the force feedback delay.

[0177] Example 4

[0178] This embodiment provides a feedback force prediction device during virtual surgery, including:

[0179] The data acquisition module is configured to: in response to a collision detected during virtual surgery, acquire user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, calculated tissue deformation value, real-time tissue deformation, and real-time feedback force;

[0180] The model dynamic update module is configured to dynamically update the trained TCN-Transformer parallel enhanced temporal fusion model based on user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, using a lightweight online learning strategy.

[0181] The force-deformation prediction module is configured to input user operation parameters, biomechanical parameters, historical tissue deformation and historical feedback force into the dynamically updated TCN-Transformer parallel enhanced temporal fusion model to obtain predicted values ​​of tissue deformation and feedback force.

[0182] The error calculation module is configured to calculate the tissue deformation prediction error based on the predicted tissue deformation value and the calculated tissue deformation value.

[0183] The prediction result determination module is configured to: if the tissue deformation prediction error is less than the preset decision threshold, then the feedback force prediction value is taken as the final prediction result.

[0184] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for predicting a feedback force in a virtual surgery procedure, the method comprising: The method comprises the following steps: In response to detecting a collision during a virtual surgery process, obtaining user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, tissue deformation calculation value, real-time tissue deformation, and real-time feedback force; Based on the user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the trained TCN-Transformer parallel enhanced time series fusion model; Input the user operation parameters, biomechanical parameters, historical tissue deformation, and historical feedback force into the dynamically updated TCN-Transformer parallel enhanced time series fusion model to obtain tissue deformation prediction value and feedback force prediction value; Based on the tissue deformation prediction value and the tissue deformation calculation value, calculate the tissue deformation prediction error; If the tissue deformation prediction error is less than the preset decision threshold, the feedback force prediction value is taken as the final prediction result; The TCN-Transformer parallel enhanced time series fusion model comprises a parallel TCN branch and a Transformer branch, and a fusion gate layer for fusing the outputs of the TCN branch and the Transformer branch; The TCN branch comprises an input layer, L-layer cavity convolution blocks connected in sequence, and a first output layer, and a residual connection is further provided between each layer cavity convolution block, wherein the L-layer cavity convolution block can cover a data frame greater than or equal to the average duration of the virtual surgery, and when the virtual surgery is a virtual liver cancer resection surgery, L is greater than or equal to 5; The Transformer branch comprises an input layer, a multi-head attention layer, and a second output layer connected in sequence, and the multi-head attention layer comprises H parallel self-attention heads, and when the virtual surgery is a virtual liver cancer resection surgery, H=8; Based on the user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force, a lightweight online learning strategy is used to dynamically update the trained TCN-Transformer parallel enhanced time series fusion model, comprising: Taking the user operation parameters, biomechanical parameters, real-time tissue deformation, and real-time feedback force as features, an online feature vector is constructed; Perform feature importance analysis on the online feature vector to obtain feature importance scores for each feature; The weights in the trained TCN-Transformer parallel enhanced time series fusion model are divided into a key weight set and a basic weight set, the key weight set comprises the weights of the fusion gate layer, the weights of the first output layer, the weights of the second output layer, and the high-level convolution weights in the TCN branch, and the basic weight set comprises the projection matrix in the residual connection, the attention projection matrix in the Transformer branch, and the low-level convolution weights in the TCN branch; the high-level convolution weights are the weights of the [L / 2]th layer cavity convolution block to the Lth layer cavity convolution layer in the TCN branch, and the low-level convolution weights are the weights of the first layer cavity convolution block to the [L / 2]-1th layer cavity convolution block in the TCN branch; wherein [ ] represents the floor function; The decision function value of each feature is calculated according to the feature importance score of each feature; The weights in the key weight set and the basic weight set are selectively updated according to the decision function value of each feature.

2. The virtual haptic feedback force prediction method of claim 1, wherein, The feature importance analysis is performed on the online feature vector to obtain the feature importance score of each feature, and the following formula is used: ； wherein, denotes a feature importance score of the -th feature in the online feature vector, denotes an index of a feature in the online feature vector, denotes a set of randomly sampled mini-batch data samples from , wherein denotes a dynamic feature window, denotes the -th time instant, denotes the online feature vector at the -th time instant, denotes the -th feature in the online feature vector at the -th time instant, denotes the one-sample prediction loss of , wherein denotes the partial derivative of denotes the two-norm; wherein, the expression for is: , represents the instrument acceleration at the time index, represents the instrument velocity at the time index, represents the instrument pose at the time index, represents the instrument-tissue contact area at the time index, represents the tissue stiffness at the time index, represents the tissue viscosity at the time index, represents the real-time tissue deformation at the time index, represents the real-time feedback force at the time index, represents the time index, represents the size of the window for real-time monitoring features;​​​​​​​​ The expression of the single-sample prediction loss is as follows: ； wherein, represents a tissue deformation prediction value, represents a tissue deformation true value, represents a feedback force prediction value, represents a feedback force true value, represents an absolute value.

3. The method of claim 1, wherein, The decision function value of each feature is calculated according to the feature importance score of each feature, and the following formula is used: ； in, In the online feature vector, the first... The decision function value of each feature. In the online feature vector, the first... Feature importance scores for each feature This represents the index of a feature in an online feature vector. In the online feature vector, the first... The feature importance score of each feature.

4. The method of claim 1, wherein, The weights in the key weight set and the basic weight set are selectively updated according to the decision function value of each feature, and the following formula is used: ； wherein, denotes a weight a first iteration value, denotes a weight a first iteration value, denotes a weight learning rate, denotes a weight gradient of the weight, denotes an online loss function, denotes a base layer update decay factor, denotes a key weight set, denotes a decision function value for the th feature in the online feature vector, denotes an index of a feature in the online feature vector, denotes a base weight set, denotes exists; The expression of the online loss function is as follows: ； wherein, represents a set of randomly sampled mini-batch data samples from , represents a dynamic feature window, represents an online feature vector at the time instant, represents a single-sample prediction loss for .

5. The method of claim 1, wherein, The tissue deformation prediction error is calculated based on the tissue deformation prediction value and the tissue deformation calculation value, and the following formula is used: ； wherein, represents the tissue deformation prediction error at the time point represents the tissue deformation prediction error at the time point represents the tissue deformation prediction value at the time point represents the tissue deformation prediction value at the time point represents the tissue deformation calculation value at the time point represents the tissue deformation calculation value at the time point represents the two-norm.

6. The method of claim 1, wherein, The expression of the decision threshold is as follows: ； wherein denotes a decision threshold, denotes a first balancing coefficient, denotes a second balancing coefficient, and , denotes a preset maximum allowed deformation error, denotes a force-deformation sensitivity.

7. The method of claim 1, wherein, Further comprising: When the TCN-Transformer parallel enhanced time series fusion model trained is dynamically updated, the decision threshold is dynamically updated synchronously; The decision threshold is dynamically updated by the following formula: ； wherein, denotes the decision threshold of the th iteration, denotes the decision threshold of the th iteration, denotes the threshold learning rate, denotes the safety loss function, denotes the decision threshold, denotes the partial derivative of with respect to The expression of the safety loss function is as follows: ； wherein, represents the online feature vector at time represents a set of randomly sampled mini-batch data samples from represents a dynamic feature window, represents an indicator function, represents the tissue deformation prediction error at time represents the feedback force calculation at time represents the feedback force prediction at time represents an absolute value.​ 8. A device for predicting a feedback force in a virtual surgery procedure based on the method according to any one of claims 1 to 7, characterized in that, Comprising: The data acquisition module is configured to: in response to detecting a collision during a virtual surgery process, acquire user operation parameters, biomechanical parameters, historical tissue deformation, historical feedback force, tissue deformation calculation value, real-time tissue deformation and real-time feedback force; The model dynamic updating module is configured to: based on the user operation parameters, the biomechanical parameters, the real-time tissue deformation and the real-time feedback force, adopt a lightweight online learning strategy to dynamically update the TCN-Transformer parallel enhanced time series fusion model trained; The force-deformation prediction module is configured to: input the user operation parameters, the biomechanical parameters, the historical tissue deformation and the historical feedback force into the TCN-Transformer parallel enhanced time series fusion model after dynamic updating, to obtain a tissue deformation prediction value and a feedback force prediction value; The error calculation module is configured to: calculate a tissue deformation prediction error based on the tissue deformation prediction value and the tissue deformation calculation value; The prediction result determination module is configured to: if the tissue deformation prediction error is less than a preset decision threshold, the feedback force prediction value is taken as the final prediction result.

Images