Autonomous planning method and system for surgical robots suitable for endoscopic surgery

Abstract

The application provides a surgical robot autonomous planning method and system suitable for a laparoscopic surgery scene, aiming to improve the intelligence and autonomy of the surgical robot in in-vivo surgery tasks such as suturing, intubation and ablation. By designing a low-dimensional feature extraction network, multi-element coupled feedback information such as multiple frames of video streams in the prior physician operation video and the configuration of the surgical robot is embedded into a low-dimensional manifold, and the low-dimensional feature code and the expert demonstration trajectory are saved to construct a prior physician operation atlas. To realize dynamic generation of the surgical robot operation trajectory, a twin network based on pose similarity is designed to extract key features during surgery, and an optimal instruction strategy is generated based on searching in the prior physician operation atlas, and the surgical robot completes online autonomous planning according to the optimal control instruction. The application realizes autonomous planning of the surgical robot, improves the intelligence and flexibility of online planning, and improves the efficiency of local trajectory planning by extracting trajectories from the prior physician operation atlas.

Description

Technical Field

[0001] This invention relates to the field of autonomous surgical robot technology, and more specifically, to an autonomous planning method and system for surgical robots suitable for laparoscopic surgery. Background Technology

[0002] Surgical robots are advanced medical devices used in surgical or laparoscopic procedures. They can perform various delicate operations inside the patient's body, leveraging their high stability, flexibility, and precision in the complex internal environment to significantly reduce the workload of surgeons, decrease the rate of errors, and improve the safety of surgeries. Surgical robots have long been a key area of ​​investment for countries worldwide, with the da Vinci surgical robot system being the most famous. China's development of surgical robots has concentrated in the last decade: examples include the "Touma" surgical robot developed by Minimally Invasive Surgical Co., Ltd.; the single-port laparoscopic robot from Surgiscience Co., Ltd.; and the "Divine Knife Hua Tuo" minimally invasive surgical robot developed by Shanghai Jiao Tong University.

[0003] In recent years, research on the autonomy of surgical robots has become a new research hotspot. Relevant national departments jointly issued the "Implementation Plan for the Application of 'Robot+'", proposing to improve the standard system for high-end medical devices and promote the autonomous development of medical robots. However, current in vivo surgical robot systems are mostly master-slave control systems, meaning that physicians remotely operate the surgical robot via a joystick, rather than the robot itself autonomously planning and operating. This results in low automation and fails to fully utilize the advantages of surgical robots. The autonomous development of surgical robots is the future trend; their high precision control and ease of feedback information acquisition offer significant application advantages. Therefore, research on autonomous planning and operation of surgical robots is essential.

[0004] Autonomous planning in surgical robots relies on the operational experience of prior surgeons and knowledge transfer techniques. The aim is for the robot to learn from expert demonstrations, achieving autonomous trajectories within the robot in various environments through knowledge graph construction and rapid online command generation. Currently, mainstream autonomous planning methods are primarily based on two categories: trajectory planning and control, and medical experience learning. Existing planning and control methods are mostly based on mission-critical waypoints and visual servoing technology, while medical experience learning often employs Gaussian mixture models or Markov decision-making. Due to their flexibility and efficiency, these methods have become important approaches for autonomous surgical planning in recent years. However, existing methods suffer from poor transfer and generalization performance and struggle to handle complex dynamic environments within the body, thus requiring further improvement.

[0005] Existing patent document CN114129263B discloses a surgical robot path planning method, system, device, and storage medium. The method includes: acquiring a 3D image model of the object to be operated on in a target area, and formulating one or more operation paths based on preset target points and target points; registering the robot intraoperatively; calculating each operation path to determine if the robot can plan all operation paths; if not, adjusting the object to be operated on until all operation paths are plannable, and sending all plannable operation paths to the robot's controller. This patent's path planning relies on a 3D image model, making it weak in handling complex surgical environments and learning from expert experience, resulting in relatively insufficient automation and flexibility.

[0006] Therefore, in order to enable surgical robots to achieve safe, efficient and autonomous online planning in complex dynamic scenarios, it is necessary to study an integrated method of environmental topology representation, task skill learning and online knowledge transfer. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this invention is to provide an autonomous planning method for surgical robots suitable for laparoscopic surgery.

[0008] The present invention provides an autonomous planning method for a surgical robot suitable for laparoscopic surgery, comprising the following steps:

[0009] Step S1: Extract the multivariate coupling information of the prior physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network to achieve feature encoding and construct a prior physician operation atlas.

[0010] Step S2: Using a similarity metric Siamese network, the low-dimensional features of the current frame obtained online are compared with the low-dimensional codes in the prior physician operation atlas one by one to obtain the similarity metric value, and the trajectory corresponding to the prior code feature with the highest similarity is extracted.

[0011] Step S3: Output the planned trajectory online based on the similarity metric.

[0012] Preferably, the multi-coupling information includes surgical field visual spatial morphological features and robot kinematic morphological features, and step S1 includes the following sub-steps:

[0013] Step S1.1, creating the visual spatial morphological features of the surgical field, including: using endoscopic images as guided input information, constructing the wound spatial morphology in the surgical field based on intelligent segmentation and 3D reconstruction algorithms, and completing feature extraction of multiple high-dimensional images by using a convolutional neural network structure to process single-frame image input and a recurrent neural network structure to process time series.

[0014] Step S1.2, creating robot kinematic morphological features, including: using a variational autoencoder neural network architecture and a self-supervised learning mechanism to represent the robot's configuration pose and trajectory tensor as a low-dimensional latent space descriptor;

[0015] Step S1.3, fusing visual features with kinematic features, including: using an attention mechanism to fuse the feature vectors obtained in steps S1.1 and S1.2, completing the feature fusion of visual and kinematic heterogeneous information, maximizing the preservation of the correlation between scene features and behavior patterns, and realizing multimodal representation and fusion, that is, achieving visual and kinematic fusion in the spatial dimension.

[0016] Preferably, in step S1.1, the visual spatial morphological features of the surgical field are represented as follows:

[0017]

[0018] in, This represents the input sequence of multiple frames, where n is the sequence step size, h is the height of a single image, and c is the number of channels in a single image. The output feature vector of the visual feature extraction network, where s represents the output stride and f img (·) represents a nonlinear visual feature extraction network.

[0019] Preferably, in step S1.2, the robot's kinematic morphological features are represented as follows:

[0020]

[0021] in, This represents the robot's configuration and pose tensor, where p represents the degrees of freedom of the surgical robot. Represents the robot trajectory tensor. Let m be the output feature vector of the kinematic feature extraction network, and f be the dimension of the feature vector. pose (·) represents a nonlinear kinematic feature extraction network.

[0022] Preferably, in step S1.3, visual features are used as query values ​​and kinematic morphological features are used as key values, and the cosine similarity between the two tensors is calculated:

[0023]

[0024] The similarity is scaled and normalized using the softmax function:

[0025] a i =softmax(Similarity) i )

[0026] Finally, attention values ​​are obtained by weighting according to weight coefficients to complete the fusion of visual and kinematic features:

[0027]

[0028] The feature fusion of heterogeneous visual and kinematic information is completed. The overall network architecture input and output are denoted as follows:

[0029] y i =f fusion (x i )

[0030] Among them, f fusion (·) represents a nonlinear feature extraction and feature fusion network, x i Include y i It is the overall low-dimensional encoding of the output.

[0031] Preferably, step 2 includes the following sub-steps:

[0032] Step S2.1: Train the similarity metric Siamese neural network to obtain the trained Siamese neural network;

[0033] Step S2.2: Measure the low-dimensional features of the current frame and the encoded features of the prior physician operation graph online to obtain a similarity metric value, and extract the trajectory corresponding to the prior encoded feature with the highest similarity.

[0034] When the surgical robot is operating online, the current scene image and robot configuration state are observed in real time and input into the trained Siamese neural network for similarity judgment to obtain a similarity metric. If the similarity metric is higher than the preset threshold, the segment with the most similar pose features under the "step" node corresponding to the current situation is extracted from the prior physician operation map generated in step S1 according to the size of the similarity metric.

[0035] Preferably, step S2.1 includes the following steps:

[0036] Step S2.1.1: Randomly select two sets of time series from the pre-collected data, and use a dynamic time-normalization design algorithm, as follows:

[0037] For two time series x1 and x2 with different input step sizes n1 and n2 respectively, first calculate their Euclidean distance:

[0038] D(i,j)=||x1(i)-x2(j)||, i=1,2,...,n1, j=1,2,...,n2

[0039] Next, the cumulative distance matrix is ​​calculated according to the recursive formula:

[0040] C(i,j)=D(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)}

[0041] Then, starting from the bottom right corner of the cumulative distance matrix, backtracking to the top left corner, we find a path with the minimum cumulative distance. This path is the best alignment path between two time series with different time lengths. The bottom right element of the cumulative distance matrix C(i,j) is the DTW distance between the two time series. After that, the two time segments are normalized.

[0042] Step S2.1.2: Establish a CNN-MLP Siamese neural network structure with shared weights. Input the data after normalization processing in step S2.1.1 into the feature extraction network in step S1 to obtain feature encoding, and input the feature encoding into the Siamese neural network for decoding.

[0043]

[0044] Where, Ti∈SE(3) represents the estimated relative pose with respect to the target point under the current feature encoding, R i It estimates the rotation matrix, t i It estimates the translation vector, f snn (·) represents a nonlinear twin neural network with shared weights in a CNN-MLP;

[0045] Step S2.1.3: Supervised training is performed using the pre-collected relative poses of the surgical robot's end effector and the target position as ground truth. The root mean square error loss function is used to calculate the Euclidean distance, guiding the network to perform backpropagation, thereby gaining the ability to judge the similarity of relative poses. The loss function is designed as follows:

[0046]

[0047] Where α is a weighting constant greater than 0. It is a true rotation matrix. It is the actual translation vector.

[0048] Preferably, step 3 includes the following steps:

[0049] Step S3.1: Train the expert trajectory decoding network to obtain the trained expert trajectory decoding network;

[0050] Step S3.2: Based on the similarity metric, generate local trajectories.

[0051] Local trajectory generation includes the following steps:

[0052] When the surgical robot is running online, the similarity metric value obtained in step S2.2 is obtained, the minimum matching threshold δ is set, and the similarity metric value is matched with the prior physician operation map.

[0053] If the similarity metric value is ≥ δ, the corresponding trajectory is extracted from the prior physician operation atlas and the trajectory is given as a local trajectory to the surgical robot for execution, thereby achieving rapid online trajectory planning.

[0054] If the similarity metric value is less than δ, the expert trajectory decoding network trained in step S3.1 takes the current frame as input, acquires the local trajectory online, and then hands the generated local trajectory over to the surgical robot for execution, thereby realizing online trajectory planning.

[0055] Preferably, step S3.1 includes the following steps:

[0056] Step S3.1.1: Extract low-dimensional feature codes and their corresponding trajectory information from the prior physician operation atlas, and align the encoded features through dynamic time-normalization.

[0057] Step S3.1.2: Establish a neural network decoder based on the diffusion model. The neural network decoder takes feedback information as input and constructs a mapping from multivariate coupling information to local trajectories. It calculates the probability distribution of the estimated expert trajectory by first adding noise to the low-dimensional feature encoding and then removing noise. Then, it generates a local trajectory dataset by sampling. The expression for expert trajectory generation using the diffusion model is as follows:

[0058] Y 0 =y i

[0059] Y γ =f diffuse (Y γ-1 ), γ=1,2,......,Γ

[0060]

[0061]

[0062] Where f diffuse (·) represents the noise addition process, f denoise (·) represents the nonlinear denoising function, Γ represents the total number of denoising steps, and the noise estimation loss function is used to guide the denoising network to backpropagate and update the network parameters:

[0063]

[0064] Step S3.1.3: Establish an MLP neural network to select the local trajectory with the highest probability likelihood from the sampling results. The set of local trajectories output in step S3.1.2 satisfies a probability distribution Y.i ~p(Y), i = 1, 2, ..., k, to obtain the optimal sampling results, train an MLP network to derive the probability likelihoods corresponding to each sampling output of the diffusion model:

[0065] E i =f sample (Y i ), i = 1, 2, ..., k

[0066] Where f sample (·) represents the nonlinear neural network function used to derive the optimal probability likelihood, and is obtained by selecting the largest E i The corresponding trajectory is taken as the optimal trajectory:

[0067] Y best =argmax i E i i = 1, 2, ..., k

[0068] This invention also provides an autonomous planning system for a surgical robot suitable for laparoscopic surgery, comprising:

[0069] Module S1 extracts multivariate coupling information from the prior physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network to achieve feature encoding and construct a prior physician operation atlas;

[0070] Module S2 employs a similarity metric Siamese network to compare the low-dimensional features of the current frame extracted online with the low-dimensional codes in the prior physician operation atlas one by one to obtain a similarity metric value, and extracts the prior code feature with the highest similarity.

[0071] Module S3 outputs the planned trajectory online based on the similarity metric.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] 1. In the decoding planning task, this invention finds the optimal low-dimensional manifold embedding encoding for multi-coupling information and achieves time step alignment by dynamically adjusting the time for different scenarios. Simultaneously, it encodes the time step information and the spatial structure information of the planning task, achieving time-space encoding registration, thus preserving the multi-faceted information features to the greatest extent and achieving a high degree of information fusion.

[0074] 2. This invention further enhances the autonomy of surgical robots, endowing them with the ability to autonomously complete online intraoperative planning. The surgical robot learns from prior physician teaching data and constructs a prior physician operation atlas for complex surgical planning tasks. This atlas contains low-dimensional manifold encoding information of the scene and actions, as well as trajectory teaching information, facilitating subsequent trajectory generation. When the surgical robot receives real-time feedback during surgery, it performs online scene measurement with the prior physician operation atlas. Subsequently, based on the measurement, it generates local trajectories, thereby achieving autonomous planning tasks for the surgical robot and improving the intelligence and flexibility of online planning. Compared with existing traditional trajectory planning methods, extracting trajectories from the prior physician operation atlas significantly improves the efficiency of local trajectory planning. Attached Figure Description

[0075] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0076] Figure 1 This is a flowchart illustrating an autonomous planning method for a surgical robot applicable to laparoscopic surgery scenarios, as described in this embodiment of the invention.

[0077] Figure 2 This is a schematic diagram of an autonomous planning system for a surgical robot applicable to laparoscopic surgery scenarios, as described in an embodiment of the present invention.

[0078] Figure 3 This is a block diagram of the similarity metric twin neural network training module in an embodiment of the present invention;

[0079] Figure 4 This is a block diagram of the expert trajectory decoding neural network training module in an embodiment of the present invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0081] This invention provides a method and system for autonomous planning of surgical robots in laparoscopic surgery scenarios, aiming to effectively improve the intelligence and autonomy of surgical robots in in vivo surgical tasks such as suturing, cannulation, and ablation. To enable physician skill learning, a low-dimensional feature extraction network is designed to embed multi-frame video streams and surgical robot configurations from prior physician operation videos into a low-dimensional manifold. The low-dimensional feature encoding and expert demonstration trajectories are then stored to construct a prior physician operation atlas. To achieve dynamic generation of the surgical robot's operation trajectory, a Siamese network based on pose similarity is designed to extract key intraoperative features. Based on this, the optimal instruction strategy is searched and generated in the prior physician operation atlas. The surgical robot then completes online autonomous planning during the operation according to the optimal control instructions.

[0082] Example 1:

[0083] Figure 1 This is a flowchart of an autonomous planning method for surgical robots applicable to laparoscopic surgery scenarios, as described in an embodiment of the present invention.

[0084] like Figure 1 As shown, this embodiment provides an autonomous planning method for surgical robots suitable for laparoscopic surgery scenarios, including the following steps:

[0085] Step S1: Extract the multivariate coupling information of the prior physician operation video stream. Specifically, by designing a feature extraction network and a feature fusion network, the multivariate coupling information such as the intraoperative online image, the trajectory of past frames, and the robot configuration is embedded in a low-dimensional latent space manifold to achieve feature encoding and construct a prior physician operation atlas.

[0086] More specifically, the multi-coupling information includes the visual spatial morphological features of the surgical field and the kinematic morphological features of the robot. Step S1 includes the following sub-steps:

[0087] Step S1.1: Create the visual spatial morphological features of the surgical field. Specifically, by using endoscopic images as guiding input information, the wound spatial morphology in the surgical field is constructed based on intelligent segmentation and three-dimensional reconstruction algorithms. In this embodiment, feature extraction of multiple high-dimensional images is completed by using a convolutional neural network structure to process single-frame image input and a recurrent neural network structure to process time series.

[0088] Specifically, the visual spatial morphological characteristics of the surgical field are represented as follows:

[0089]

[0090] in, This represents the input sequence of multiple frames, where n is the sequence step size, h is the height of a single image, and c is the number of channels in a single image. The output feature vector of the visual feature extraction network, where s represents the output stride and f img (·) represents a nonlinear visual feature extraction network.

[0091] Step S1.2: Create robot kinematic morphological features. Specifically, with the help of variational autoencoder neural network architecture and self-supervised learning mechanism, the robot configuration posture and robot trajectory tensor are represented as low-dimensional latent space descriptors.

[0092] Specifically, the kinematic morphological characteristics of the robot are represented as follows:

[0093]

[0094] in, This represents the robot's configuration and pose tensor, where p represents the degrees of freedom of the surgical robot. Represents the robot trajectory tensor. Let m be the output feature vector of the kinematic feature extraction network, and f be the dimension of the feature vector. pose (·) represents a nonlinear kinematic feature extraction network.

[0095] Step S1.3: The visual features and kinematic features are fused. Specifically, an attention mechanism is used to fuse the feature vectors obtained in steps S1.1 and S1.2.

[0096] Specifically, visual features are used as query values, and kinematic morphological features are used as key values. The cosine similarity between the two tensors is calculated as follows:

[0097]

[0098] The similarity is scaled and normalized using the softmax function:

[0099] a i =softmax(Similarity) i )

[0100] Finally, attention values ​​are obtained by weighting according to weight coefficients to complete the fusion of visual and kinematic features:

[0101]

[0102] This step completes the feature fusion of visual and kinematic heterogeneous information, maximizing the preservation of the correlation between scene features and behavioral patterns, and achieving multimodal representation and fusion, that is, achieving visual and kinematic fusion in the spatial dimension. The overall network architecture input and output are denoted as:

[0103] y i =f fusion (xi )

[0104] Among them, f fusion (·) represents a nonlinear feature extraction and feature fusion network, x i Include y i It is the overall low-dimensional encoding of the output.

[0105] Step S2: Using a similarity metric Siamese network, the low-dimensional features of the current frame obtained online are compared with the low-dimensional codes in the prior physician operation atlas one by one to obtain the similarity metric value, and the trajectory corresponding to the prior code feature with the highest similarity is extracted.

[0106] Specifically, step 2 includes the following sub-steps:

[0107] Step S2.1: Train the similarity metric Siamese neural network to obtain the trained Siamese neural network.

[0108] Figure 3 This is a block diagram of the similarity metric twin neural network training module in an embodiment of the present invention.

[0109] like Figure 3 As shown, step S2.1 includes the following steps:

[0110] Step S2.1.1: Randomly select two sets of time series from the pre-collected data, and use Dynamic Time Warping (DTW) to design a normalization algorithm to ensure the temporal consistency of the output of the feature encoding network and reduce semantic discrepancies. The specific implementation method is as follows:

[0111] For two time series x1 and x2 with different input step sizes n1 and n2 respectively, first calculate their Euclidean distance:

[0112] D(i,j)=||x1(i)-x2(j)||, i=1,2,...,n1, j=1,2,...,n2

[0113] Next, the cumulative distance matrix is ​​calculated according to the recursive formula:

[0114] C(i,j)=D(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)}

[0115] Then, starting from the bottom right corner of the cumulative distance matrix, backtracking to the top left corner, a path with the minimum cumulative distance is found. This path is the best alignment path between two time series with different time lengths. The bottom right element of the cumulative distance matrix C(i,j) is the DTW distance between the two time series. After that, the two time segments are normalized.

[0116] Step S2.1.2: Establish a CNN-MLP Siamese neural network structure with shared weights. Input the data after normalization processing in step S2.1.1 into the feature extraction network in step S1 to obtain feature encoding, and input the feature encoding into the Siamese neural network for decoding.

[0117]

[0118] Among them, T i ∈SE(3) represents the estimated relative pose with respect to the target point under the current feature encoding, R i It estimates the rotation matrix, t i It estimates the translation vector, f snn (·) represents a nonlinear twin neural network of CNN-MLP with shared weights.

[0119] Step S2.1.3: Supervised training is performed using the pre-collected relative poses of the surgical robot's end effector and the target position as ground truth. The root mean square error (RMSE) loss function is used to calculate the Euclidean distance, guiding the network to perform backpropagation, thereby gaining the ability to judge the similarity of relative poses. The loss function is designed as follows:

[0120]

[0121] Where α is a weighting constant greater than 0. It is a true rotation matrix. It is the actual translation vector.

[0122] Step S2.2: Measure the low-dimensional features of the current frame and the encoded features of the prior physician operation graph online to obtain a similarity metric value, and extract the trajectory corresponding to the prior encoded feature with the highest similarity.

[0123] Specifically, step S2.2 includes the following steps:

[0124] When the surgical robot is operating online, the current scene image and robot configuration state are observed in real time and input into the trained Siamese neural network for similarity judgment to obtain a similarity metric. If the similarity metric is higher than a preset threshold, the segment with the most similar pose features under the "step" node corresponding to the current situation is extracted from the prior physician operation map generated in step S1 according to the magnitude of the similarity metric. The "step" node is a different stage or action node in the prior physician operation process. The extracted segment provides a reference for the output trajectory of the subsequent step S3.

[0125] Step S3: Output the planned trajectory online based on the similarity metric.

[0126] Specifically, step 3 includes the following steps:

[0127] Step S3.1: Train the expert trajectory decoding network to obtain the trained expert trajectory decoding network.

[0128] Figure 4 This is a block diagram of the expert trajectory decoding neural network training module in an embodiment of the present invention.

[0129] like Figure 4 As shown, step S3.1 includes the following steps:

[0130] Step S3.1.1: Extract low-dimensional feature codes and their corresponding trajectory information from the prior physician operation atlas, and align the encoded features by normalizing them through dynamic time-warping (DTW).

[0131] Step S3.1.2: Establish a neural network decoder based on the diffusion model. The neural network decoder takes feedback information such as surgical field image and robot configuration as input, constructs a mapping from multivariate coupling information to local trajectories, calculates the probability distribution of the estimated expert trajectory by first adding noise to the low-dimensional feature encoding and then denoising it, and then generates a local trajectory dataset by sampling. The expression for expert trajectory generation by the diffusion model is as follows:

[0132] Y 0 =y i

[0133] Y γ =f diffuse (Y γ-1 ), γ=1,2,......,Γ

[0134]

[0135]

[0136] Where f diffuse (·) represents the noise addition process, f denoise (·) represents the nonlinear denoising function, Γ represents the total number of denoising steps, and the noise estimation loss function is used to guide the denoising network to backpropagate and update the network parameters:

[0137]

[0138] Step S3.1.3: Establish an MLP neural network to select the local trajectory with the highest probability likelihood from the sampling results. The set of local trajectories output in step S3.1.2 satisfies a probability distribution Y. i ~p(Y), i = 1, 2, ..., k, to obtain the optimal sampling results, train an MLP network to derive the probability likelihoods corresponding to each sampling output of the diffusion model:

[0139] E i =fsample (Y i ), i = 1, 2, ..., k

[0140] Where f sample (·) represents the nonlinear neural network function used to derive the optimal probability likelihood, and is obtained by selecting the largest E i The corresponding trajectory is taken as the optimal trajectory:

[0141] Y best =argmax i E i i = 1, 2, ..., k

[0142] Step S3.2: Generate local trajectories based on similarity metrics.

[0143] Specifically, local trajectory generation includes the following steps:

[0144] When the surgical robot is running online, the similarity metric value obtained in step S2.2 is acquired, the minimum matching threshold δ is set, and the similarity metric value is matched with the prior physician operation map.

[0145] If the similarity metric value is ≥δ, it means that there are features in the prior physician atlas that match the current frame. Then, the corresponding trajectory is extracted from the prior physician operation atlas and the trajectory is given as a local trajectory to the surgical robot for execution, so as to achieve fast online trajectory planning.

[0146] If the similarity metric value is less than δ, it means that there are no features in the prior physician atlas that match the current frame, and the trajectory cannot be directly extracted. In this case, the expert trajectory decoding network trained in step S3.1 is used to obtain the local trajectory online by taking the current frame as input. The generated local trajectory is then handed over to the surgical robot for execution to achieve online trajectory planning.

[0147] Example 2:

[0148] The present invention also provides an autonomous planning system for surgical robots in laparoscopic surgery scenarios. The autonomous planning system for surgical robots in laparoscopic surgery scenarios can be implemented by executing the process steps of the autonomous planning method for surgical robots in laparoscopic surgery scenarios. That is, those skilled in the art can understand the autonomous planning method for surgical robots in laparoscopic surgery scenarios as a preferred implementation of the autonomous planning system for surgical robots in laparoscopic surgery scenarios.

[0149] Figure 2 This is a schematic diagram of an autonomous planning system for a surgical robot applicable to laparoscopic surgery scenarios, as described in an embodiment of the present invention.

[0150] like Figure 2As shown, this embodiment provides an autonomous planning system for surgical robots suitable for laparoscopic surgery scenarios, including:

[0151] Module S1 extracts multivariate coupling information from the prior physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network to achieve feature encoding, and constructing a prior physician operation atlas.

[0152] Module S2 employs a similarity metric Siamese network to compare the low-dimensional features of the current frame extracted online with the low-dimensional codes in the prior physician operation atlas one by one to obtain a similarity metric value, and then extracts the prior code feature with the highest similarity.

[0153] Module S3 outputs the planned trajectory online based on the similarity metric.

[0154] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0155] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A surgical robotic autonomous planning system suitable for use in endoscopic surgery, characterized in that, Comprise: The module M1 is used for realizing step S1, extracting the multivariate coupling information of the prior physician operation video stream, comprising: embedding the multivariate coupling information into a low-dimensional hidden space manifold by designing a feature extraction network and a feature fusion network, realizing feature coding, and constructing a prior physician operation atlas; The multivariate coupling information comprises surgical field visual space morphological features and robot kinematics morphological features; The module M2 is used for realizing step S2, adopting a similarity measurement twin network to compare the low-dimensional features of the current frame obtained online with the low-dimensional codes in the prior physician operation atlas one by one, obtaining a similarity measurement value, and extracting the trajectory corresponding to the highest similarity prior coding feature; The module M3 is used for realizing step S3, outputting a planning trajectory online according to the similarity measurement value; The step S3 comprises the following substeps: Step S3.1, training an expert trajectory decoding network to obtain a trained expert trajectory decoding network; Step S3.2, based on the similarity measurement value, generating a local trajectory, the local trajectory generation comprising: acquiring the similarity measure value when the surgical robot is online measuring matching the similarity measure value with the prior physician operation atlas; If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value If the similarity measure value 2. The surgical robot autonomous planning system suitable for use in a laparoscopic surgery of claim 1, wherein, The step S1 comprises the following substeps: Step S1.1, creating surgical field visual space morphological features, comprising: taking a laparoscope image as guided input information, constructing a surgical field wound space morphology based on intelligent segmentation and three-dimensional reconstruction algorithm, and completing feature extraction of multi-frame high-dimensional images by adopting a convolutional neural network structure to process single-frame picture input and adopting a recurrent neural network structure to process time series; Step S1.2, creating robot kinematics morphological features, comprising: using a variational autoencoder neural network framework and a self-supervised learning mechanism to represent robot configuration posture and robot trajectory tensor as low-dimensional hidden space descriptors; Step S1.3, fusing visual features and kinematics features, comprising: using an attention mechanism to fuse the feature vectors obtained in steps S1.1 and S1.2, completing feature fusion of visual and kinematics heterogeneous information, maximizing the correlation between scene features and behavior patterns, realizing multi-modal representation and fusion, i.e. achieving fusion of vision and kinematics in spatial dimension.

3. The surgical robot autonomous planning system suitable for use in a laparoscopic surgery of claim 2, wherein, In the step S1.1, the surgical field visual space morphological features are represented as: wherein, represents an input multi-frame picture sequence, is a sequence step, is a height of a single picture, is a width of a single picture, is a channel number of a single picture, is an output feature vector of a visual feature extraction network, represents an output step, represents a nonlinear visual feature extraction network.

4. The autonomous planning system for surgical robots suitable for use in endoscopic surgery of claim 2, wherein, In the step S1.2, the robot kinematics morphological features are represented as: wherein, represents a robot configuration pose tensor, is a surgical robot degree of freedom, represents a robot trajectory tensor, is an output feature vector of the kinematic feature extraction network, is a dimension of the feature vector, represents a nonlinear kinematic feature extraction network.

5. The self-planning system for surgical robots for use in endoscopic surgery of claim 2, wherein, In the step S1.3, taking the visual features as query values and the kinematics morphological features as key values, the cosine similarity of the two tensors is calculated: The similarity is scaled and normalized by using a softmax function: Finally, the attention value is obtained by weighting according to the weight coefficient to complete fusion of visual features and kinematics features: The feature fusion of visual and kinematics heterogeneous information is completed, and the overall network architecture input and output are denoted as: wherein, represents a nonlinear feature extraction and feature fusion network, comprises , is the output overall low-dimensional encoding.

6. The surgical robot autonomous planning system suitable for use in a laparoscopic surgery of claim 1, wherein, The step S2 comprises the following substeps: Step S2.1, training a similarity measurement twin neural network to obtain a trained twin neural network; Step S2.2, the low-dimensional features of the current frame are measured online with the encoded features of the prior physician operation atlas to obtain a similarity value, and the trajectory corresponding to the highest similarity prior encoded feature is extracted; When the surgical robot is online, the current scene image and the robot configuration state are observed in real time and input to the trained twin neural network for similarity judgment to obtain a similarity value, and if the similarity value is higher than a preset threshold, the segment with the most similar pose feature under the "module" node corresponding to the current situation is extracted from the prior physician operation atlas generated by the module M1 according to the size of the similarity value.

7. The surgical robot autonomous planning system suitable for use in a laparoscopic surgery of claim 6, wherein, The step S2.1 includes the following sub-steps: Step S2.1.1, two groups of time series are randomly selected from the pre-acquired data, and a dynamic time warping design normalization algorithm is used, which is as follows: For two time series with different step lengths and , the step lengths are and , respectively, the Euclidean distance between them is calculated first: Then, the cumulative distance matrix is calculated according to the recursive formula: Then, a path with the minimum accumulated distance is found by backtracking from the bottom right corner to the top left corner of the accumulated distance matrix, which is the optimal alignment path between the two time series of different time scales, the bottom right element of the accumulated distance matrix is the DTW distance between the two time series, and then the two time segments are normalized. Step S2.1.2, a CNN-MLP twin neural network structure with shared weights is established, the data normalized by the step S2.1.1 is input to the feature extraction network of the module M1 to obtain feature encoding, and the feature encoding is input to the twin neural network for decoding; wherein, represents an estimated relative pose of the target point under the current feature encoding, is an estimated rotation matrix, is an estimated translation vector, represents a CNN-MLP nonlinear twin neural network with shared weights; Step S2.1.3, the relative pose between the end of the surgical robot and the target position pre-acquired is used as the true value for supervised training, the Euclidean distance is calculated by using the root mean square error loss function, the network is guided to perform back propagation, and then the ability to judge the similarity of the relative pose is obtained, and the loss function is designed as follows: wherein is a weight constant greater than 0, is a real rotation matrix, is a real translation vector.

8. The surgical robot autonomous planning system suitable for use in a laparoscopic surgery of claim 1, wherein, The step S3.1 includes the following steps: Step S3.1.1, the low-dimensional feature encoding and the corresponding trajectory information are extracted from the prior physician operation atlas, and the encoded features are aligned by dynamic time warping normalization; Step S3.1.2, a neural network decoder based on a diffusion model is established, the neural network decoder takes feedback information as input, constructs a mapping of multi-element coupled information to local trajectory, calculates the probability distribution of the estimated expert trajectory by first adding noise to the low-dimensional feature encoding and then denoising, and generates a local trajectory dataset by sampling, and the expression of the diffusion model for expert trajectory generation is as follows: wherein represents a noise adding process, represents a nonlinear denoising function, denotes the total number of denoising steps, and the noise estimation loss function is used to guide the backpropagation update of the network parameters of the denoising network: Step S3.1.3, establishing the MLP neural network to select the local trajectory with the highest probability likelihood of the sampling result, the local trajectory set output by step S3.1.2 satisfies a probability distribution In order to obtain the optimal sampling result, the MLP network is trained to derive the probability likelihood corresponding to each sampling output of the diffusion model wherein represents a non-linear neural network function that derives an optimal probability likelihood, and by selecting the maximum the trajectory corresponding to the maximum value as the optimal trajectory: 。

Images