Surgical robot autonomous planning method and system suitable for endoscopic surgery

Abstract

The invention provides a surgical robot autonomous planning method and system suitable for an endoscopic surgery scene, and aims to improve the intelligence and autonomy of a surgical robot in in-vivo surgical tasks such as suturing, intubation and ablation. By designing a low-dimensional feature extraction network, low-dimensional manifold embedding is achieved on multi-frame video streams, surgical robot configuration and other multi-element coupling feedback information in a priori physician operation video, and low-dimensional feature codes and expert demonstration tracks are stored to construct a priori physician operation atlas. In order to realize dynamic generation of an operation track of the surgical robot, a twin network based on pose similarity is designed to extract key features in the operation, an optimal instruction strategy is searched and generated in a priori physician operation map, and the surgical robot completes online autonomous planning in the operation according to an optimal control instruction. According to the invention, the autonomous planning task of the surgical robot is realized, the intelligence and flexibility of online planning are improved, and the efficiency of local trajectory planning is improved by extracting the trajectory from the priori physician operation map.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous surgical robots, and in particular to an autonomous planning method and system for surgical robots suitable for laparoscopic surgery. Background Art

[0002] A surgical robot is an advanced medical device used for surgical or laparoscopic surgery. It can perform various delicate operations inside the patient's body. It can give full play to its advantages of high stability, flexible operation, and motion accuracy in complex internal environments, greatly reducing the intensity of surgery for surgeons, reducing the rate of misoperation, and improving the safety of internal surgery. Surgical robots have always been a key investment direction for countries around the world, and the most famous of them is the da Vinci surgical robot system. The development of surgical robots in China has been concentrated in the past decade: such as the "Touma" surgical robot developed by MicroPort; the single-port laparoscopic robot of Surui; and the "Shendao Huatuo" minimally invasive surgical robot developed by Shanghai Jiaotong University.

[0003] In recent years, the research on the autonomous process of surgical robots has become a new research hotspot. The relevant national departments jointly issued the "Robot +" Application Action Implementation Plan, proposing the need to improve the high-end medical equipment standard system and promote the autonomous development of medical robots. However, the current in vivo surgical robot system is mostly a master-slave control mode, that is, the doctor remotely operates the surgical robot through a remote joystick, rather than the surgical robot to achieve autonomous planning and operation. The degree of automation is low and the advantages of surgical robots are not fully utilized. The autonomous development of surgical robots is the trend of the future. The high precision of its manipulation and the convenience of obtaining feedback information have huge application advantages. Therefore, it is necessary to study the autonomous planning and operation of surgical robots.

[0004] The autonomous planning of surgical robots relies on the learning of prior physicians' operational experience and knowledge transfer technology, which aims to allow robots to learn from expert demonstrations and realize autonomous planning of operation trajectories in different environments by constructing knowledge graphs and quickly generating online instructions. At present, the mainstream robot autonomous planning methods are mainly based on trajectory planning control and medical experience learning. Most of the existing planning and control methods are based on mission-critical waypoints and visual servoing technology, while medical experience learning often uses Gaussian mixture models or Markov decisions. Because the methods are flexible and efficient, they are important methods for autonomous surgical planning operations in recent years. However, the existing methods have poor transfer and generalization effects, and it is difficult to handle the dynamic and complex environment in the body, so further improvement is still needed.

[0005] The existing patent document CN114129263B discloses a surgical robot path planning method, system, device and storage medium, the method includes: obtaining a three-dimensional image model of the object to be operated in the target area, and formulating one or more operation paths based on the preset target points and targets; performing intraoperative registration of the robot; calculating each operation path to determine whether the robot can plan all operation paths, if not, adjusting the object to be operated until all operation paths can be planned, and sending all plannable operation paths to the robot controller. The path planning of this patent relies on a three-dimensional image model, is weak in coping with complex surgical environments and learning expert experience, and has relatively insufficient automation and flexibility.

[0006] Based on this, in order to enable surgical robots to achieve safe, efficient and autonomous online planning in complex dynamic scenarios, it is necessary to study an integrated method of environmental topology representation, task skill learning and online knowledge transfer. Summary of the invention

[0007] In view of the defects in the prior art, an object of the present invention is to provide a surgical robot autonomous planning method suitable for laparoscopic surgery.

[0008] According to the present invention, a surgical robot autonomous planning method applicable to laparoscopic surgery comprises the following steps:

[0009] Step S1, extracting multivariate coupling information of the a priori physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network, realizing feature encoding, and constructing a priori physician operation map;

[0010] Step S2, using a similarity measurement twin network, the low-dimensional features of the current frame extracted online are compared with the low-dimensional codes in the prior physician operation map one by one for similarity traversal, to obtain similarity measurement values, and the trajectory corresponding to the prior coding feature with the highest similarity is extracted;

[0011] Step S3: output the planned trajectory online according to the similarity metric value.

[0012] Preferably, the multivariate coupling information includes the visual spatial morphological features of the surgical field and the kinematic morphological features of the robot, and step S1 includes the following sub-steps:

[0013] Step S1.1, creating visual spatial morphological features of the surgical field, including: using the laparoscope image as the guiding input information, based on the intelligent segmentation and three-dimensional reconstruction algorithm, constructing the spatial morphology of the wound in the surgical field, and completing the feature extraction of multiple frames of high-dimensional images by using a convolutional neural network structure to process single-frame image input and a recursive neural network structure to process time series;

[0014] Step S1.2, creating robot kinematic morphological features, including: using a variational autoencoder neural network architecture and a self-supervised learning mechanism to characterize the robot configuration posture and robot trajectory tensor as a low-dimensional latent space descriptor;

[0015] Step S1.3, fusing visual features with kinematic features, including: using an attention mechanism to fuse the feature vectors obtained in steps S1.1 and S1.2, completing the feature fusion of visual and kinematic heterogeneous information, maximizing the retention of the correlation between scene features and behavior patterns, and realizing multimodal representation and fusion, that is, achieving visual and kinematic fusion in the spatial dimension.

[0016] Preferably, in step S1.1, the visual spatial morphological features of the surgical field are expressed as:

[0017]

[0018] in, represents the input multi-frame image sequence, n is the sequence step, h is the height of a single image, h is the height of a single image, and c is the number of channels of a single image. is the output feature vector of the visual feature extraction network, s represents the output step length, and f img (·) represents a nonlinear visual feature extraction network.

[0019] Preferably, in step S1.2, the robot kinematic morphological characteristics are expressed as:

[0020]

[0021] in, represents the robot configuration posture tensor, p is the degree of freedom of the surgical robot, represents the robot trajectory tensor, is the output feature vector of the kinematic feature extraction network, m is the dimension of the feature vector, and f pose (·) represents the nonlinear kinematic feature extraction network.

[0022] Preferably, in step S1.3, the visual feature is used as the query value, the kinematic morphological feature is used as the key value, and the cosine similarity of the two tensors is calculated:

[0023]

[0024] The similarity is scaled and normalized using the softmax function:

[0025] a i =softmax(Similarity i )

[0026] Finally, the attention value is obtained by weighting according to the weight coefficient to complete the fusion of visual features and kinematic features:

[0027]

[0028] Complete the feature fusion of visual and kinematic heterogeneous information. The input and output of the overall network architecture are recorded as:

[0029] y i =f fusion (x i )

[0030] Among them, f fusion (·) represents the nonlinear feature extraction and feature fusion network, x i Include y i is the overall low-dimensional encoding of the output.

[0031] Preferably, step 2 includes the following sub-steps:

[0032] Step S2.1, training the similarity metric twin neural network to obtain a trained twin neural network;

[0033] Step S2.2, online measuring the low-dimensional features of the current frame and the encoding features of the prior physician operation map to obtain a similarity measurement value, and extracting the trajectory corresponding to the prior encoding feature with the highest similarity;

[0034] When the surgical robot is working online, it observes the current scene image and the robot configuration state in real time, and inputs them into the trained twin neural network for similarity judgment to obtain a similarity measurement value. If the similarity measurement is higher than the preset threshold, the fragment with the most similar posture feature under the "step" node corresponding to the current situation is extracted from the prior physician operation map generated in step S1 according to the size of the similarity measurement value.

[0035] Preferably, step S2.1 comprises the following steps:

[0036] Step S2.1.1, randomly select two groups of time series from the pre-collected data, and use the dynamic time normalization design normalization algorithm, the operation is as follows:

[0037] For two sets of time series x1 and x2 with different step sizes, the step sizes are n1 and n2 respectively. First, calculate the Euclidean distance between them:

[0038] D(i,j)=||x1(i)-x2(j)||, i=1,2,...,n1, j=1,2,...,n2

[0039] Next, the cumulative distance matrix is ​​calculated according to the recursive formula:

[0040] C(i,j)=D(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)}

[0041] Then, start from the lower right corner of the cumulative distance matrix and trace back to the upper left corner to find a path with the minimum cumulative distance. The path is the best alignment path between two time series of different step lengths. The lower right corner element of the cumulative distance matrix C(i, j) is the DTW distance between the two time series. After that, the two time segments are normalized.

[0042] Step S2.1.2, establishing a CNN-MLP twin neural network structure with shared weights, inputting the data normalized in step S2.1.1 into the feature extraction network of step S1 to obtain feature coding, and inputting the feature coding into the twin neural network for decoding;

[0043]

[0044] Where Ti∈SE(3) represents the estimated relative pose with respect to the target point under the current feature encoding, R i is the estimated rotation matrix, t i is the estimated translation vector, f snn (·) represents a CNN-MLP nonlinear twin neural network with shared weights;

[0045] Step S2.1.3, use the pre-collected relative posture of the surgical robot end and the target position as the true value for supervised training, use the root mean square error loss function to calculate the Euclidean distance, guide the network to perform back propagation, and then obtain the ability to judge the similarity of relative postures. The loss function is designed as follows:

[0046]

[0047] Where α is a weight constant greater than 0, is the true rotation matrix, is the true translation vector.

[0048] Preferably, step 3 comprises the following steps:

[0049] Step S3.1, training an expert trajectory decoding network to obtain a trained expert trajectory decoding network;

[0050] Step S3.2, based on the similarity metric, generate local trajectories.

[0051] Local trajectory generation includes the following steps:

[0052] When the surgical robot is running online, the similarity measurement value obtained in step S2.2 is obtained, a minimum matching threshold δ is set, and the similarity measurement value is metrically matched with the prior physician operation map;

[0053] If the similarity metric value is ≥δ, the corresponding trajectory is extracted from the prior physician operation map, and the trajectory is given to the surgical robot as a local trajectory to achieve fast online trajectory planning;

[0054] If the similarity measure value is less than δ, the expert trajectory decoding network trained in step S3.1 takes the current frame as input, obtains the local trajectory online, and then sends the generated local trajectory to the surgical robot for execution, thereby realizing online trajectory planning.

[0055] Preferably, step S3.1 comprises the following steps:

[0056] Step S3.1.1, extract low-dimensional feature codes and their corresponding trajectory information from the prior physician operation map, and align the code features through dynamic time normalization;

[0057] Step S3.1.2, establish a neural network decoder based on the diffusion model. The neural network decoder takes the feedback information as input, constructs a mapping from multivariate coupling information to local trajectories, calculates the probability distribution of the estimated expert trajectory by first adding noise to the low-dimensional feature encoding and then denoising it, and then generates a local trajectory data set by sampling. The expression for expert trajectory generation using the diffusion model is as follows:

[0058] Y 0 =y i

[0059] Y γ =f diffuse (Y γ-1 ),γ=1,2,...,Γ

[0060]

[0061]

[0062] where f diffuse (·) represents the noise adding process, f denoise (·) represents the nonlinear denoising function, Γ represents the total number of denoising steps, and the noise estimation loss function is used to guide the denoising network to back propagate and update the network parameters:

[0063]

[0064] Step S3.1.3, establish an MLP neural network to select the local trajectory with the highest probability likelihood from the sampling results. The local trajectory set output by step S3.1.2 satisfies a probability distribution Yi ~p(Y), i = 1, 2, ..., k, to obtain the optimal sampling results, train the MLP network to derive the probability likelihood corresponding to each sampling output of the diffusion model:

[0065] E i =f sample (Y i ), i = 1, 2, ..., k

[0066] where f sample (·) represents the nonlinear neural network function that derives the optimal probability likelihood, and by selecting the largest E i The corresponding trajectory is taken as the optimal trajectory:

[0067] Y best = argmax i E i ,,i=1,2,......,k

[0068] The present invention also provides a surgical robot autonomous planning system suitable for laparoscopic surgery, comprising:

[0069] Module S1, extracting multivariate coupling information of the a priori physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network, realizing feature encoding, and constructing a priori physician operation map;

[0070] Module S2 uses a similarity measurement twin network to traverse and compare the low-dimensional features of the current frame extracted online with the low-dimensional codes in the prior physician operation map one by one to obtain similarity measurement values ​​and extract the prior coding features with the highest similarity;

[0071] Module S3 outputs the planned trajectory online based on the similarity metric value.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] 1. In the decoding planning task, the present invention seeks the best low-dimensional manifold embedding code for the multivariate coupling information, and realizes the alignment of the time step by dynamically normalizing different scenes. At the same time, the time step information is encoded with the spatial structure information of the planning operation to realize the time-space coding registration, which retains the multivariate information characteristics to the greatest extent and realizes a higher degree of information fusion.

[0074] 2. The present invention can further enhance the autonomy of the surgical robot and endow the surgical robot with the ability to autonomously complete online intraoperative planning. The surgical robot learns the prior physician teaching data and constructs a prior physician operation map for complex surgical planning tasks. The map contains low-dimensional manifold encoding information of scenes and actions and trajectory teaching information to facilitate subsequent trajectory generation. When the surgical robot receives real-time feedback during the operation, it will perform online scene measurement with the prior physician operation map, and then generate local trajectories based on the measurement, thereby realizing the autonomous planning task of the surgical robot and improving the intelligence and flexibility of online planning. Compared with the existing traditional trajectory planning methods, extracting trajectories from the prior physician operation map will greatly improve the efficiency of local trajectory planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0076] Figure 1 This is a flow chart of a method for autonomous planning of a surgical robot applicable to a laparoscopic surgery scenario in an embodiment of the present invention;

[0077] Figure 2 is a schematic diagram of an autonomous planning system for a surgical robot suitable for a laparoscopic surgery scenario in an embodiment of the present invention;

[0078] Figure 3 It is a block diagram of a similarity metric twin neural network training module in an embodiment of the present invention;

[0079] Figure 4 This is a block diagram of an expert trajectory decoding neural network training module in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0081] The present invention provides a method and system for autonomous planning of surgical robots in laparoscopic surgery scenarios, aiming to effectively improve the intelligence and autonomy of surgical robots in in vivo surgical tasks such as suturing, intubation, and ablation. In order to achieve physician skill learning, a low-dimensional feature extraction network is designed to implement low-dimensional manifold embedding of multi-frame video streams, surgical robot configurations and other multi-coupling feedback information in the prior physician operation video, and save the low-dimensional feature encoding and expert demonstration trajectory to construct the prior physician operation map. In order to achieve dynamic generation of surgical robot operation trajectories, a twin network based on posture similarity is designed to extract key features during surgery, and based on this, the optimal instruction strategy is searched and generated in the prior physician operation map. The surgical robot completes the online autonomous planning during surgery according to the optimal control instruction.

[0082] Embodiment 1:

[0083] Figure 1 The present invention is a flowchart of a method for autonomous planning of a surgical robot suitable for laparoscopic surgery scenarios in an embodiment of the present invention.

[0084] like Figure 1 As shown, this embodiment provides a surgical robot autonomous planning method suitable for laparoscopic surgery scenarios, including the following steps:

[0085] Step S1, extracting the multivariate coupling information of the a priori physician operation video stream. Specifically, by designing a feature extraction network and a feature fusion network, the multivariate coupling information such as intraoperative online images, trajectories of past frames, and robot configurations are embedded into a low-dimensional latent space manifold to achieve feature encoding and construct a priori physician operation map.

[0086] More specifically, the multivariate coupling information includes the visual spatial morphological features of the surgical field and the kinematic morphological features of the robot. Step S1 includes the following sub-steps:

[0087] Step S1.1, creating visual spatial morphological features of the surgical field. Specifically, by using the laparoscope image as the guiding input information, based on the intelligent segmentation and three-dimensional reconstruction algorithm, the spatial morphology of the wound in the surgical field is constructed. In this embodiment, the feature extraction of multiple frames of high-dimensional images is completed by using a convolutional neural network structure to process a single-frame image input and a recursive neural network structure to process a time series.

[0088] Specifically, the visual spatial morphological characteristics of the surgical field are expressed as:

[0089]

[0090] in, represents the input multi-frame image sequence, n is the sequence step, h is the height of a single image, h is the height of a single image, and c is the number of channels of a single image. is the output feature vector of the visual feature extraction network, s represents the output step length, and f img (·) represents a nonlinear visual feature extraction network.

[0091] Step S1.2, creating the robot kinematic morphological features. Specifically, with the help of the variational autoencoder neural network architecture and the self-supervised learning mechanism, the robot configuration posture and the robot trajectory tensor are represented as a low-dimensional latent space descriptor.

[0092] Specifically, the robot kinematic morphological characteristics are expressed as:

[0093]

[0094] in, represents the robot configuration posture tensor, p is the degree of freedom of the surgical robot, represents the robot trajectory tensor, is the output feature vector of the kinematic feature extraction network, m is the dimension of the feature vector, and f pose (·) represents the nonlinear kinematic feature extraction network.

[0095] Step S1.3, fusing the visual features with the kinematic features. Specifically, the feature vectors obtained in steps S1.1 and S1.2 are fused using an attention mechanism.

[0096] Specifically, the visual features are used as query values ​​and the kinematic morphological features are used as key values. The cosine similarity of the two tensors is calculated:

[0097]

[0098] The similarity is scaled and normalized using the softmax function:

[0099] a i =softmax(Similarity i )

[0100] Finally, the attention value is obtained by weighting according to the weight coefficient to complete the fusion of visual features and kinematic features:

[0101]

[0102] After this step, the feature fusion of visual and kinematic heterogeneous information is completed, the correlation between scene features and behavior patterns is retained to the maximum extent, and multimodal representation and fusion are achieved, that is, the fusion of vision and kinematics in the spatial dimension is achieved. The input and output of the overall network architecture are recorded as:

[0103] y i =f fusion (xi )

[0104] Among them, f fusion (·) represents the nonlinear feature extraction and feature fusion network, x i Include y i is the overall low-dimensional encoding of the output.

[0105] In step S2, a similarity measurement twin network is used to traverse and compare the low-dimensional features of the current frame obtained by online extraction with the low-dimensional codes in the prior physician operation map one by one to obtain the similarity measurement value, and extract the trajectory corresponding to the prior coding feature with the highest similarity.

[0106] Specifically, step 2 includes the following sub-steps:

[0107] Step S2.1, train the similarity metric twin neural network to obtain the trained twin neural network.

[0108] Figure 3 This is a block diagram of a similarity metric twin neural network training module in an embodiment of the present invention.

[0109] like Figure 3 As shown, step S2.1 includes the following steps:

[0110] Step S2.1.1, randomly select two groups of time series from the pre-collected data, and use dynamic time warping (DTW) to design a normalization algorithm to ensure the temporal consistency of the output of the feature encoding network and reduce semantic divergence. The specific implementation method is as follows:

[0111] For two sets of time series x1 and x2 with different step sizes, the step sizes are n1 and n2 respectively. First, calculate the Euclidean distance between them:

[0112] D(i,j)=||x1(i)-x2(j)||, i=1,2,...,n1, j=1,2,...,n2

[0113] Next, the cumulative distance matrix is ​​calculated according to the recursive formula:

[0114] C(i,j)=D(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)}

[0115] Then, starting from the lower right corner of the cumulative distance matrix, backtrack to the upper left corner to find a path with the minimum cumulative distance. The path is the optimal alignment path between two time series of different step lengths. The lower right corner element of the cumulative distance matrix C(i, j) is the DTW distance between the two time series. After that, the two time segments are normalized.

[0116] Step S2.1.2, establishing a CNN-MLP twin neural network structure with shared weights, inputting the data normalized in step S2.1.1 into the feature extraction network in step S1 to obtain feature coding, and inputting the feature coding into the twin neural network for decoding;

[0117]

[0118] Among them, T i ∈SE(3) represents the estimated relative pose with the target point under the current feature encoding, R i is the estimated rotation matrix, t i is the estimated translation vector, f snn (·) represents a CNN-MLP nonlinear twin neural network with shared weights.

[0119] In step S2.1.3, the relative posture of the surgical robot end and the target position collected in advance is used as the true value for supervised training. The root mean square error (RMSE) loss function is used to calculate the Euclidean distance, guide the network to perform back propagation, and then obtain the ability to judge the similarity of relative postures. The loss function is designed as follows:

[0120]

[0121] Where α is a weight constant greater than 0, is the true rotation matrix, is the true translation vector.

[0122] Step S2.2, online measuring the low-dimensional features of the current frame and the encoding features of the prior physician operation map to obtain a similarity measurement value, and extracting the trajectory corresponding to the prior encoding feature with the highest similarity;

[0123] Specifically, step S2.2 includes the following steps:

[0124] When the surgical robot is working online, it observes the current scene image and the robot configuration state in real time, and inputs them into the trained twin neural network for similarity judgment to obtain a similarity measure value. If the similarity measure is higher than the preset threshold, the fragment with the most similar posture feature under the "step" node corresponding to the current situation is extracted from the prior physician operation map generated in step S1 according to the size of the similarity measure value. The "step" node is a different stage or action node in the prior physician operation process. The extracted fragment provides a reference basis for the output trajectory of the subsequent S3 step.

[0125] Step S3: output the planned trajectory online according to the similarity metric value.

[0126] Specifically, step 3 includes the following steps:

[0127] Step S3.1, training an expert trajectory decoding network to obtain a trained expert trajectory decoding network.

[0128] Figure 4 This is a block diagram of an expert trajectory decoding neural network training module in an embodiment of the present invention.

[0129] like Figure 4 As shown, step S3.1 includes the following steps:

[0130] Step S3.1.1, extract low-dimensional feature codes and their corresponding trajectory information from the prior physician operation map, and normalize and align the code features through dynamic time warping (DTW);

[0131] Step S3.1.2, establish a neural network decoder based on the diffusion model. The neural network decoder takes the surgical field image and robot configuration and other feedback information as input, constructs a mapping from multivariate coupling information to local trajectories, and calculates the probability distribution of the estimated expert trajectory by first adding noise to the low-dimensional feature code and then denoising it. Then, a local trajectory data set is generated by sampling. The expression for expert trajectory generation using the diffusion model is as follows:

[0132] Y 0 =y i

[0133] Y γ =f diffuse (Y γ-1 ),γ=1,2,...,Γ

[0134]

[0135]

[0136] where f diffuse (·) represents the noise adding process, f denoise (·) represents the nonlinear denoising function, Γ represents the total number of denoising steps, and the noise estimation loss function is used to guide the denoising network to back propagate and update the network parameters:

[0137]

[0138] Step S3.1.3, establish an MLP neural network to select the local trajectory with the highest probability likelihood from the sampling results. The local trajectory set output by step S3.1.2 satisfies a probability distribution Y i ~p(Y), i = 1, 2, ..., k, to obtain the optimal sampling results, train the MLP network to derive the probability likelihood corresponding to each sampling output of the diffusion model:

[0139] E i =fsample (Y i ), i = 1, 2, ..., k

[0140] where f sample (·) represents the nonlinear neural network function that derives the optimal probability likelihood, and by selecting the largest E i The corresponding trajectory is taken as the optimal trajectory:

[0141] Y best = argmax i E i ,,i=1,2,......,k

[0142] Step S3.2, based on the similarity metric value, local trajectory generation is performed.

[0143] Specifically, local trajectory generation includes the following steps:

[0144] When the surgical robot is running online, the similarity measurement value obtained in step S2.2 is obtained, the minimum matching threshold δ is set, and the similarity measurement value is metrically matched with the prior physician operation map.

[0145] If the similarity metric value is ≥δ, it means that there are features in the prior physician's atlas that match the current frame. Then the corresponding trajectory is extracted from the prior physician's operation atlas, and the trajectory is given to the surgical robot as a local trajectory to achieve fast online trajectory planning.

[0146] If the similarity measure value is less than δ, it means that there is no feature matching the current frame in the prior physician atlas, and the trajectory cannot be directly extracted. Then, the expert trajectory decoding network trained in step S3.1 is used to take the current frame as input, obtain the local trajectory online, and then give the generated local trajectory to the surgical robot for execution to achieve online trajectory planning.

[0147] Embodiment 2:

[0148] The present invention also provides a surgical robot autonomous planning system suitable for laparoscopic surgery scenarios. The surgical robot autonomous planning system suitable for laparoscopic surgery scenarios can be implemented by executing the process steps of a surgical robot autonomous planning method suitable for laparoscopic surgery scenarios, that is, technical personnel in this field can understand the surgical robot autonomous planning method suitable for laparoscopic surgery scenarios as a preferred implementation of the surgical robot autonomous planning system suitable for laparoscopic surgery scenarios.

[0149] Figure 2 Schematic diagram of an autonomous planning system for a surgical robot suitable for laparoscopic surgery scenarios in an embodiment of the present invention.

[0150] like Figure 2As shown, this embodiment provides a surgical robot autonomous planning system suitable for laparoscopic surgery scenarios, including:

[0151] Module S1 extracts the multivariate coupling information of the prior physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network, realizing feature encoding, and constructing a prior physician operation map.

[0152] Module S2 uses a similarity metric twin network to traverse and compare the low-dimensional features of the current frame obtained by online extraction with the low-dimensional codes in the prior physician operation map one by one to obtain the similarity metric value and extract the prior coding features with the highest similarity.

[0153] Module S3 outputs the planned trajectory online based on the similarity metric value.

[0154] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, the system and its various devices, modules, and units provided by the present invention can be completely realized in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps to realize the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0155] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A surgical robot autonomous planning method suitable for laparoscopic surgery, characterized in that: The steps include: Step S1, extracting multivariate coupling information of a priori physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network, realizing feature encoding, and constructing a priori physician operation map; Step S2, using a similarity measurement twin network, traversing and comparing the low-dimensional features of the current frame obtained by online extraction with the low-dimensional codes in the prior physician operation map one by one to obtain a similarity measurement value, and extracting the trajectory corresponding to the prior coding feature with the highest similarity; Step S3: outputting the planned trajectory online according to the similarity metric value.

2. A surgical robot autonomous planning method suitable for laparoscopic surgery according to claim 1, characterized in that: The multivariate coupling information includes the visual spatial morphological features of the surgical field and the kinematic morphological features of the robot. Step S1 includes the following sub-steps: Step S1.1, creating visual spatial morphological features of the surgical field, including: using the laparoscope image as the guiding input information, based on the intelligent segmentation and three-dimensional reconstruction algorithm, constructing the spatial morphology of the wound in the surgical field, and completing the feature extraction of multiple frames of high-dimensional images by using a convolutional neural network structure to process single-frame image input and a recursive neural network structure to process time series; Step S1.2, creating robot kinematic morphological features, including: using a variational autoencoder neural network architecture and a self-supervised learning mechanism to characterize the robot configuration posture and robot trajectory tensor as a low-dimensional latent space descriptor; Step S1.3, fusing visual features with kinematic features, including: using an attention mechanism to fuse the feature vectors obtained in steps S1.1 and S1.2, completing the feature fusion of visual and kinematic heterogeneous information, maximizing the retention of the correlation between scene features and behavior patterns, and realizing multimodal representation and fusion, that is, achieving visual and kinematic fusion in the spatial dimension.

3. According to the method for autonomous planning of a surgical robot suitable for laparoscopic surgery as described in claim 2, it is characterized in that: In step S1.1, the visual spatial morphological features of the surgical field are expressed as: in, represents the input multi-frame image sequence, n is the sequence step, h is the height of a single image, h is the height of a single image, and c is the number of channels of a single image. is the output feature vector of the visual feature extraction network, s represents the output step length, and f img (·) represents a nonlinear visual feature extraction network.

4. The method for autonomous planning of a surgical robot suitable for laparoscopic surgery according to claim 2, characterized in that: In step S1.2, the robot kinematic morphological characteristics are expressed as: in, represents the robot configuration posture tensor, p is the degree of freedom of the surgical robot, represents the robot trajectory tensor, is the output feature vector of the kinematic feature extraction network, m is the dimension of the feature vector, and f pose (·) represents the nonlinear kinematic feature extraction network.

5. The method for autonomous planning of a surgical robot suitable for laparoscopic surgery according to claim 2, characterized in that: In step S1.3, the visual feature is used as the query value, the kinematic morphological feature is used as the key value, and the cosine similarity of the two tensors is calculated: The similarity is scaled and normalized using the softmax function: a i =softmax(Similarity i ) Finally, the attention value is obtained by weighting according to the weight coefficient to complete the fusion of visual features and kinematic features: Complete the feature fusion of visual and kinematic heterogeneous information. The input and output of the overall network architecture are recorded as: y i =f fusion (x i ) Among them, f fusion (·) represents the nonlinear feature extraction and feature fusion network, x i Include y i is the overall low-dimensional encoding of the output.

6. The method for autonomous planning of a surgical robot suitable for laparoscopic surgery according to claim 1, characterized in that: The step 2 includes the following sub-steps: Step S2.1, training the similarity metric twin neural network to obtain a trained twin neural network; Step S2.2, online measuring the low-dimensional features of the current frame and the encoding features of the prior physician operation map to obtain a similarity measurement value, and extracting the trajectory corresponding to the prior encoding feature with the highest similarity; When the surgical robot is operating online, the current scene image and the robot configuration state are observed in real time, and the images are input into the trained twin neural network for similarity judgment to obtain a similarity measurement value. If the similarity measurement is higher than a preset threshold, the fragment with the most similar posture feature under the "step" node corresponding to the current situation is extracted from the prior physician operation map generated in step S1 according to the size of the similarity measurement value.

7. A surgical robot autonomous planning method suitable for laparoscopic surgery according to claim 6, characterized in that: The step S2.1 comprises the following steps: Step S2.1.1, randomly select two groups of time series from the pre-collected data, and use the dynamic time normalization design normalization algorithm, the operation is as follows: For two sets of time series x1 and x2 with different step sizes, the step sizes are n1 and n2 respectively. First, calculate the Euclidean distance between them: D(i,j)=||x1(i)-x2(j)||,i=1,2,…,n1,j=1,2,…,n2 Next, the cumulative distance matrix is ​​calculated according to the recursive formula: C(i,j)=D(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)} Then, start from the lower right corner of the cumulative distance matrix and trace back to the upper left corner to find a path with the minimum cumulative distance. The path is the optimal alignment path between two time series of different step lengths. The lower right corner element of the cumulative distance matrix C(i, j) is the DTW distance between the two time series. After that, the two time segments are normalized. Step S2.1.2, establishing a CNN-MLP twin neural network structure with shared weights, inputting the data normalized in step S2.1.1 into the feature extraction network of step S1 to obtain feature coding, and inputting the feature coding into the twin neural network for decoding; Among them, T i ∈SE(3) represents the estimated relative pose with the target point under the current feature encoding, R i is the estimated rotation matrix, t i is the estimated translation vector, f snn (·) represents a CNN-MLP nonlinear twin neural network with shared weights; Step S2.1.3, use the pre-collected relative posture of the surgical robot end and the target position as the true value for supervised training, use the root mean square error loss function to calculate the Euclidean distance, guide the network to perform back propagation, and then obtain the ability to judge the similarity of relative postures. The loss function is designed as follows: Where α is a weight constant greater than 0, is the true rotation matrix, is the true translation vector.

8. The method for autonomous planning of a surgical robot suitable for laparoscopic surgery according to claim 1, characterized in that: The step 3 comprises the following steps: Step S3.1, training an expert trajectory decoding network to obtain a trained expert trajectory decoding network; Step S3.2, based on the similarity metric, generate local trajectories. Local trajectory generation includes the following steps: When the surgical robot is running online, the similarity measurement value obtained in step S2.2 is obtained, a minimum matching threshold δ is set, and the similarity measurement value is metrically matched with the prior physician operation map; If the similarity metric value is ≥δ, the corresponding trajectory is extracted from the prior physician operation map, and the trajectory is given to the surgical robot as a local trajectory to perform, so as to achieve fast online trajectory planning; If the similarity measure value <δ, the expert trajectory decoding network trained in step S3.1 takes the current frame as input, obtains the local trajectory online, and then sends the generated local trajectory to the surgical robot for execution, thereby realizing online trajectory planning.

9. The method for autonomous planning of a surgical robot suitable for laparoscopic surgery according to claim 1, characterized in that: The step S3.1 comprises the following steps: Step S3.1.1, extracting low-dimensional feature codes and their corresponding trajectory information from the prior physician operation map, and aligning the code features through dynamic time normalization; Step S3.1.2, establish a neural network decoder based on the diffusion model. The neural network decoder takes the feedback information as input, constructs a mapping from multivariate coupling information to local trajectories, calculates the probability distribution of the estimated expert trajectory by first adding noise to the low-dimensional feature encoding and then denoising it, and then generates a local trajectory data set by sampling. The expression for expert trajectory generation by the diffusion model is as follows: AND 0 =and i Y γ =f diffuse (Y γ-1 ),γ=1,2,……,Γ where f diffuse (·) represents the noise adding process, f denoise (·) represents the nonlinear denoising function, Γ represents the total number of denoising steps, and the noise estimation loss function is used to guide the denoising network to back propagate and update the network parameters: Step S3.1.3, establish an MLP neural network to select the local trajectory with the highest probability likelihood from the sampling results. The local trajectory set output by step S3.1.2 satisfies a probability distribution Y i ~p(Y),i=1,2,……,k, in order to obtain the optimal sampling results, the MLP network is trained to derive the probability likelihood corresponding to each sampling output of the diffusion model: E i =f sample (Y i ),i=1,2,……,k where f sample (·) represents the nonlinear neural network function that derives the optimal probability likelihood, and by selecting the largest E i The corresponding trajectory is taken as the optimal trajectory: Y best =argmax i E i ,,i=1,2,……,k 10. A surgical robot autonomous planning system suitable for laparoscopic surgery, characterized in that: include: Module S1, extracting multivariate coupling information of a priori physician operation video stream, including: embedding the multivariate coupling information into a low-dimensional latent space manifold by designing a feature extraction network and a feature fusion network, realizing feature encoding, and constructing a priori physician operation map; Module S2 uses a similarity measurement twin network to perform a similarity traversal comparison between the low-dimensional features of the current frame obtained by online extraction and the low-dimensional codes in the prior physician operation map one by one, obtain a similarity measurement value, and extract the prior coding features with the highest similarity; Module S3 outputs the planned trajectory online according to the similarity metric value.

Images