Improved RRT path planning method for robot-assisted puncture based on SAC guided sampling

Abstract

The invention discloses an improved RRT path planning method for robot-assisted puncture based on SAC guided sampling, and the method comprises the steps: carrying out the reconstruction of a human chest and abdomen actual physiological structure three-dimensional model based on a CT image, and obtaining the human chest and abdomen actual physiological structure three-dimensional model as an obstacle for subsequent path planning; establishing a reinforcement learning environment based on the obstacle, including setting a state space, an action space and a reward function; in a reinforcement learning environment, based on obstacles, adopting an SAC algorithm to carry out SAC guide sampling model training of flexible needle path planning; adjusting and optimizing target parameters of the SAC guide sampling model based on obstacles; and sampling in the improved RRT path planning based on a mixed sampling strategy by using the parameter-optimized SAC guide sampling model to obtain a feasible path. Through a mixed sampling strategy, balance is obtained between the global search capability of uniform sampling and the local optimization capability of non-uniform guide sampling, so that the adaptability and planning efficiency of the algorithm in a complex scene are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning of robot-assisted puncture, and particularly relates to an improved RRT (Rapidly-exploring Random Tree) path planning method for robot-assisted puncture based on SAC (Soft Actor-Critic) guided sampling. BACKGROUND

[0002] Due to high diagnostic specificity, small tissue damage and low postoperative complications, image-guided percutaneous puncture surgery has been widely used in many fields of modern clinical intervention, such as tumor ablation, tissue biopsy, targeted drug injection and the like. In the surgical process, the puncture needle needs to avoid important organs, blood vessels, nerves and other tissues in the human physiological structure to avoid tissue damage and accurately reach the target position for biopsy or treatment. The traditional rigid puncture needle can only puncture along a straight line, which lacks sufficient flexibility and is difficult to completely avoid obstacles in the human physiological structure. The flexible needle with good toughness and easy bending can be bent to avoid important organs, blood vessels and nerves and other tissues and achieve curve puncture due to the imbalance of the tissue force received by the asymmetric oblique angle tip in the puncture process.

[0003] The flexible needle produces a complex trajectory in the human body, and it is very difficult to complete the puncture completely relying on the doctor. Through robot-assisted puncture operation of the doctor, the flexible needle puncture path is accurately planned to effectively avoid important structures in the human body, such as bones, nerves, blood vessels and trachea, which can significantly reduce the damage to the surrounding tissues during the surgical process, reduce the risk of postoperative complications, and further improve the safety and controllability of the puncture surgery.

[0004] Flexible needle path planning is mainly to find a feasible path under the condition of meeting various constraints. The search method can quickly obtain a feasible path through random sampling and space search. Due to its fast search speed and high search efficiency, the rapid exploring random tree (RRT) algorithm is widely used in flexible needle path planning. However, due to its randomness, the path obtained may not be optimal and has a large randomness. The RRT* algorithm can generate a smoother optimized path through optimization processes such as parent node reselection and node reconnection. However, due to the significant increase in search calculation, its planning efficiency is low. The bidirectional RRT algorithm generates search trees from the starting point and the target point at the same time until the two trees are connected, greatly reducing the search time and increasing the convergence speed. However, bidirectional search may generate redundant nodes and generate a more tortuous path. The complex tortuous path causes more damage to the patient and increases the difficulty of controlling the flexible needle. The RRT path planning algorithm based on sampling faces the challenges of low calculation efficiency or node redundancy and path tortuosity. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies and shortcomings of the prior art, and to provide an improved RRT path planning method for robot-assisted puncture based on SAC guided sampling.

[0006] An improved RRT path planning method for robot-assisted puncture based on SAC guided sampling, comprising:

[0007] S1. Reconstructing a three-dimensional model of the actual physiological structure of the human chest and abdomen based on CT images to obtain a three-dimensional model of the actual physiological structure of the human chest and abdomen as an obstacle for subsequent path planning;

[0008] S2. Establishing a reinforcement learning environment based on the obstacle, including setting the state space, action space, and reward function;

[0009] S3. In the reinforcement learning environment, training the SAC guided sampling model for flexible needle path planning based on the obstacle using the SAC algorithm;

[0010] S4. Optimizing the target parameters of the SAC guided sampling model based on the obstacle;

[0011] S5. Using the SAC guided sampling model with optimized parameters to sample in the improved RRT path planning based on the hybrid sampling strategy to obtain a feasible path.

[0012] In step S1, the CT image reconstructs a three-dimensional model of the actual physiological structure of the human chest and abdomen, which includes:

[0013] Adopting adaptive median filtering algorithm, in the whole CT image with sliding cubic window way point by point execution, remove the noise in CT image, while preserving the microstructure;

[0014] Adopting morphological gradient method for noise removal after CT image preprocessing, enhance the edge and detail features in CT image;

[0015] The edge and detail feature enhancement after segmentation of CT image using adaptive seed point growing algorithm for bone, trachea and bronchus, and blood vessels and other tissue segmentation;

[0016] The segmented image using Flying-Edges three-dimensional reconstruction algorithm for three-dimensional modeling of each tissue, get the actual physiological structure of human chest and abdomen three-dimensional model containing bone, trachea and bronchus, and blood vessels and other important organs and tissues.

[0017] Among them, the adaptive median filtering algorithm is used, in the whole CT image with sliding cubic window way point by point execution, remove the noise in CT image, including:

[0018] Define the characteristic value H1, H2, H3, H4;

[0019]

[0020] Among them, H xyz is the image center gray of the current filtering cubic window C xyz , H med , H max , H min is the image median gray, maximum gray and minimum gray of the current filtering cubic window C xyz ;

[0021] Set the maximum volume of the filtering cubic window to V max , define the size of the initial filtering cubic window, calculate the characteristic value, and make the following judgments:

[0022] If H1<0, H2>0 and H3<0, H4>0, it is considered that the current image does not belong to noise point, the center gray value is kept unchanged; if H1<0, H2>0 and H3≥0 or H4≤0, the median gray value is used instead of the center gray value; if H1≥0 or H2≤0, increase the size of the filtering cubic window, repeat the calculation of the characteristic value, and make the judgment, until the judgment is completed or the maximum volume V max of the filtering cubic window is reached.

[0023] Among them, in the preprocessing of CT image after noise removal by morphological gradient method, the expression of morphological gradient is as follows:

[0024]

[0025] where V is a local stereoscopic sampling window, i,j,k represent the change in three directions within the sampling window, D(x,y,z) is the dilation of the image, and E(x,y,z) is the erosion of the image.

[0026] In step S2, the state space includes a current node position, a direction vector between the current node and a target point, a distance between the current node and the target point, a distance between the current node and a starting point, and a minimum distance between the current node and an obstacle; the action space is a set of unit direction vectors in a conical space with the current needle tip direction as the axis, with an included angle not exceeding a maximum deflection angle; the reward function includes an approaching target position reward function, a negative reward function for approaching an obstacle or colliding with the obstacle, a great reward function for reaching the target position, and a path curvature constraint reward function;

[0027] The state space is

[0028] where p node =(x n ,y n ,z n ) is the current node position, is a direction vector between the current node and a target point p goal =(x g ,y g ,z g ), d goal =||p goal -p node || is a distance between the current node and the target point, d start =||p node -p start || is a distance between the current node and a starting point, is a minimum distance between the current node and a jth obstacle , and M is the number of obstacles;

[0029] The action space is After the direction vector of the node sampling is determined, the next sampling node position is p sample =p node +a t l depth ;

[0030] where a t is a direction vector of the next sampling direction, is a current needle tip direction vector, β max represents a maximum deflection angle, l depth ∈[0,l max ] is a node sampling depth, and lmax for the maximum node sampling depth;

[0031] The reward function is R = ω1R near + ω2R collision + ω3R arrive + ω4R constraint ;

[0032] The reward function for approaching the target position is R near = r1(||p i-1 -p goal || - ||p i -p goal ||);

[0033] The negative reward function is

[0034] The huge reward function is

[0035] The path curvature constraint reward function is

[0036] In the formula, ||p i-1 -p goal || is the distance from the i-1th node to the target point, ||p i -p goal || is the distance from the i th node to the target point, d safe is the threshold distance of the node from the obstacle, ε is the threshold distance of the node from the target, k t is the current path curvature, k max is the maximum curvature of the flexible needle bending, r1, r2, r3, r4 are the normalization coefficients of each reward function respectively, ω1, ω2, ω3, ω4 are the weight coefficients of each reward function respectively.

[0037] In step S3, the SAC guided sampling model for flexible needle path planning based on obstacles is trained in the reinforcement learning environment, including:

[0038] S21. Training parameter initialization; initializing the flexible needle path planning environment; the flexible needle path planning environment is a three-dimensional space, including the starting point, the target point and the obstacle space distribution of the flexible needle path planning, and the obstacles are discrete as a spatial point cloud;

[0039] S22. Reset the training environment; reset the current sampling node to the starting point of the path planning;

[0040] S23. Collect the state information s i of the current node, the SAC policy network generates an action a i , and the state is updated according to the action a i to obtain the updated state si+1 , immediate reward r i , and a flag d i ∈{0,1};

[0041] S24. Determine whether the current sampling point collides with the obstacle. If yes, return to step S23 to regenerate the sampling point. If no, store the current collected information (s i ,a i ,r i ,s i+1 ,d i ) in the experience replay buffer D.

[0042] S25. Determine whether there is certain data. If yes, obtain the data from the experience replay buffer D, calculate the target Q value using the target Q network, optimize the SAC policy network and the current Q network parameters by gradient descent minimization of the loss function, perform soft update of the target Q network, and proceed to the next step. If no, proceed to the next step.

[0043] S26. Determine whether the current sampling point exceeds the maximum step limit. If yes, determine that the current path sampling task is completed, and proceed to the next step. If no, determine whether the current sampling point reaches the target area. If yes, determine that the current path sampling task is completed, and proceed to the next step. If no, collect the state information of the next node, and return to step S23.

[0044] S27. Determine whether the training process reaches the maximum training round. If yes, end the training of the SAC policy network, and obtain the SAC guided sampling model. If no, proceed to the next training round, return to step S22, reset the training environment, and start the sampling again to continue the training.

[0045] wherein the target Q value calculation formula is as follows:

[0046]

[0047] wherein y t is the target Q value, a i+1 is the next action output by the policy network, r t represents the immediate reward, d t represents the indication variable of whether it is in the termination state, and γ is the discount factor, is the target double Q network, and α is the weight coefficient of the policy entropy, and logπ φ (a t+1 |s t+1 ) is the policy entropy.

[0048] The gradient descent minimization loss function is

[0049] wherein N is the batch size, and Qθ is the current Q network.

[0050] wherein the soft update of the target Q network is performed by using the following formula.

[0051] θ target ← τθ + (1-τ)θ target ;

[0052] wherein θ is a parameter of the current Q network, θ target is a parameter of the target Q network, and τ is a soft update coefficient.

[0053] In step S4, the target parameters include a learning rate, a discount factor, and a batch size; the training of the SAC guided sampling model of the SAC algorithm is performed by using different combinations of target parameters, and the reward curves of the models trained under different target parameters are compared to obtain the guided sampling training model under the optimized target parameters.

[0054] In step S5, the SAC guided sampling model after parameter tuning is used to sample in the improved RRT path planning based on the hybrid sampling strategy to obtain a feasible path, including:

[0055] initializing path planning parameters to determine a starting point, a target point, and obstacles;

[0056] determining whether the starting point and the target point are within the obstacles, if not, adding the starting point to the RRT search tree, and then entering the node sampling stage, if yes, ending;

[0057] in the node sampling stage, sampling based on the hybrid sampling strategy to obtain a sampling node; and then entering the path constraint stage;

[0058] in the path constraint stage, determining whether the sampling node is within the obstacles; if not, performing obstacle collision detection to determine whether the sampling node collides with the obstacles, if not, reconnecting the node and then performing flexible needle kinematics constraint judgment; if the flexible needle kinematics constraint is met, entering the path acquisition stage, adding the sampling node to the RRT search tree, and then determining whether the target point is reached, if the target point is reached, backtracking the RRT search tree to find a path, obtaining a feasible path, and then ending;

[0059] wherein if the sampling node is determined to be within the obstacles, the sampling node is determined to collide with the obstacles, the sampling node is determined not to meet the flexible needle kinematics constraint, or the target point is not reached, returning to the node sampling stage for node sampling;

[0060] the hybrid sampling strategy uses the SAC guided sampling model to generate a sampling point for guided sampling in a first proportion, and randomly generates a sampling point for random sampling in a second proportion; the first proportion and the second proportion add up to 100%.

[0061] The improved RRT path planning method of robot-assisted puncture based on SAC guided sampling of the application can be called SAC guided sampling RRT method (SAC-RRT method), proposes a guided sampling strategy based on SAC reinforcement learning algorithm training, can effectively guide the search direction in the sampling process, reduce the blindness of the search process, reduce the generation of invalid nodes, and improve the planning efficiency. At the same time, a hybrid sampling strategy is proposed, a certain proportion of SAC strategy guided sampling is used to accelerate convergence, and the rest is used for traditional uniform sampling in the whole space to ensure that potential feasible paths are not ignored. Through the hybrid sampling strategy, a balance is achieved between the global search ability of uniform sampling and the local optimization ability of non-uniform guided sampling, thereby improving the adaptability and planning efficiency of the algorithm in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The flowchart of the improved RRT path planning method of robot-assisted puncture based on SAC guided sampling of the embodiment of the application.

[0063] Figures 2A-2C The schematic diagrams of the chest and abdominal bones, trachea and bronchus, and lung vessels formed after the segmentation of the pretreated CT image of the embodiment of the application.

[0064] Figures 2D-2F The front view, side view and top view of the three-dimensional model formed by reconstructing the actual physiological structure of the human chest and abdomen based on the CT image of the embodiment of the application.

[0065] Figure 3 The training flowchart of the SAC guided sampling model training for flexible needle path planning using the SAC algorithm of the embodiment of the application.

[0066] Figure 4 The flowchart of the improved RRT path planning based on SAC guided sampling of the embodiment of the application. DETAILED DESCRIPTION

[0067] The application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0068] The embodiment of the application trains a SAC guided sampling model in a reinforcement learning space based on a SAC algorithm, executes a guided sampling strategy using the trained SAC guided sampling model, reduces the blindness of the search process in the search-based path planning algorithm, effectively guides the search direction in the sampling process, reduces the generation of invalid nodes, and improves the path planning efficiency.

[0069] The embodiment of the application accelerates the convergence by using a certain proportion of SAC strategy guided sampling to improve the quality and success rate of the planned path, and the rest uses traditional uniform sampling in the whole space to ensure that potential feasible paths are not ignored. A balance is achieved between the global search ability of uniform sampling and the local optimization ability of guided sampling of non-uniform sampling, improving the adaptability and planning efficiency of the algorithm in complex scenes.

[0070] Referring to Figure 1 In the exemplary embodiment of the application, the improved RRT path planning method for robot-assisted puncture based on SAC guided sampling (referred to as the method) comprises the following steps:

[0071] Step 1: Reconstruct a three-dimensional model of the actual physiological structure of the human chest and abdomen based on a CT image to obtain a three-dimensional model of the actual physiological structure of the human chest and abdomen as an obstacle for subsequent path planning.

[0072] Specifically, the preoperative CT (Computed Tomography, computed tomography) image of the patient can be collected, and then the adaptive median filtering and morphological gradient are used to preprocess the CT image to obtain a CT image with clear tissue contrast and enhanced edge features; the preprocessed CT image is segmented into chest and abdominal bones, trachea and bronchus, and lung blood vessels using an adaptive seed point growing algorithm, as shown in Figures 2A-2C The segmented image is used to perform three-dimensional modeling of each tissue using a Flying-Edges three-dimensional reconstruction algorithm to obtain a three-dimensional model of the actual physiological structure of the human chest and abdomen containing important organs and tissues such as chest and abdominal bones, trachea and bronchus, and lung blood vessels, as an obstacle for subsequent path planning, as shown in Figures 2D-2F

[0073] In the embodiment of the application, in step 1, an adaptive median filtering algorithm is used to remove noise in the CT image while preserving small structures, including:

[0074] The characteristic values H1, H2, H3, and H4 are defined as shown in formula (1).

[0075]

[0076] H xyz is the image center gray value of the current filtering cubic window C xyz , H med , H max , and H min are the image median gray value, maximum gray value, and minimum gray value of the current filtering cubic window C xyz , respectively.

[0077] ​Set the maximum volume of the filter cube window as V max ; define the initial filter cube window size, calculate the eigenvalue, and make the following judgments:

[0078] If H1<0, H2>0, and H3<0, H4>0, it is considered that the current image does not belong to a noise point, the central gray value is kept unchanged, and the judgment is completed. If H1<0, H2>0, and H3>0 or H4<0, the median gray value is used to replace the central gray value, and the judgment is completed. If H1>0 or H2<0, the size of the filter cube window is increased, and the eigenvalue is calculated repeatedly, the judgment is made, and the judgment is completed or the maximum volume of the filter cube window is reached. Through the above calculation and judgment process, the adaptive median filtering processing of the CT image is completed.

[0079] In the embodiment of the application, in step 1, after the CT image adaptive median filtering is completed, the morphological gradient G(x, y, z) is used to continue the image preprocessing, which is used to enhance the edge and detail features in the image.

[0080] The morphological gradient G(x, y, z) is shown in formula (2).

[0081]

[0082] Wherein, V is a local three-dimensional sampling window, i, j, k represent the change in three directions within the sampling window. D(x, y, z) is the expansion of the image, and E(x, y, z) is the corrosion of the image.

[0083] In the embodiment of the application, in step 1, the three-dimensional model of the human thoracic and abdominal anatomical structure containing thoracic and abdominal bones, trachea and bronchus, and lung blood vessels reconstructed from the CT image has a size of 534mm×295mm×397mm, as shown in FIG. 2.

[0084] Step 2, establish a reinforcement learning environment based on obstacles, including setting a state space, an action space, and a reward function.

[0085] The state space includes the current node position, the direction vector between the current node and the target point, the distance between the current node and the target point, the distance between the current node and the starting point, and the minimum distance between the current node and the obstacle. The action space is a set of unit direction vectors in a conical space with the current needle tip direction as the axis, and the included angle is not more than the maximum deflection angle. The reward function includes an approach target position reward function, a negative reward function for approaching obstacles or colliding with obstacles, a huge reward function for reaching the target position, and a path curvature constraint reward function.

[0086] In the embodiment of the application, in step 2, a total of 5 state variables are used to represent the state space S, as shown in formula 3.

[0087]

[0088] where p node = (x n , y n , z n ) is the current node position, is the direction vector from the current node to the target point p goal = (x g , y g , z g ), d goal = ||p goal -p node || is the distance between the current node and the target point, d start = ||p node -p start || is the distance between the current node and the start point, is the minimum distance between the current node and the jth obstacle .

[0089] In the embodiments of the present application, in step 2, the action space A is defined as a set of unit direction vectors in a conical space with the current needle tip direction as the axis, and the included angle is not more than the maximum deflection angle β max , as shown in equation (4). After the direction vector of the node sampling is determined, the position p sample of the next sampling node can be represented as equation (5).

[0090]

[0091] p sample = p node + a t l depth ; (5)

[0092] where a t is the direction vector of the next sampling direction, is the current needle tip direction vector, l depth ∈ [0, l max ] is the node sampling depth, and l max is the maximum node sampling depth.

[0093] In the embodiments of the present application, in step 2, the reward function is set as equation (6).

[0094] R = ω1R near + ω2R collision + ω3R arrive + ω4R constraint (6)

[0095] Among them, R near =r1(||p i-1 -p goal ||-||p i -p goal ||) is the reward function for approaching the target position, ||p i-1 -p goal || is the distance from the (i-1)th node to the target point, ||p i -p goal || represents the distance from the i-th node to the target point. If the node is close to the target location, this value is positive.

[0096] For the negative reward function of approaching or colliding with an obstacle, d safe This represents the threshold distance to the obstacle.

[0097] Let ε be the reward function for reaching the target location, and ε be the threshold distance to the target.

[0098] Let k be the reward function for path curvature constraints. t k represents the curvature of the current path. max R1 represents the maximum curvature of the flexible needle. R1, R2, R3, and R4 are the normalized coefficients of each reward function. R1, R2, R3, and R4 are the weighting coefficients of each reward function.

[0099] Step 3: In a reinforcement learning environment, train the SAC-guided sampling model for flexible needle path planning using the SAC algorithm. The specific training process is as follows: Figure 3 As shown. First, the training parameters are initialized, the flexible needle path planning environment is initialized, the training environment is reset to its initial state, and the state information s of the current node is collected. i SAC policy network generates action a i Then, based on action a i Update the state to get the new state s i+1 Instant rewards i And the flag d indicating whether it is terminated. i ∈{0,1}, determine whether the current sampling point collides with an obstacle. If yes, regenerate the sampling point; otherwise, store the currently collected information (s) i ,a i ,r i ,s i+1 ,d i) into the experience replay buffer D; then a batch of data is taken out from the experience replay buffer, the target Q value is calculated, the policy network and the target Q network parameters are optimized by gradient descent to minimize the loss function; then a soft update of the target Q network is performed to smooth the update of the target Q network and ensure stable training of the algorithm. If the current sampling point exceeds the maximum step limit or reaches the target region, the current round of path sampling task is determined to be completed, the environment is reset, and sampling is restarted. The above training process is repeated until the maximum training round - is reached, the training of the SAC policy network is completed, and the SAC guided sampling model based on the SAC policy network is obtained.

[0100] In the embodiment of the present application, in step 3, the flexible needle path planning environment is initialized as a three-dimensional space, including a starting point, a target point and obstacle space distribution of the flexible needle path planning, wherein the obstacle is part of the three-dimensional model of the human chest and abdominal anatomical structure obtained in step 1, wherein the obstacle space size is 260mmx320mmx170mm, and the corresponding obstacle three-dimensional model is discretized into a spatial point cloud with a discretization interval of 1mm. The number of obstacle point clouds after discretization is 291940.

[0101] In the embodiment of the present application, in step 3, the sampling environment is reset to the initial state, and the current sampling node is reset to the starting point of path planning.

[0102] In the embodiment of the present application, in step 3, the state information s u of the current node is collected, and the state information s i of the current node is calculated according to the five state variables defined in formula (3), including the current node position, the direction vector between the current node and the target point, the distance between the current node and the target point, the distance between the current node and the starting point, and the minimum distance between the current node and the obstacle.

[0103] wherein the immediate reward r i is calculated according to the reward function in formula (6), and the normalization coefficients r1, r2, r3 and r4 of each reward function are set to 0.1, 1, 5 and 1 respectively, and the weight coefficients ω1, ω2, ω3 and ω4 of each reward function are set to 0.2, 0.3, 0.4 and 0.1 respectively.

[0104] wherein the threshold distance d safe to the obstacle is set to 2mm, the threshold distance ε to the target is set to 0.1mm, and the maximum curvature k max of the flexible needle bending is set to 0.005.

[0105] In the embodiment of the present application, in step 3, the target Q network is used to calculate the target Q value, and the target Q value calculation formula (7) is as follows:

[0106]

[0107] where y t is the target Q value, a t+1 is the next action output by the policy network, and γ is the discount factor. is the target double Q network, and α is the weight coefficient of the policy entropy, and logπ φ (a t+1 |s t+1 ) is the policy entropy.

[0108] In the step 3, the gradient descent is used to minimize the loss function, as shown in equation (8).

[0109]

[0110] where N is the batch size, Q θ is the current Q network.

[0111] In the step 3, the soft update of the target Q network is performed, as shown in equation (9).

[0112] θ target ← τθ + (1 - τ)θ target , (9)

[0113] In the step 3, if the current sampling point reaches the target area, collides with the obstacle, or exceeds the maximum step limit, the current path sampling task is determined to be completed, the environment is reset, and the sampling is restarted. The above training process is repeated until the maximum training round or the total training step is reached, the SAC policy network training is completed, and the SAC guided sampling model based on the SAC policy network is obtained.

[0114] Step 4, the target parameters of the SAC guided sampling model are optimized based on the obstacle.

[0115] The sensitivity analysis of the key parameters of the SAC guided sampling model in the training process is performed, such as the learning rate, the discount factor, the batch size, etc. The SAC guided sampling model training based on the SAC algorithm is performed with different parameters. The reward curve after the model training under different parameters is compared, and the SAC guided sampling model under the optimized parameters is obtained, which is used to improve the execution of the guided sampling strategy of the RRT path planning.

[0116] In the step 4, the model parameter optimization training process, the obstacle is part of the biological anatomy model obtained in the step 1, the obstacle space range is 260mm x 320mm x 170mm, the path planning starting point is set to (188, 340, -294), and the end point is set to (65, 304, -270). The maximum bending curvature kmax The discount factor is set to 0.95, the batch size is set to 128, the learning rate is set to [0.00001, 0.00005, 0.0001, 0.0005, 0.001] respectively, and the reward curve under different learning rates is obtained by training. When the learning rate is 0.001 and 0.0005, the reward is higher, but the curve fluctuates more. When the learning rate is 0.00001, the reward is lower and the curve converges more slowly. When the learning rate is 0.0001 and 0.00005, the curve converges relatively quickly, the reward is relatively high and the fluctuation is relatively small, and it is more stable.

[0117] In the parameter tuning, first set the discount factor to 0.95, the batch size to 128, the learning rate to [0.00001, 0.00005, 0.0001, 0.0005, 0.001] respectively, and the reward curve under different learning rates is obtained by training. When the learning rate is 0.001 and 0.0005, the reward is higher, but the curve fluctuates more. When the learning rate is 0.00001, the reward is lower and the curve converges more slowly. When the learning rate is 0.0001 and 0.00005, the curve converges relatively quickly, the reward is relatively high and the fluctuation is relatively small, and it is more stable.

[0118] Then set the learning rate to 0.0001, the batch size to 128, and the discount factor to [0.99, 0.97, 0.95, 0.93, 0.90] respectively, and the reward curve under different discount factors is obtained by training. The discount factor γ∈[0,1], close to 1, suitable for long-term planning and long-term optimization of strategy. The discount factor has little effect on the reward curve. When the discount factor is 0.95, the curve converges quickly, the reward is high and the fluctuation is small, and it is relatively balanced.

[0119] Then set the learning rate to 0.0001, the discount factor to 0.95, and the batch size to [256, 128, 64, 32, 16] respectively, and the reward curve under different batch sizes is obtained by training. The batch size will affect the learning efficiency of model training. Small batch size will lead to unstable reward curve and high volatility, while large batch size will lead to lower reward due to suboptimal learning. When the batch size is 64 and 128, the reward curve fluctuates less and is more stable.

[0120] By training with the above selected different parameter combinations, it is found that the parameters of learning rate 0.0001, discount factor 0.95 and batch size 128 are the optimal parameters. Under these optimal parameters, the SACSAC guided sampling model is trained and optimized, and the trained SACSAC guided sampling model is saved. It will be used in the subsequent execution of the guided sampling strategy for improving the RRT path planning.

[0121] Step 5, the improved RRT path planning is sampled based on the hybrid sampling strategy by using the parameter-optimized SAC guided sampling model, and a feasible path is obtained.

[0122] As shown in Figure 4 , the hybrid sampling strategy is used for sampling, and then the detection of collision with obstacles is performed. When it is judged that there is no collision with obstacles, the step of node reconnection is entered. After the node reconnection is completed, it is judged whether the kinematic constraint is met. According to the judgment result, it is determined whether the sampling point is put into the RRT search tree. Then, the sampling process is repeated until the target point is reached. After that, the RRT search tree is backtracked, and a feasible path is obtained. In the node sampling, it is judged whether to guide sampling. If yes, the SAC guided sampling model is used for SAC guided sampling. If no, uniform sampling in the whole space is performed, and finally the sampling nodes under different proportions of sampling strategies are obtained. The node reconnection is that the newly generated sampling point is re-traversed in the search tree to find a parent node that can make the path from the current node to the root node / target point shorter and safer (better obstacle avoidance), replace the original parent node, and optimize the search tree structure.

[0123] The embodiments of the present application adopt a hybrid sampling strategy. The sampling points generated by the SAC strategy network are used with a certain probability or proportion to accelerate convergence and improve the quality and success rate of the planned path. At the same time, the traditional random sampling is reserved with a certain probability or proportion to ensure that potential feasible paths are not ignored.

[0124] To verify the feasibility of the sampling of the present application, different hybrid sampling proportions are used for sampling, and path planning simulation experiments are performed to compare the performance of the corresponding path planning. In the corresponding simulation experiments, the SAC strategy guided sampling with proportions of 0%, 25%, 50%, and 75% is used, and the remaining hybrid sampling strategy using traditional uniform sampling in the whole space is used. 100 path planning simulation experiments are performed to compare the performance of the corresponding path planning.

[0125] Through the simulation experiment, it is found that, compared with the traditional RRT algorithm without adding SAC guided sampling (i.e., the hybrid proportion is 0%), the path planning algorithm with SAC guided sampling has a relatively reduced path length. When the hybrid proportion is 75%, the average relative path length can reach 1.20. Compared with the traditional RRT algorithm without adding SAC guided sampling, the average relative path length is reduced by 19%. It is shown that the SAC guided sampling strategy proposed in the embodiments of the present application can effectively guide the search direction, reduce the generation of invalid nodes, shorten the path length, and optimize the path.

[0126] The simulation experiment finds that when the mixed sampling ratio is 25%, 50%, and 75% respectively, the average planning time is reduced by 11%, 27%, and 36% respectively. Compared with the traditional RRT algorithm without adding SAC guided sampling, the number of nodes of the feasible path found by the path planning algorithm with SAC guidance and the average iteration number are also reduced. When the mixed sampling ratio is 25%, 50%, and 75% respectively, the average number of nodes of the path found is reduced by 28%, 53%, and 62% respectively, and the average iteration number when finding the path is reduced by 28%, 50%, and 55% respectively.

[0127] The above shows that the mixed sampling strategy proposed in the embodiments of the application achieves a good balance between global search and local optimization capability. Although the performance of the path planning algorithm with a higher guided sampling ratio is better, a higher guided sampling ratio does not necessarily make the performance of the path planning algorithm better. When the guided sampling ratio reaches 100%, due to the lack of global search capability, it may fall into local optimization, resulting in that the path planning algorithm does not converge and cannot find a feasible path. Therefore, the performance of the algorithm can be further improved by appropriately configuring the mixed sampling ratio.

[0128] The improved RRT path planning algorithm is obtained by appropriately configuring the mixed sampling ratio, and is compared with other path planning algorithms. The simulation experiment shows that when the SAC guided sampling ratio is 75%, the corresponding path planning algorithm has relatively optimal performance. The SAC-RRT algorithm with a guided sampling ratio of 75% is compared with the bidirectional RRT algorithm (Bi-RRT) and the greedy heuristic RRT algorithm (GH-RRT). The relative path length of the SAC-RRT algorithm proposed in the embodiments of the application is better than that of Bi-RRT and GH-RRT, and the relative path length is reduced by 6.6% and 4.2% respectively, which shows that the SAC-RRT algorithm can effectively optimize the path and shorten the path length.

[0129] Since the reduction of path length is of great significance to reducing tissue damage caused by puncture and can effectively improve the safety of puncture surgery, the calculation efficiency and convergence speed of the SAC-RRT algorithm proposed in the embodiments of the application are also better than those of Bi-RRT and GH-RRT, and the average calculation time is reduced by 18.72% and 19.88% respectively, and the average number of nodes is reduced by 32.76% and 17.00% respectively.

[0130] By comparing the iteration numbers required for convergence of the three algorithms under the same obstacle condition, the average iteration number of the SAC-RRT algorithm proposed in the embodiment of the present application is 104.03, which indicates that the SAC guided sampling proposed in the embodiment of the present application can provide effective sampling guidance in the path planning process, reduce the generation of invalid nodes, and also reduce the number of nodes in convergence, thereby accelerating the convergence process and effectively improving the planning efficiency.

[0131] The basic principles and main features of the present application and the advantages of the present application are shown and described above, and it is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application;

[0132] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application.

[0133] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity, those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be properly combined to form other embodiments which can be understood by those skilled in the art.

Claims

1. An improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling, characterized in that, Comprising the following steps: S1. Reconstruct the three-dimensional model of the actual physiological structure of the human chest and abdomen based on the CT image, obtain the three-dimensional model of the actual physiological structure of the human chest and abdomen, and use it as an obstacle for subsequent path planning; S2. Establish a reinforcement learning environment based on the obstacle, including setting the state space, action space, and reward function; S3. In the reinforcement learning environment, train the SAC guided sampling model for flexible needle path planning based on the obstacle using the SAC algorithm; S4. Optimize the target parameters of the SAC guided sampling model based on the obstacle; S5. Use the SAC guided sampling model after parameter optimization to sample in the improved RRT path planning based on the hybrid sampling strategy to obtain a feasible path.

2. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling according to claim 1, wherein, In step S1, the CT image is used to reconstruct the three-dimensional model of the actual physiological structure of the human chest and abdomen, which includes: Using an adaptive median filtering algorithm, execute point by point in a sliding cubic window manner throughout the CT image to remove noise in the CT image while preserving small structures; Using a morphological gradient method to preprocess the CT image after noise removal, enhancing the edge and detail features in the CT image; Using an adaptive seed point growing algorithm to segment the CT image after enhancing the edge and detail features to segment tissues such as bones, trachea and bronchi, and blood vessels; Using a Flying-Edges three-dimensional reconstruction algorithm to model each tissue based on the segmented image to obtain a three-dimensional model of the actual physiological structure of the human chest and abdomen, including important organs and tissues such as bones, trachea and bronchi, and blood vessels.

3. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling of claim 2, wherein, The adaptive median filtering algorithm is used to remove noise in the CT image by executing point by point in a sliding cubic window manner throughout the CT image, which includes: Defining characteristic values H1, H2, H3, and H4; wherein H xyz is the image center gray scale of the current filter cubic window C xyz H med , H max , H min are the image median gray scale, the maximum gray scale and the minimum gray scale of the current filter cubic window C xyz , respectively. The maximum volume of the filter cube window is set as V max , the size of the initial filter cube window is defined, the eigenvalue is calculated, and the following judgment is made: If H1 < 0, H2 > 0 and H3 < 0, H4 > 0, it is considered that the current image does not belong to noise points, and the central gray value is kept unchanged; if H1 < 0, H2 > 0 and H3 ≥ 0 or H4 ≤ 0, the median gray value is used to replace the central gray value; if H1 ≥ 0 or H2 ≤ 0, the size of the filter cubic window is increased, the characteristic values are repeatedly calculated, the judgment is performed, and the judgment is completed or the maximum volume V of the filter cubic window is reached max .

4. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling of claim 2, wherein, In the preprocessing of the CT image after noise removal using the morphological gradient method, the expression of the morphological gradient is as follows: Where V is a local cubic sampling window, i, j, and k represent the change in the three directions within the sampling window, D(x, y, z) is the dilation of the image, and E(x, y, z) is the erosion of the image.

5. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling of claim 1, wherein, In step S2, the state space includes the current node position, the direction vector between the current node and the target point, the distance between the current node and the target point, the distance between the current node and the starting point, and the minimum distance between the current node and the obstacle; the action space is a set of unit direction vectors within a conical space with the current needle tip direction as the axis, with an included angle not exceeding the maximum deflection angle; the reward function includes an approach to the target position reward function, a negative reward function for approaching the obstacle or colliding with the obstacle, a large reward function for reaching the target position, and a path curvature constraint reward function; The state space is Where, p node =(x n ,y n ,z n Current node position, For the distance from the current node to the target point p goal =(x g ,y g ,z g The direction vector between ) and d goal =||p goal -p node || represents the distance between the current node and the target point, d start =||p node -p start || represents the current node and the starting point p. start The distance between them For the current node and the j-th obstacle The minimum distance between them, where M is the number of obstacles; The action space is After the direction vector of the node sampling is determined, the next sampling node position is p sample = p node + a t l depth ; where a t is a direction vector of the next sampling direction, is the current needle tip direction vector, β max denotes the maximum deflection angle, l depth ∈ [0, l max ] is the node sampling depth, l max is the maximum node sampling depth; The reward function is R = ω1R near + ω2R collision + ω3R arrive + ω4R constraint ; The proximity target position reward function is R near = r1(||p i-1 -p goal ||) - r2(||p i -p goal ||); The negative reward function is The huge reward function is The path curvature constraint reward function is where ||p i-1 -p goal || is the distance from the i-1th node to the target point, ||p i -p goal || is the distance from the ith node to the target point, d safe is the threshold distance of the node to the obstacle, ε is the threshold distance of the node to the target, k t is the curvature of the current path, k max is the maximum curvature of the flexible needle bending, r1, r2, r3, r4 are the normalization coefficients of each reward function respectively, and ω1, ω2, ω3, ω4 are the weight coefficients of each reward function respectively.

6. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling according to claim 5, wherein, In step S3, in the reinforcement learning environment, the SAC guided sampling model for flexible needle path planning is trained based on the obstacle using the SAC algorithm, which includes: S21. Initialize the training parameters; initialize the flexible needle path planning environment, which is a three-dimensional space containing the starting point, target point, and obstacle space distribution of the flexible needle path planning, and the obstacle is discretized as a point cloud; S22. Reset the training environment; reset the current sampling node as the starting point of path planning; S23. Collect state information s of current node i , SAC policy network generates action a i , update state according to action a i , get updated state s i+1 , immediate reward r i , and flag d of whether to terminate i ∈{0,1}; S24. Determine whether the current sampling point collides with the obstacle. If yes, return to step S23 to generate a new sampling point. If no, store the current collected information (s i , a i , r i , s i+1 , d i ) in the experience replay buffer D; S25. Determine whether there is certain data, if yes, obtain the data from the experience replay buffer D, calculate the target Q value using the target Q network, optimize the SAC policy network and the current Q network parameters by gradient descent minimization of the loss function, perform soft update of the target Q network, if no, execute the next step; S26. Determine whether the current sampling point exceeds the maximum step limit, if yes, determine that the current path sampling task is completed, execute the next step; if no, continue to determine whether the current sampling point reaches the target area, if yes, determine that the current path sampling task is completed, execute the next step; if no, collect the next node state information, return to step S23; S27. Determine whether the training process reaches the maximum training round, if yes, end, complete the training of the SAC policy network, and obtain the SAC guided sampling model; if no, enter the next training round, return to step S22, reset the training environment, and start sampling again to continue training.

7. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling according to claim 6, wherein, The target Q value calculation formula is as follows: where y t is the target Q value, a t+1 is the next action output by the SAC policy network, r t represents the immediate reward, d t is an indicator variable representing whether in the terminal state, and γ is the discount factor, is the target double Q network, and a is the weight coefficient of policy entropy, logπ φ (a t+1 |s t+1 ) is the policy entropy; Gradient descent minimizes the loss function as where N is the batch size, Q θ is the current Q network.

8. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling of claim 1, wherein, The soft update of the target Q network is performed by the following formula. θ target ← τθ + (1 - τ)θ target ; where θ is the parameter of the current Q network, θ target is the parameter of the target Q network, and τ is a soft update coefficient.

9. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling of claim 1, wherein, In step S4, the target parameters include learning rate, discount factor, and batch size; the SAC guided sampling model of the SAC algorithm is trained by using different combinations of target parameters, the reward curves after model training under different target parameters are compared, and the guided sampling training model under the optimized target parameters is obtained.

10. The improved RRT path planning method for robot-assisted puncture based on SAC-based guided sampling of claim 1, wherein, In step S5, the SAC guided sampling model after parameter optimization is used to sample in the improved RRT path planning based on the mixed sampling strategy, and a feasible path is obtained, including: Path planning parameter initialization, determining the starting point, target point, and obstacle; Determine whether the starting point and the target point are in the obstacle, if no, add the starting point to the RRT search tree, and then enter the node sampling stage, if yes, end; In the node sampling stage, sampling is performed based on the mixed sampling strategy to obtain a sampling node; then enter the path constraint stage; In the path constraint stage, determine whether the sampling node is in the obstacle; if not in the obstacle, perform obstacle collision detection to determine whether it collides with the obstacle, if not collision, reconnect the node and perform flexible needle kinematics constraint judgment; if the flexible needle kinematics constraint is met, enter the path acquisition stage, add the sampling node to the RRT search tree, and then determine whether the target point is reached, if the target point is reached, backtrack the RRT search tree to find the path, obtain the feasible path, and end; If it is determined that the sampling node is in the obstacle, the sampling node collides with the obstacle, the sampling node does not meet the flexible needle kinematics constraint, or the target point is not reached, return to the node sampling stage for node sampling. The mixed sampling strategy uses the SAC guided sampling model to generate a sampling point for guided sampling in a first proportion, and randomly generates a sampling point for random sampling in a second proportion; the first proportion and the second proportion add up to 100%.

Images