An improved DQN-based three-dimensional layered dynamic path planning method for ablation probe

Abstract

The application discloses an ablation probe three-dimensional dynamic path planning method based on an improved DQN, belongs to the field of medical robots and minimally invasive interventional surgery, and aims to solve the technical problems of insufficient dynamic adaptability and low exploration efficiency of traditional path planning in liver radiofrequency ablation. The method comprises three steps: firstly, a bounding box search pre-planning is carried out based on an improved RRT algorithm, and a global initial path is quickly constructed to introduce prior information for DQN training; secondly, a RRT-guided DQN dynamic hierarchical planning is adopted, a "skin-liver" double-layer tissue blood vessel model is built by fusing the respiratory coupling characteristics, a guide mechanism and a multi-objective reward function are designed, and fine path optimization is completed by combining hierarchical dynamic collision detection; and finally, the optimal operation path is output by quantitative screening through a multi-dimensional evaluation system such as safety and smoothness. The application combines the advantages of global exploration and local optimization, adapts to the three-dimensional dynamic environment of the liver, and can provide reliable path support for robot-assisted radiofrequency ablation surgery.

Description

Technical Field

[0001] This invention belongs to the field of medical robots and minimally invasive interventional surgery path planning technology, and specifically relates to a three-dimensional dynamic path planning method for a flexible ablation probe in liver radiofrequency ablation surgery. Background Technology

[0002] Primary liver cancer is the sixth most common malignant tumor worldwide and has an extremely high mortality rate. Radiofrequency ablation, as a core minimally invasive interventional treatment, has been widely used to treat liver cancer patients who cannot undergo surgical resection. In radiofrequency ablation surgery, the puncture path planning of the flexible ablation probe is a crucial factor determining the safety and precision of the procedure. The accuracy of the path planning directly affects the tumor ablation effect and the risk of intraoperative complications. Therefore, developing an ablation probe path planning algorithm that can be readily applied clinically is of great significance.

[0003] Existing flexible needle puncture path planning methods can be mainly divided into three categories: The first category is the traditional geometric and potential field method, which is simple to calculate and has strong real-time performance, but it is prone to getting trapped in local optima and is difficult to adapt to complex three-dimensional anatomical environments; The second category is sampling-based path planning algorithms, which have probabilistic completeness and global rapid exploration capabilities, but their random sampling characteristics can only guarantee path feasibility, not path optimality. The generated paths are mostly spliced ​​polylines, which are difficult to match the circular kinematic characteristics of flexible needles and have extremely poor adaptability to dynamic environments; The third category is intelligent planning algorithms based on reinforcement learning, which have strong nonlinear fitting capabilities and dynamic environment adaptation capabilities, but the pure DQN algorithm has core problems such as blind exploration in the early stage of training, slow convergence speed, and low learning efficiency under sparse rewards. Most studies have only completed the verification in simplified simulation environments and have not fully considered the tissue layering characteristics and respiratory dynamic deformation in clinical practice. Meanwhile, most existing path planning studies are based on static anatomical models, ignoring the periodic displacement of the liver and blood vessels caused by the patient's respiratory movements during clinical surgery, and also failing to consider the differences in mechanical properties between the skin and liver layers. The planning results are seriously out of touch with the actual clinical environment, and safety risks such as needle tip trajectory deviation and accidental vascular injury are likely to occur during surgery.

[0004] Therefore, it is necessary to design a three-dimensional dynamic path planning algorithm for a flexible ablation probe specifically for clinical liver radiofrequency ablation surgery to address the aforementioned core pain points in existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, such as the coarse path of pure RRT algorithms, which cannot guarantee optimality and have poor dynamic adaptability; the static planning model failing to consider respiratory motion and tissue layering characteristics, resulting in a disconnect from clinical practice; and the lack of a multi-dimensional path quality quantification evaluation system for clinical punctures, this invention proposes a three-dimensional dynamic path planning method for ablation probes based on improved DQN. The technical solution adopted by this invention to solve the above problems specifically includes the following steps:

[0006] Step 1: Load the obstacle point cloud, set the path start point and path target point, and pre-plan using the Fast Expanding Random Tree (RRT) algorithm with a limited search bounding box range to generate a globally feasible initial path; modify the RRT algorithm to dynamically adjust the boundary of the search bounding box according to the puncture start point and tumor target point to limit the random sampling space and generate a globally feasible initial path.

[0007] Step 2: On the global feasible initial path, execute the RRT-guided DQN algorithm for dynamic hierarchical path planning, construct a clinically relevant tissue physics model and dynamic vascular model, design the RRT-guided DQN full-process mechanism and multi-objective fusion reward function, and complete the fine optimization of the path by combining hierarchical dynamic collision detection.

[0008] Step 3: Quantitatively evaluate the planned path through a multi-dimensional path quality assessment system, select the best planned path for multiple needle insertion points, and finally output the optimal puncture path.

[0009] Furthermore, the improved RRT algorithm bounding box search pre-planning in step one specifically includes:

[0010] S11. Environment initialization: Load the 3D point cloud models of liver, intrahepatic artery, portal vein, tumor and skin reconstructed based on the 3Dircadb dataset, merge the artery and portal vein point clouds as the core obstacle set, set the needle entry point on the skin surface as the path starting point and the tumor target point as the path target point, and construct a 3D Cartesian programming space.

[0011] S12. Definition of Bounding Box: An adaptive search bounding box is constructed based on the starting point and the target point. The formula for calculating the boundary of the bounding box is:

[0012] ,

[0013] In the formula These are the three-dimensional coordinates of the starting point; It is the three-dimensional coordinate formula of the target point. It is an extended bounding box centered on the line connecting the start and end points. It is a dimension of spatial planning; The bounding box volume is calculated using the formula for defining the boundary length of the defined range:

[0014] ,

[0015] S13. Target-biased sampling strategy design: Within the volume range of the cropped adaptive search bounding box, a hybrid distribution sampling strategy is used to generate candidate random sampling points in space. The sampling formula is:

[0016] ,

[0017] In the formula, For random sampling points, It is the target bias probability. It is a Dirac distribution centered on the target point. It is a uniform distribution within the bounding box;

[0018] S14. Adaptive exploration step size and collision detection: Design an adaptive step size adjustment strategy, starting from the end of the currently generated path and extending towards the random sampling point to form an extended line segment, and perform collision detection on the extended line segment; wherein, in the initial state, the end of the path is the starting point of the path.

[0019] The step size is positively correlated with the distance from the current point to the nearest obstacle; collision detection is performed on the extended line segment, starting from... arrive The formula for calculating the number of line segment sampling points is:

[0020]

[0021] In the formula It is the base step size. Indicates the sampling step size, and Check each sampling point for collisions. If any point collides, the entire line segment is considered to be in collision.

[0022] Furthermore, in step S14, the collision detection needs to meet three major clinical constraints: path length constraint, safety distance constraint, and needle insertion angle constraint.

[0023] Specifically, the path length constraint is as follows:

[0024] ,

[0025] In the formula Indicates the actual starting point of the flexible needle puncture path. Indicates the target point of the flexible needle. Indicates the threshold for the insertion length of the flexible needle;

[0026] Specifically, the safety distance constraint is as follows:

[0027] ,

[0028] In the formula It is the center of the vascular model. It is the radius of the blood vessel model. It is a set value that retains a certain boundary distance;

[0029] Specifically, the needle insertion angle constraint is as follows:

[0030] ,

[0031] In the formula This is the axis vector of the flexible needle insertion. Let be the tangent vector on the surface of the liver capsule. This is a constraint on the angle between the path and the liver capsule.

[0032] Furthermore, the RRT-guided DQN algorithm dynamic hierarchical path planning in step two specifically includes:

[0033] S21. Construct a physical model of a two-layer tissue “skin-liver” and set the mechanical parameters and maximum curvature constraints of each layer of tissue to establish a mathematical representation of the layered structure.

[0034] S22. Based on the aforementioned two-layer tissue physical model, a dynamic vascular model coupled with respiratory motion is constructed, and a layered dynamic collision detection strategy is designed to realize path planning in a three-dimensional dynamic environment.

[0035] S23. Design a full-process guidance and planning mechanism for DQN using RRT based on the globally feasible initial path;

[0036] S24. Design a multi-objective fusion reward function, construct an end-to-end optimization objective, and combine it with hierarchical dynamic collision detection to complete the fine-grained optimization of the path.

[0037] Furthermore, in step S21, the physical model of the double-layer tissue and the curvature calculation method are specifically as follows:

[0038] First, a physical model of the "skin-liver" two-layer tissue is constructed, defining the maximum permissible curvature of the skin layer. Maximum permissible curvature of the liver layer Safe collision radius ;

[0039] Furthermore, the hierarchical structure is mathematically represented for any puncture needle position. Define its minimum distance to the skin layer: Minimum distance to the liver layer: Set a uniform entry threshold ;

[0040] The condition for determining whether the puncture needle has entered the skin layer is: The minimum distance to the skin layer is less than the set uniform entry threshold. ,Right now:

[0041] ,

[0042] The conditions for entering the liver layer are The minimum distance to the liver layer is less than the set uniform entry threshold. ,Right now:

[0043] ,

[0044] When the above conditions are met, it is considered that the puncture needle has touched the surface of the layer, and the position is recorded as the entry point;

[0045] Finally, the curvature of discrete path points is calculated based on numerical difference. In the actual puncture path, the set of path points is a discrete sequence. Using numerical difference approximation, and assuming the path point spacing may be uneven, a first-order difference vector is defined as follows: Define a second-order difference vector: ;

[0046] For interior points The discrete approximation formula for its curvature is:

[0047] ,

[0048] In the formula Approximate as a point The tangential direction at that point, i.e., the velocity; It is approximately the rate of change in the tangential direction, i.e., acceleration.

[0049] Puncture can only be actually achieved at the physical level when the curvature of the discrete path points meets the maximum permissible curvature of the skin layer and the maximum permissible curvature of the liver layer.

[0050] Furthermore, in step S22, the method for constructing the dynamic vascular model is specifically as follows:

[0051] Let the initial time be , The position of a certain point on a blood vessel at any given time is ,but Represented as:

[0052] ,

[0053] In the formula It is a displacement vector. , The amplitude represents the maximum displacement of the blood vessel during the respiratory cycle. It's the respiratory rate. It is the initial phase. It is the unit vector of the direction of motion;

[0054] The specific method for layered dynamic collision detection is as follows:

[0055] This invention employs a planning-then-verification approach: first, a preliminary path is generated under static environment assumptions using a hybrid algorithm combining RRT and DQN; then, the safety of this path during vascular movement is verified through dynamic puncture simulation. The simulation process discretizes the path into several time points, updates the vessel position chronologically, performs collision detection, and records the number of collisions and layered entry points. The final quality assessment of the path comprehensively considers static indicators such as length, curvature, and safety radius, as well as dynamic indicators such as the number of collisions and the dynamic success rate. The dynamic success rate is defined as... :

[0056]

[0057] in, For the number of collisions, To check the total number of attempts, if a certain path can be safely completed for puncture in multiple simulations, then... If the value is close to 1, it is considered to have good dynamic adaptability.

[0058] set up For at any time The minimum distance from the needle tip to the point cloud of the obstacle; if this minimum distance is less than the safe radius... If so, a collision is determined to have occurred.

[0059] Furthermore, in step S23, the RRT's full-process guided planning mechanism for DQN includes:

[0060] First, calculate the direction vector of RRT-guided DQN, specifically:

[0061] ,

[0062] In the formula This is the current state of DQN; These are discrete points on the global path pre-planned by the RRT algorithm; Distance on the RRT path The nearest index; from A vector pointing to the next point (if the nearest point is not the endpoint) or the endpoint on the RRT path;

[0063] This leads to state enhancement and state enhancement concatenation, specifically:

[0064] ,

[0065] in It is the guiding direction of normalization. As a scaling factor, the numerical range is adjusted to be comparable to that of other state components; It is the current position after normalization; It is the normalized relative target position, that is, the target minus the current position; It is a historical position shift, reflecting recent movement trends; , , and Concatenate into a 12-dimensional state vector ;

[0066] Furthermore, a hybrid action selection strategy is designed, integrating RRT guidance with DQN autonomous decision-making, to balance guided learning and autonomous exploration. Specifically:

[0067] ,

[0068] In the formula, the RRT action decision is to move one step in the direction guided by the RRT; probability , These are the weighting coefficients. It is the attenuation factor;

[0069] The attenuation factor depends on the degree of deviation of the current state from the RRT path. If the current position deviates far from the RRT path, the attenuation factor decreases, and the probability of using the RRT action decreases; otherwise, the action corresponding to the maximum Q value output by the DQN network is selected.

[0070] The RRT guidance process employs a multi-objective fusion reward function. Specifically:

[0071] ,

[0072] In the formula It consists of basic rewards, including reaching the goal, collision penalties, and redundant step penalties; constants. , , , , The weighting coefficients are used; the guiding reward consists of two parts: encouraging the agent to move closer to the RRT path and encouraging the agent to move along the RRT path; M is the maximum index value of all path points in the RRT search; the weighting coefficients and decay factors are consistent with those in action selection, so that the guiding reward is significant only when the agent is close to the RRT path.

[0073] Finally, the Q-value is guided to learn and update, specifically as follows:

[0074] In temporal differential updates, an RRT guiding term is introduced to optimize the update direction of the value function. The formula for calculating the objective Q-value is as follows:

[0075] ,

[0076] In the formula It is the normalized RRT guiding direction; It is the gradient of the Q-function with respect to the state position component; The consistency between the direction of RRT guidance and the direction of Q value increase is measured. If they are consistent, the target Q value is increased, thereby encouraging the selection of actions consistent with RRT guidance. It controls the intensity of the guidance.

[0077] Further, in step S23, the multi-objective fusion reward function includes distance reward, RRT guidance reward, target reward, collision penalty, and hierarchical compliance reward; wherein, the hierarchical compliance reward includes a basic curvature score that evaluates the curvature of a single path point and a layer boundary smoothness score that evaluates the curvature near the boundary.

[0078] Multi-objective fusion reward function Specifically:

[0079] ,

[0080] The sub-items are: distance rewards RRT Guided Rewards Target Rewards Collision penalty and tiered compliance rewards ;

[0081] The distance reward Specifically:

[0082] ,

[0083] in, for , is the progress weighting coefficient; for , is the efficiency weighting coefficient; and This is a preset constant; It is a progress reward; It is an efficiency reward;

[0084] The RRT guidance reward Specifically:

[0085] ,

[0086] in, This represents the weighting of the reward based on the distance between the agent and the RRT path, such that the reward decreases when the agent moves further away from the RRT path. The specific expression is:

[0087] ,

[0088] in For path-following rewards, ; Rewards for consistent progress; For orientation alignment rewards, .

[0089] The target reward Specifically, when the distance between the needle tip and the target point is less than a preset distance value, it is determined that the target area has been reached.

[0090] The collision penalty Specifically: when the needle tip collides with an obstacle, a negative penalty is imposed and the current training round is terminated;

[0091] The tiered compliance reward It consists of a basic curvature score and a layer boundary smoothness score.

[0092] The basic curvature score is used to assess whether the curvature at a single point on the path is within the allowable range; for any point on the path... Let its curvature be The maximum allowable curvature of the layer containing this point is The basic curvature score is:

[0093] ,

[0094] In the formula, when the curvature of the current point Not exceeding the maximum permissible curvature When the curvature of the current point exceeds the allowable value (i.e., the maximum allowable curvature), a score of 1.0 is given, indicating that the path at that point is sufficiently smooth. At that time, the score decays linearly, specifically by the excess percentage. The larger the value, the lower the score; the critical point where the score decays to 0 occurs at... At that time, ;like If it is larger, the score is negative;

[0095] Layer boundary smoothness score:

[0096] ,

[0097] in , representing the average curvature of the boundary segment, is used to assess the degree of path curvature near the boundary. and Let be the curvatures of the two points before and after the boundary;

[0098] Single-layer compliance score:

[0099] ,

[0100] In the formula, the binary function Used to indicate the first on the path The location of each point;

[0101] Further, in step S24, the dynamic blood vessel model and the hierarchical dynamic collision detection specifically involve:

[0102] First, a dynamic blood vessel model is constructed, with an initial time set. , The position of a certain point on a blood vessel at any given time is ,but Represented as:

[0103] ,

[0104] In the formula It is a displacement vector. ;in, The amplitude is measured in millimeters and represents the maximum displacement of blood vessels during the respiratory cycle. It's the respiratory rate. It is the initial phase. It is the unit vector of the direction of motion;

[0105] Furthermore, a hierarchical dynamic collision detection strategy is constructed, and a hierarchical dynamic collision detection strategy is established. For at any time The minimum distance from the needle tip to the point cloud of the obstacle; if this minimum distance is less than the safe radius... The formula for determining if a collision has occurred is as follows:

[0106] .

[0107] Furthermore, the multi-dimensional path quality assessment system in step three specifically includes:

[0108] S31. Construct a 6-dimensional normalized evaluation index. The values ​​of the safety, smoothness, feasibility, efficiency, linearity, and hierarchical compliance indices are all within the range of [0,1].

[0109] (1) Specifically, regarding security, let the path point be... Obstacle point cloud is , For point clouds Points within, defined The minimum distance from a path point to a point within the point cloud. The global minimum distance of the path is The safety radius is Specifically, it is expressed as:

[0110]

[0111] (2) Smoothness is specifically designed based on the maximum curvature and average curvature of the path.

[0112] ,

[0113] in and The specific calculation method is as follows:

[0114] ;

[0115] (3) Feasibility specifically involves checking whether each line segment in the path is collision-free, and for each pair of adjacent points A preset number of intermediate points are collected at a preset sampling density, and the collision between these intermediate points is checked. If all sampling points of a line segment are free of collision, the segment is considered feasible.

[0116] ,

[0117] in , This represents the number of line segments that passed the verification.

[0118] (4) Efficiency is specifically defined as the ratio of the actual path length to the straight-line distance from the starting point to the ending point. Let the actual path length be:

[0119] ;

[0120] The straight-line distance is:

[0121] ;

[0122] Efficiency is defined as the ratio of straight-line distance to actual length. ;

[0123] (5) The linearity is specifically defined as the linearity of each intermediate point on the path. Calculate the perpendicular distance from the starting point to the line connecting the starting point and the ending point, take the average value, and then normalize it; let the starting point be... ,end , direction vector of a line Line length For points , , vertical point vertical distance Average vertical distance Straightness fraction is: ;

[0124] (6) Layered compliance specifically refers to:

[0125] For any point on the path Let its curvature be ,and In mm -1 In units, the maximum allowable curvature of the layer containing this point is... The basic curvature score is:

[0126] ,

[0127] In the formula, when the curvature of the current point Not exceeding the maximum permissible curvature When the curvature of the current point exceeds the allowable value (i.e., the maximum allowable curvature), a score of 1.0 is given, indicating that the path at that point is sufficiently smooth. At that time, the score decays linearly, specifically by the excess percentage. The larger the value, the lower the score; the critical point where the score decays to 0 occurs at... At that time, ;like If it is larger, the score is negative;

[0128] Layer boundary smoothness score:

[0129] ,

[0130] in The single-layer compliance score is:

[0131] ,

[0132] In the formula, the binary function Used to indicate the first on the path The location of each point;

[0133] Therefore, the multi-level compliance score is:

[0134] ,

[0135] In the formula, This represents the total number of path points.

[0136] S32: Comprehensive score calculation, using a weighted summation formula:

[0137]

[0138] In the formula , , , , Using preset weights, the planned paths for multiple needle insertion points are sorted based on the comprehensive score, and the optimal puncture path with the highest comprehensive score is output.

[0139] The planned puncture paths for multiple needle insertion points are sorted based on the comprehensive score, and the optimal puncture path with the highest comprehensive score is output.

[0140] The planned puncture paths for multiple needle insertion points are sorted based on the comprehensive score, and the optimal puncture path with the highest comprehensive score is output.

[0141] Furthermore, candidate needle entry points for multiple needle entry points are automatically generated through the following steps:

[0142] (1) Downsample the skin point cloud and control the number of point clouds within a certain range. Within points, This is a preset constant;

[0143] (2) Calculate the Euclidean distance from each skin point to the tumor center, and select the point with the smallest distance. The candidate set consists of 10 points. This is a preset constant;

[0144] (3) Eliminate candidate points whose distance from the blood vessel is less than the safe threshold by obstacle collision filtering;

[0145] (4) Cluster the remaining candidate points using K-means clustering. kind, For each class, the point closest to the tumor center is selected as the final candidate needle insertion point, using a preset constant.

[0146] The beneficial effects of this invention are as follows:

[0147] This invention improves the adaptive bounding box and biased sampling of the RRT algorithm, limiting the search range of the algorithm and realizing fast search of globally feasible paths, solving the problems of blind training and slow convergence. At the same time, it improves the smoothness of the path by coordinating the DQN mechanism and multi-objective fusion reward function guided by the entire RRT process.

[0148] This invention constructs a "skin-liver" two-layer tissue physical model to meet the mechanical and curvature constraints of layered puncture, introduces a dynamic vascular model coupled with respiratory motion and layered dynamic collision detection, incorporates the time dimension into path planning, and achieves safe obstacle avoidance in a highly realistic dynamic environment. This effectively avoids the risk of flexible needle trajectory deviation and intraoperative vascular injury, and is more in line with clinical practice.

[0149] This invention constructs a quantitative evaluation system that includes multiple dimensions such as safety, feasibility, and stratified compliance. This system can comprehensively consider the quality of clinical puncture, intelligently sort multiple candidate needle insertion points, and ensure that the optimal puncture path with the highest comprehensive score is output. Attached Figure Description

[0150] Figure 1 This describes a hierarchical dynamic puncture path planning process based on the RRT-DQN algorithm. Detailed Implementation Specific Implementation Example 1:

[0152] This embodiment provides a specific implementation of an ablation probe three-dimensional dynamic path planning method based on improved DQN, combined with... Figure 1 This implementation method is described below. The hardware platform configuration is as follows: CPU: Intel Xeon Gold 5218, RAM: 64GB, GPU: NVIDIA Tesla T4 (16GB VRAM), Operating System: Ubuntu 20.04; The algorithm is implemented based on Python 3.8, and the core dependency libraries include PyVista 0.38, NumPy 1.23, SciPy 1.9, and PyTorch 1.13. The specific steps are as follows:

[0153] Step 1: Load the obstacle point cloud, set the path start point and path target point, and generate a globally feasible initial path by pre-planning using the Fast Expanding Random Tree algorithm with a limited search bounding box range; by modifying the RRT algorithm, dynamically adjust the boundary of the search bounding box according to the puncture start point and tumor target point to limit the random sampling space and generate a globally feasible initial path.

[0154] First, load the 3D point cloud models of the liver, intrahepatic artery, portal vein, tumor, and skin reconstructed based on the 3Dircadb dataset. The artery model contains 12,309 points and 4,475 faces, the portal vein model contains 9,908 points and 9,737 faces, the tumor model contains 1,002 points and 2,000 faces, the liver surface model contains 5,008 points and 10,000 faces, and the skin model contains 10,974 points and 19,986 faces.

[0155] The point clouds of arteries and portal veins were further merged into a core obstacle set. DBSCAN clustering (eps=6.0, min_samples=2) was used to extract core points and boundary points. Voxel downsampling (voxel_size=5.0) was used to control the number of obstacle point clouds to within 3000 points, balancing computational efficiency and collision detection accuracy.

[0156] Further, the needle entry point on the skin surface is set as the starting point of the path. Through an automatic needle entry point selection algorithm, three candidate needle entry points are generated on the skin surface, namely A [120.5,85.2,65.8], B [140.0,75.0,70.7], and C [110.0,95.0,60.0]. The tumor target point is set as the target point of the path, and a three-dimensional Cartesian programming space is constructed with coordinate units in millimeters.

[0157] For each candidate needle insertion point, an adaptive exploration bounding box is constructed with a basic step size of 12.0 mm, a basic target bias probability of 0.2, and a maximum number of iterations of 2000.

[0158] A biased target sampling strategy and adaptive step size adjustment are adopted to complete the global search of RRT, and collision detection is performed on the extended line segments. The safety boundary threshold is e=5mm and the minimum needle entry angle is 20 degrees.

[0159] For each needle insertion point, a feasible initial path from the starting point to the target point is generated, and the sequence of key points on the path is extracted to provide prior knowledge for subsequent DQN training.

[0160] Step 2: Execute RRT-guided DQN algorithm for dynamic hierarchical path planning, construct clinically relevant tissue physics models and dynamic vascular models, design the RRT-guided DQN full-process mechanism and multi-objective fusion reward function, and combine hierarchical dynamic collision detection to complete the fine-tuning of the path.

[0161] 1. Construct a physical model of the "skin-liver" two-layer tissue, clarify the mechanical parameters and maximum curvature constraints of each layer, and establish a mathematical representation of the layered structure and a method for calculating curvature:

[0162] Based on the anatomical structure and biomechanical properties of the human abdominal region, a two-layer tissue structure of "skin-liver" was constructed, with a maximum curvature of 0.05 mm⁻¹ for the skin layer, a maximum curvature of 0.02 mm⁻¹ for the liver layer, a safe collision radius of 7.0 mm, and a clinical puncture speed of 20 mm / s.

[0163] 2. Design a full-process guided planning mechanism for DQN using RRT.

[0164] The Dueling DQN network structure is constructed, with a three-layer fully connected feature extraction layer (512→256→128), which is separated into value stream (128→64→1) and advantage stream (128→64→53). The state space is a 12-dimensional augmentation vector, and the action space consists of 53 discrete actions.

[0165] Set the training hyperparameters as follows: learning rate lr = 3e-5, discount factor = 0.95, initial exploration rate = 1.0, decay rate, minimum exploration rate, experience replay buffer size 30000, batch size, and soft update of the target network every 150 steps. It controls the intensity of the guidance.

[0166] 3. Design a multi-objective fusion reward function that integrates distance reward, smoothness reward, RRT-guided reward, target reward, collision penalty, and hierarchical compliance reward. DQN training is completed through a full-process RRT-guided mechanism, with each needle insertion point completing training independently. The reward value and target point distance changes during the training process are recorded to solve the sparse reward problem in reinforcement learning.

[0167] 4. Construct a dynamic vascular model coupled with respiratory motion, with arterial amplitude of 2.0 mm, frequency of 0.2 Hz, and initial phase of 0; and portal vein amplitude of 1.5 mm, frequency of 0.2 Hz, and initial phase of 0. A layered dynamic collision detection strategy was designed to achieve path planning in a three-dimensional dynamic environment.

[0168] Perform full-time dynamic security verification on the trained path.

[0169] Step 3: Quantitatively evaluate the planned puncture path using a multi-dimensional path quality assessment system, select the best path from those with multiple needle insertion points, and finally output the optimal puncture path.

[0170] 1. Construct a 6-dimensional normalized path quality evaluation system, where all indicators range from [0,1]. Higher scores indicate better performance in that dimension. Specifically:

[0171] (1) Safety: Let the path point be Obstacle point cloud is ,definition The global minimum distance of the path is The safety radius is Specifically, it is expressed as:

[0172]

[0173] (2) Smoothness: Designed based on the maximum curvature and average curvature of the path in and The following formula:

[0174]

[0175] (3) Feasibility: Check whether each line segment in the path is collision-free (passable). For each pair of adjacent points Sample several intermediate points (density parameter sample_density) (Maximum number of samples: 10) Check these points for collisions. If all sampled points of a line segment are free of collisions, the segment is considered feasible. in , This represents the number of line segments that passed the verification.

[0176] (4) Efficiency: Based on the ratio of the actual path length to the straight-line distance from the starting point to the ending point (i.e., the degree of path tortuosity). Let the actual path length be... The straight-line distance is Efficiency is defined as the ratio of the straight-line distance to the actual length.

[0177] (5) Straightness: Measures the degree to which a path deviates from a straight line. For each intermediate point on the path... Calculate the perpendicular distance from the starting point to the line connecting the starting point and the ending point (the distance from the point to the line), take the average value, and normalize. Let the starting point be... ,end , direction vector of a line Line length For point , , vertical point vertical distance Average vertical distance The straightness fraction is:

[0178] (6) Layered compliance: For any point on the path Let its curvature be ,and In mm -1 In units, the maximum allowable curvature of the layer containing this point is... The basic curvature score is:

[0179] ,

[0180] In the formula, when the curvature of the current point Not exceeding the maximum permissible curvature When the curvature of the current point exceeds the allowable value (i.e., the maximum allowable curvature), a score of 1.0 is given, indicating that the path at that point is sufficiently smooth. At that time, the score decays linearly, specifically by the excess percentage. The larger the value, the lower the score; the critical point where the score decays to 0 occurs at... At that time, ;like If it is larger, the score is negative;

[0181] Layer boundary smoothness score:

[0182] ,

[0183] in The single-layer compliance score is:

[0184] ,

[0185] Therefore, the multi-level compliance score is:

[0186] ,

[0187] In the formula, This represents the total number of path points.

[0188] 2. Comprehensive Score Calculation and Optimal Path Output: The comprehensive path score is calculated using a weighted summation formula based on clinical weight allocation.

[0189]

[0190] Based on the comprehensive score, the planned paths for multiple needle insertion points are sorted in descending order, and the optimal puncture path with the highest comprehensive score is output. At the same time, the evaluation indicators of each dimension and the path visualization results are also output.

[0191] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any person skilled in the art can make some modifications or alterations to the disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention, without departing from the scope of the present invention, are still within the protection scope of the present invention. The content of the present invention is not limited to the above embodiments; a combination of one or more specific embodiments can also achieve the purpose of the invention.

Claims

1. A three-dimensional hierarchical dynamic path planning method for ablation probes based on improved DQN, characterized in that, Includes the following steps: Step 1: Load the obstacle point cloud, set the path start point and path target point, and pre-plan using the Fast Expanding Random Tree (RRT) algorithm with a limited search bounding box range to generate a globally feasible initial path; modify the RRT algorithm to dynamically adjust the boundary of the search bounding box according to the puncture start point and tumor target point to limit the random sampling space and generate a globally feasible initial path. Step 2: On the global feasible initial path, execute the RRT-guided DQN algorithm for dynamic hierarchical path planning, construct a clinically relevant tissue physics model and dynamic vascular model, design the RRT-guided DQN full-process mechanism and multi-objective fusion reward function, and complete the fine optimization of the path by combining hierarchical dynamic collision detection. Step 3: Quantitatively evaluate the planned path through a multi-dimensional path quality assessment system, select the best planned path for multiple needle insertion points, and finally output the optimal puncture path.

2. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 1, characterized in that, The modification to the RRT algorithm in step one includes pre-planning the bounding box search, specifically as follows: S11. Environment initialization: Load the point cloud model of the reconstructed tissue or organ, merge the point clouds of arteries and portal veins as the core obstacle set, set the needle entry point on the skin surface as the starting point of the path, the tumor target point as the target point of the path, and construct a three-dimensional Cartesian programming space. S12. Construct an adaptive search bounding box based on the starting point and the target point. The formula for calculating the boundary of the bounding box is: ， In the formula These are the three-dimensional coordinates of the starting point; These are the three-dimensional coordinates of the target point; It is an extended bounding box centered on the line connecting the start and end points; It is a dimension of spatial planning; It defines the length of the boundary of the range; The formula for calculating the volume of the bounding box is: ， The planned space is based on the bounding box volume. Cut; S13. Within the volume range of the cropped adaptive search bounding box, a hybrid distribution sampling strategy is used to generate candidate random sampling points in space; S14. Starting from the end of the currently generated path, extend towards the random sampling point to form an extended line segment, and perform collision detection on the extended line segment; wherein, in the initial state, the end of the path is the starting point of the path.

3. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 2, characterized in that, In step S14, the collision detection needs to meet the path length constraint, the safety distance constraint, and the needle insertion angle constraint. Specifically, the path length constraint is as follows: ， In the formula Indicates the actual starting point of the flexible needle puncture path. Indicates the target point of the flexible needle. This indicates the threshold for the insertion length of the flexible needle; Specifically, the safety distance constraint is as follows: ， In the formula It is the center of the vascular model. It is the radius of the blood vessel model. It is a set value that retains a certain boundary distance; Specifically, the needle insertion angle constraint is as follows: ， In the formula This is the axis vector of the flexible needle insertion. Let be the tangent vector on the surface of the liver capsule. The angle constraint between the path and the liver capsule.

4. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 1, characterized in that, The RRT-guided DQN algorithm dynamic hierarchical path planning in step two specifically includes: S21. Construct a physical model of a two-layer tissue, "skin-liver", and set the mechanical parameters and maximum curvature constraints of each layer of tissue to establish a mathematical representation of the layered structure. S22. Based on the aforementioned two-layer tissue physical model, a dynamic vascular model coupled with respiratory motion is constructed, and a layered dynamic collision detection strategy is designed to realize path planning in a three-dimensional dynamic environment. S23. Design a full-process guidance and planning mechanism for DQN using RRT based on the globally feasible initial path; S24. Design a multi-objective fusion reward function, construct an end-to-end optimization objective, and combine it with hierarchical dynamic collision detection to complete the fine-grained optimization of the path.

5. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 4, characterized in that, In step S21, the physical model of the double-layer tissue and the curvature calculation method are as follows: First, a physical model of the "skin-liver" two-layer tissue is constructed, defining the maximum permissible curvature of the skin layer, the maximum permissible curvature of the liver layer, and the safe collision radius. Then, the layered structure is mathematically represented for any puncture needle position. The condition for determining whether the puncture needle has entered the skin layer is: The minimum distance to the skin layer is less than the set uniform entry threshold. The criteria for determining entry into the liver layer are: The minimum distance to the liver layer is less than the set uniform entry threshold. When the above conditions are met, it is considered that the puncture needle has touched the surface of the corresponding layer, and the position is recorded as the entry point. Finally, the discrete curvature approximation value of the path point is calculated based on numerical difference. The discrete curvature approximation value of the path point satisfies the actual physical conditions for the puncture needle to enter the tissue at the maximum allowable curvature of the skin layer and the maximum allowable curvature of the liver layer.

6. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 4, characterized in that, In step S22, the method for constructing the dynamic blood vessel model is as follows: Let the initial time be , The position of a certain point on a blood vessel at any given time is ,but Represented as: ， In the formula It is a displacement vector. , The amplitude represents the maximum displacement of the blood vessel during the respiratory cycle. It's the respiratory rate. It is the initial phase. It is the unit vector of the direction of motion; The specific method for the layered dynamic collision detection is as follows: set up For at any time The minimum distance from the needle tip to the point cloud of the obstacle; if this minimum distance is less than the safe radius... If so, a collision is determined to have occurred.

7. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 4, characterized in that, In step S23, the RRT-guided planning mechanism for the entire process of DQN includes: calculating the direction vector of RRT-guided DQN and performing state enhancement splicing; designing a hybrid action selection strategy that integrates RRT guidance and DQN autonomous decision-making, and adopting a multi-objective fusion reward function to realize RRT guidance; and introducing an RRT guidance term in the temporal differential update to optimize the update direction of the value function.

8. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 4, characterized in that, In step S24, the multi-target fusion reward function includes distance reward, RRT guidance reward, target reward, collision penalty, and hierarchical compliance reward; wherein, the hierarchical compliance reward includes a basic curvature score that evaluates the curvature of a single path point and a layer boundary smoothness score that evaluates the curvature near the boundary.

9. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 1, characterized in that, The multi-dimensional path quality assessment system in step three specifically includes: S31. Construct normalized evaluation indicators, specifically including safety, smoothness, feasibility, efficiency, linearity, and hierarchical compliance. The value range of each evaluation indicator is [0,1], where: Design safety indicators based on safety radius Smoothness index designed based on maximum and average path curvature. Feasibility indicators Check if each line segment in the path is collision-free; design a straightness index based on the distance relationship between each point on the path and the path's start and end points. ; Based on the maximum curvature allowed by the layer where this point is located. Determine stratified compliance indicators ; S32: Overall Score The calculation uses a weighted summation formula: constants in the formula , , , , Using preset weights, the planned paths for multiple needle insertion points are sorted based on the comprehensive score, and the optimal puncture path with the highest comprehensive score is output. The planned puncture paths for multiple needle insertion points are sorted based on the comprehensive score, and the optimal puncture path with the highest comprehensive score is output.

10. The three-dimensional dynamic path planning method for ablation probes based on improved DQN according to claim 9, characterized in that, Candidate needle entry points for multiple needle entry points are automatically generated through the following steps: (1) Downsample the skin point cloud and control the number of point clouds within a certain range. Within points, This is a preset constant; (2) Calculate the Euclidean distance from each skin point to the tumor center, and filter them in ascending order of distance. The candidate set consists of 10 points. This is a preset constant; (3) Eliminate candidate points whose distance from the blood vessel is less than the safe threshold by obstacle collision filtering; (4) Cluster the remaining candidate points using K-means clustering. kind, As a preset constant, the point closest to the tumor center in each class is selected as the final candidate needle insertion point.

Images