A Multi-Flexible Needle Path Planning Method Based on Deep Reinforcement Learning

Abstract

This invention belongs to the field of flexible needle path planning technology, specifically disclosing a multi-flexible needle path planning method based on deep reinforcement learning. This invention addresses the problems of existing multi-slanted flexible needle path planning methods, such as the lack of unified modeling for multi-needle collaboration, tissue dynamic deformation, and the complexity of high-dimensional state space; the susceptibility to multi-needle path conflicts and the lack of reasonable task allocation and spatial coordination mechanisms; and poor adaptability to internal tissue structural disturbances. The invention includes: S1: Construction of a three-dimensional intelligent agent interaction environment and data preparation; S2: Obstacle modeling and deformation function design; S3: Three-dimensional kinematic model of the flexible needle and path generation; S4: Multi-agent collaborative control and path optimization; S5: Path evaluation and verification.

Description

Technical Field

[0001] This invention relates to a multi-flexible needle path planning method based on deep reinforcement learning, belonging to the field of flexible needle path planning technology. Background Technology

[0002] Multi-point flexible needle path planning technology is one of the core methods for achieving high-precision medical interventions. The goal of this technology is to plan safe and efficient paths from the initial position to the preset target point for multiple flexible needles with beveled tips. This technology is widely used in minimally invasive surgeries and interventional treatments such as tumor ablation and tissue biopsy. It not only reduces damage to normal tissues but also shortens postoperative recovery time, significantly improving clinical treatment outcomes and surgical safety.

[0003] Traditional methods for path planning with a single flexible needle mainly include geometric modeling and mechanical modeling. Geometric modeling constructs a spatial geometric model of the human tissue and the flexible needle, and uses mathematical methods to calculate the path; while mechanical modeling derives the deformation trajectory of the needle in the tissue based on the forces between the needle and the tissue. However, with the increasing complexity and precision requirements of surgical tasks, single-needle path planning methods are insufficient for complex scenarios involving coordinated puncture at multiple targets and through multiple channels. Therefore, multi-beveled flexible needle path planning technology has emerged. In existing research, some path planning methods are based on optimization algorithms, such as heuristic search, greedy algorithms, or graph-based path solving methods. Although these methods can plan multi-needle paths to some extent, they often have poor adaptability to complex human tissue structures and suffer from high computational load and sensitivity to initial parameters.

[0004] Meanwhile, the application of deep reinforcement learning methods in path planning has gradually emerged in recent years, especially path planning algorithms based on Deep Q-Network (DQN) and its variants, which have achieved good results in multiple robot navigation scenarios. Related research has also preliminarily introduced these methods into the field of flexible needle path planning, and achieved some results. However, most of these methods still focus on single agent or single needle path generation, lacking in-depth research on multi-needle cooperative control mechanisms.

[0005] Current methods for planning the path of multi-point flexible needles based on deep reinforcement learning still face many challenges, mainly in the lack of collaborative modeling and comprehensive consideration of multiple factors. A systematic solution that can simultaneously take into account multi-needle collaborative planning, agent information sharing, and dynamic changes in complex surgical environments has not yet been formed.

[0006] Specifically, in terms of multi-needle collaboration, existing methods mostly focus on single-needle path planning or prediction, lacking an effective framework to support parallel planning of multiple needles and collaborative operation of targets, making it difficult to meet the clinical needs of multi-channel and multi-target operations in complex surgeries. In terms of agent interaction, the information sharing mechanism between multiple agents is poorly designed under traditional methods or DQN algorithms, resulting in low training efficiency and poor policy learning stability in large-scale state-action spaces. In terms of environment modeling, traditional path planning methods often simplify the dynamic interaction process between flexible needles and biological tissues, failing to fully consider factors such as tissue deformation and internal structural disturbances generated during needle advancement, resulting in insufficient feasibility and robustness of the generated paths in real surgeries. Summary of the Invention

[0007] To address the problems in existing multi-point flexible needle path planning methods, such as the lack of unified modeling of multi-needle collaboration, tissue dynamic deformation and high-dimensional state space complexity, the tendency for multi-needle paths to conflict and the lack of reasonable task allocation and spatial coordination mechanisms, as well as poor adaptability to internal tissue structural disturbances, this invention proposes a multi-flexible needle path planning method based on deep reinforcement learning.

[0008] The technical solution adopted by the present invention to solve the above problems is as follows: The present invention includes the following steps:

[0009] Step 1: Read liver and tumor tissue data from medical imaging data and construct a three-dimensional intelligent interactive environment including a set of available needle insertion points, tumor region, liver region, and obstacle region;

[0010] Step 2: Randomly generate obstacles in the constructed 3D intelligent agent interactive environment using the elastic deformation method, and set the deformation behavior, recovery mechanism, and obstacle position update mechanism when the obstacles are subjected to external forces.

[0011] Step 3: Establish a three-dimensional kinematic model of the flexible needle using the circular arc motion calculation function, set the arc radius and center direction angle of the intelligent agent, construct a rotational motion path around the center of the intelligent agent based on the arc radius and center direction angle, and introduce nonholonomic motion constraints and the interaction mechanism between the intelligent agent and the voxel discrete environment. When encountering one of the obstacles or the boundary of the three-dimensional intelligent agent interaction environment, the path is judged and adjusted in real time.

[0012] Step 4: Use the multi-agent proximal policy optimization algorithm to perform multi-flexible needle two-stage path cooperative planning, and obtain several feasible multi-flexible needle two-stage path cooperative planning results.

[0013] Step 5: Establish the needle entry point scoring function, path evaluation function, and endpoint ablation area coverage evaluation function. Based on the needle entry point scoring function, path evaluation function, and endpoint ablation area coverage evaluation function, screen the feasible multi-flexible needle two-stage path collaborative planning results and obtain the optimal multi-flexible needle two-stage path collaborative planning result.

[0014] Furthermore, step 1 specifically includes:

[0015] Based on category information, medical image regions are labeled as tumor regions, liver regions, and obstacle regions;

[0016] For the tumor region, obtain the voxel set of the tumor region. Set the number of agents to be N K-means clustering algorithm was used to analyze the voxel set in the tumor region. Divide into N There are 10 clusters, and the centroid of each cluster is used as the target endpoint of the corresponding agent.

[0017] For the liver region, obtain the voxel set of the liver region. According to voxel sets Define the set of surface points of the liver region Set of surface points in the liver region In the middle, an equal spacing strategy is used to generate K A number of candidate needle insertion points were selected, and the puncture path length threshold parameter was used. Directional deviation threshold parameter Surface curvature threshold parameter and safe distance threshold parameters Set constraints and select the set of points that satisfy the constraints from the candidate needle entry points as the needle entry point set;

[0018] The obstacle is simplified by treating it as a homogeneous solid sphere and focusing only on the deformation of the part of the needle body near the flexible needle.

[0019] Furthermore, step 2 specifically includes:

[0020] Step 2.1: Model the obstacle as a solid sphere using the elastic deformation method, and set the initial position of the obstacle. and radius ;

[0021] Step 2.2: Set the tissue elasticity coefficient and radius of influence Based on the distance between the agent and the obstacle Organizational elasticity coefficient and radius of influence Set the deformation intensity of the obstacle When the distance between the agent and the obstacle smaller than the radius of influence At that time, the obstacle deforms;

[0022] Step 2.3: Based on the agent's advance distance L Maximum tolerance distance between the agent and the obstacle Calculate the deformation intensity of obstacles during the propulsion of the intelligent agent. Based on the initial position of the obstacle Radius of influence The unit vector pointing from the agent to the obstacle. Deformation intensity of obstacles Deformation intensity of obstacles during the propulsion of the intelligent agent Update the obstacle positions;

[0023] Step 2.4: Based on the deformation intensity of the obstacle Deformation intensity of obstacles during the propulsion of the intelligent agent Calculate the overall deformation of the obstacle and adjust its radius in real time based on this deformation. This adjustment is applied when the distance between the agent and the obstacle changes. Larger than the radius of influence At that time, the obstacle returns to its initial state.

[0024] Furthermore, step 3 specifically includes:

[0025] Step 3.1: Select a reference vector that is not collinear with the agent's current movement direction, construct a local orthogonal basis, and perform cross product calculation based on the constructed local orthogonal basis to obtain the spatial position of the circle center;

[0026] Step 3.2: Set the current position of the agent Direction vector radius of arc and the direction angle of the center According to the set arc radius And the distance of intelligent agent advancement L Calculate the rotation angle ;

[0027] Step 3.3: Determine the rotation axis based on the cross product of the current direction vector and the center vector. Based on the rotation axis Construct a three-dimensional rotation matrix;

[0028] Step 3.4: Update the agent's current position and direction of motion based on the 3D rotation matrix to obtain the agent's new position vector and direction of motion;

[0029] Step 3.5: Discretize the rotation angle according to a fixed number of steps to generate a sequence of intermediate path points to represent the arc trajectory in three-dimensional space;

[0030] Step 3.6: After each path is generated, the generated path points are interactively detected with the 3D intelligent agent's interactive environment. If a path point overlaps with an obstacle object or boundary, it is considered a collision. After a collision, a path adjustment mechanism is triggered, and the path is adjusted in real time by replanning the motion direction or adjusting the arc parameters.

[0031] Furthermore, step 4 specifically includes:

[0032] Step 4.1: Assign each agent a flexible needle and set a comprehensive reward function. The comprehensive reward function includes: positive reward for each step when the needle tip is closer to the tumor center during the agent's movement along the generated path, encouraging the path to approach the target endpoint; introducing a step length penalty term to control the bending frequency; if the path collides or exceeds an obstacle, a strong penalty is applied; if the target endpoint is successfully reached without collision, the reward is weighted and accumulated with the remaining steps.

[0033] Step 4.2: Construct an independent policy network and value network for each flexible needle, and share environmental state information and comprehensive reward function during the training phase;

[0034] Step 4.3: Introduce a spatial safety constraint mechanism among multiple agents, including but not limited to minimum spacing constraints, directional angle limits, and dynamic priority discrimination, to ensure that the flexible needle avoids trajectory intersections, interference, and structural collisions during insertion;

[0035] Step 4.4: Divide the multi-flexible needle dual-stage path collaborative planning into needle entry point selection and path planning stages. In the needle entry point stage, the agent selects the best position from the set of needle entry points.

[0036] Step 4.5: In the path planning stage, a conflict detection and resolution mechanism is constructed. During the advancement of the flexible needle path, the spatial overlap, intersection or close proximity between the flexible needle path and other agent paths is monitored in real time. If a conflict occurs with other agents, the current needle entry point position selection is trapped in a local optimum, triggering the conflict resolution mechanism to reset the needle entry point selection, allowing the agent to learn new rewards, escape the local optimum, and output feasible multi-flexible needle two-stage path collaborative planning results.

[0037] Furthermore, step 5 specifically includes:

[0038] Step 5.1: Establish the needle insertion point scoring function, path evaluation function, and endpoint ablation area coverage evaluation function;

[0039] Step 5.2: Based on the needle entry point scoring function, comprehensively consider the proximity, directional consistency, incident curvature and safe distance from obstacles of all needle entry points, and sort and select the best of all needle entry points by normalized weighted scores;

[0040] Step 5.3: By comprehensively considering three indicators—path length, minimum safe distance to obstacles, and minimum distance between paths of multiple agents—the geometric merits of each feasible multi-flexible needle two-stage path collaborative planning are quantified through the path point scoring function. Feasible paths are then ranked and optimized through normalized weighted scores.

[0041] Step 5.4: Establishing a needle tip ellipsoidal ablation zone based on a temperature conduction model The total ablation area is calculated, and the effective coverage rate and over-ablation and uncovered areas are evaluated through the endpoint ablation area coverage evaluation function. The optimal multi-flexible needle two-stage path collaborative planning result is output.

[0042] The beneficial effects of this invention are:

[0043] 1. This invention constructs a multi-agent deep reinforcement learning framework based on MAPPO, achieving collaborative path planning for multiple flexible needles and uniformly solving challenges such as multi-needle coordination, tissue dynamic deformation, and high-dimensional state space modeling. This invention significantly improves system stability and collaborative efficiency, providing reliable intelligent decision-making capabilities for complex puncture tasks.

[0044] 2. In terms of environmental modeling, this invention introduces an obstacle deformation modeling module to effectively simulate the interactive deformation behavior between flexible needles and soft tissue, improving the realism of environmental simulation and thus training a strategy with stronger generalization ability. Simultaneously, by dividing path planning into two stages—needle entry point selection and path generation—the adaptability of the agent to complex environments is enhanced, improving the success rate of path planning and the overall path quality.

[0045] 3. This invention constructs a spatial safety constraint and conflict detection mechanism between multiple needles to avoid path intersection interference and ensure the safety of the puncture process. By setting multiple evenly distributed endpoints in the target area, uniform coverage of the tumor area is achieved, improving treatment efficacy. The feasibility of the path is ensured by combining a nonholonomic motion model and a voxel interaction mechanism, and a systematic evaluation method is provided, verifying the reliability and effectiveness of this method in practical applications.

[0046] 4. This invention divides the path planning process into an insertion point selection stage and a path generation stage. The insertion point selection stage enables each agent to autonomously and flexibly determine the optimal insertion position based on the target location and tissue environment, improving overall path quality and operational feasibility. The path generation stage, based on the determined insertion point, uses a policy network trained with the MAPPO algorithm to guide each needle in making dynamic decisions within the tissue environment, ultimately planning a collision-free, low-damage puncture path. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a multi-flexible needle path planning method based on deep reinforcement learning.

[0048] Figure 2 This is a schematic diagram showing the deformation effect of the obstacle deformation function and its comparison with finite element simulation.

[0049] Figure 3 The deformation effect of the obstacle deformation function and the comparison curve with the finite element simulation are shown.

[0050] Figure 4 This is a schematic diagram of the trajectory of a three-dimensional kinematic model of a flexible needle.

[0051] Figure 5 This is a schematic diagram of the multi-agent training environment, obstacle settings, alternative starting point settings, and dual-needle planning path constructed based on NII medical images.

[0052] Figure 6 To compare the path planning lengths of the present invention with those of the traditional two-needle and three-needle methods in a simulation experiment with the same six obstacles, box plots were used. Figure 6 In this context, Path Length is the path length, Random-2 is a 2-needle path plan without entry point optimization, Entry Optimized-2 is a 2-needle path plan with two-stage entry point optimization, Random-3 is a 3-needle path plan without entry point optimization, and Entry Optimized-3 is a 3-needle path plan with two-stage entry point optimization.

[0053] Figure 7 This diagram illustrates the importance assessment results of the multi-needle collaboration proposed in this invention and the traditional non-collaboration approach in 2-needle, 3-needle, and 4-needle scenarios. Figure 7 In the context of the code, "No Collab" means no collaboration, "With Collab" means collaboration, "Scuess Rate" means success rate, "2Needles" means 2-needle scenario, "3Needles" means 3-needle scenario, and "4Needles" means 4-needle scenario. Detailed Implementation

[0054] Combination Figures 1-7This implementation method is described as follows: Figure 1 As shown, the steps of the multi-flexible needle path planning method based on deep reinforcement learning described in this embodiment include:

[0055] S1: Construction of a 3D intelligent agent interaction environment and data preparation;

[0056] This implementation aims to construct a three-dimensional surgical environment for multi-agent flexible needle path planning and prepare target point and needle entry point data. This step specifically includes:

[0057] S101: Information on liver and tumor tissue is read from medical imaging data. This implementation uses NII format medical imaging data as an example and employs an image processing tool, specifically Nibabel, to load and segment the medical images. After segmentation, the image voxels are labeled by category: liver regions are set to 1, tumor regions to 2, and regions containing obstacles such as blood vessels to 0. The expression for the voxel set of the tumor region is:

[0058] (1);

[0059] To achieve uniform and coordinated puncture within the tumor by multiple agents, this implementation assigns a spatially distributed target endpoint to each agent, and sets the number of agents to be [number missing]. N Voxel clustering in tumor regions was performed using the K-means clustering algorithm. Divide into N Clusters And the centroid of each cluster is used as the target endpoint of the intelligent agent. The objective function of K-means is as follows:

[0060] (2);

[0061] In formula (2), The coordinates of the centroid of the cluster are given. .

[0062] Through the above processing, the endpoints of each intelligent agent can be evenly distributed within the tumor volume, thereby improving the spatial coverage of the tumor area and the integrity of the ablation treatment.

[0063] S102: To achieve a reasonable puncture path planning from the liver surface to the target point, this embodiment constructs a set of feasible candidate needle entry points, assuming the liver region voxel set is... Then its surface point set is defined as:

[0064] (3);

[0065] From surface point concentration In the middle, an equal spacing strategy is used to generate K A number of representative candidate needle insertion points were selected as the basis for screening. To ensure the feasibility of the approach and the safety of the surgery, [the following was done]. The following constraints are selected to form the final set of usable needle insertion points. :

[0066] (4);

[0067] In formula (4), , For the first i The target point of each agent. For point The unit normal vector, For the local surface curvature of the modified point, Given a known set of obstacle objects, , , and These are the parameters for puncture path length, directional deviation, surface curvature, and safe distance threshold, respectively.

[0068] Simplify the settings for obstacles such as blood vessels: treat them as homogeneous solid spheres and only focus on the deformation of the part near the needle.

[0069] S2: Obstacle modeling and deformation function design;

[0070] In this step, obstacles are modeled using elastic deformation theory to simulate the interaction between the flexible needle and the tissue. Each obstacle is modeled as a spherical solid with an initial position. and radius The distance between the flexible needle and the obstacle smaller than the radius of influence At that time, the obstacle deforms. Deformation intensity. It is inversely proportional to the distance between the needle and the obstacle, expressed as The effect of deformation is determined by the propulsion distance. The constraints and calculation formula are as follows: The obstacle's position is updated as follows: .in, It is a unit vector pointing from the needle to the obstacle. The radius of the obstacle is adjusted according to the degree of deformation. When the distance exceeds the radius of influence, the obstacle returns to its initial state.

[0071] Furthermore, after completing the above settings, this embodiment used finite element method to compare the effect of the simulated deformation algorithm. The deformation effect of the obstacle deformation function and the comparison with finite element simulation are shown in the figure. Figure 2 As shown; the deformation effect of the obstacle deformation function and the comparison curve with the finite element simulation are as follows.Figure 3 As shown, the test results show that the direction cosine similarity is 0.9589 (approaching 1, indicating a high degree of consistency in direction); the Hausdorff distance is 0.2040 (reflecting a small maximum local deviation); the Frechet distance is 0.2874 (indicating a high similarity in trajectory shape); and the average distance error is only 0.1018 (reflecting excellent overall consistency).

[0072] In terms of environmental modeling, this implementation introduces an obstacle deformation modeling module, which effectively simulates the interactive deformation behavior between flexible needles and soft tissue, improving the realism of environmental simulation and thus training a strategy with stronger generalization ability.

[0073] S3: Flexible needle 3D kinematics model and path generation;

[0074] In this step, a three-dimensional kinematic model of a flexible needle is used to plan the agent's path. The movement of each agent is determined by its current position. Direction vector radius of arc Angle with the center of the circle Definition. Where the radius of the arc... The curvature of the trajectory is determined by the angle of the center of the circle. Once the direction of the arc curvature is determined and the parameters are finalized, the following path generation steps are performed:

[0075] S301: Select a reference vector that is not collinear with the current direction. (when Time to take Otherwise take Construct a local orthogonal basis. Calculate using the cross product. and Thus, the position of the center of the circle is determined. .

[0076] S302: Based on the propulsion distance Calculate the rotation angle The rotation axis is determined by the cross product of the current direction and the center vector. ;

[0077] S303: Based on a rotating axis Constructing a 3D rotation matrix:

[0078] (5);

[0079] S304: Apply the rotation matrix to update position and orientation: and .

[0080] S305: The rotation angle is discretized into multiple steps to generate a sequence of path points. Each path point is calculated using a rotation matrix corresponding to the angle. The path points interact with obstacles in the environment to ensure the feasibility and continuity of the path. If necessary, the arc parameters are adjusted or the direction is replanned. The final trajectory of the flexible needle's three-dimensional kinematic model is as follows: Figure 4 As shown.

[0081] S4: Multi-agent cooperative control and path optimization;

[0082] To achieve collaborative insertion path planning for multiple flexible needles, this implementation proposes a planning framework based on multi-agent reinforcement learning. Each agent corresponds to one flexible needle, and its action space consists of the starting position selection and the three-dimensional kinematic model parameters of the flexible needle at each step during insertion, specifically represented as follows:

[0083] (6);

[0084] In formula (6), For the first The starting point of the needle, The arc length and deflection angle of each insertion path are used to approximate the actual movement trajectory of the flexible needle. The set consisting of ternary real arrays is , For the number of intelligent agents, The total number of steps in the path for each agent.

[0085] Each agent's state vector at each step includes the current position of the needle tip. Directional unit vector Location of the tumor center And whether it is the initial stage flag. There are a total of 10 dimensions. The corresponding actions are represented by triples. constitute, Indicates the starting point number. Indicates the insertion step size. This indicates a circular arc corner.

[0086] To guide strategy learning, this implementation method designs a comprehensive reward function. Each step reduces the distance between the needle tip and the tumor center, resulting in a positive reward to encourage the path to approach the target. Simultaneously, a step length penalty term is introduced to control the bending frequency and avoid path oscillations. If the path collides or traverses an infeasible region, a strong penalty is applied; conversely, if the target is successfully hit without collision, the reward is weighted and accumulated with the remaining steps. Through this modeling, each agent can autonomously learn how to collaboratively select the insertion starting point, adjust the insertion direction and step length, to achieve efficient and safe multi-needle path planning in complex anatomical structures.

[0087] To achieve collaborative policy learning among multiple flexible needles, this implementation introduces a multi-agent reinforcement learning paradigm with centralized training and distributed execution, and employs the MAPPO algorithm to optimize the policies of each agent. Let each agent... Having local strategies ,in For its local observation, For its actions, policy parameters Information coupling across agents is achieved through sharing or partial sharing during training.

[0088] Within the MAPPO framework, this implementation uses centralized value functions. or joint state action value function ,in Policy updates stabilize the training process using a clipped surrogate objective.

[0089] (7);

[0090] In formula (7), The probability ratio of the strategies. This is a generalized advantage estimate, used to measure the superiority or inferiority of the current action relative to the baseline. This is the pruning factor, used to limit the magnitude of policy updates.

[0091] This implementation divides the multi-flexible needle dual-stage path collaborative planning into needle entry point selection and path planning stages. In the needle entry point stage, the agent selects the optimal position from the set of needle entry points. In the path planning stage, a conflict detection and resolution mechanism is constructed to monitor the spatial overlap, intersection, or close proximity between the flexible needle path and other agent paths in real time during the advancement of the flexible needle path. If a conflict occurs with other agents, the current needle entry point position selection falls into a local optimum, triggering the conflict resolution mechanism to reset the needle entry point selection, allowing the agent to learn new rewards and escape the local optimum.

[0092] To enhance the coordination of multi-needle paths, this implementation method employs an attention mechanism or a shared network structure in centralized training to achieve cross-agent information fusion, such as using a state encoder. And through attention mechanisms, it aggregates the influence of others' strategic behaviors, which can be formalized as follows:

[0093] (8);

[0094] In formula (8), To aggregate information, The encoded state of agent i;

[0095] Ultimately, the information will be aggregated. With one's own state The concatenated parts are used for motion generation: .

[0096] Through the above mechanism, MAPPO can effectively coordinate the spatial conflicts and target distribution of multiple flexible needles in path selection, and achieve the overall optimal insertion strategy.

[0097] This invention enhances the agent's adaptability to complex environments and improves the success rate and overall path quality by dividing path planning into two stages: entry point selection and path generation.

[0098] S5: Path evaluation and verification;

[0099] In this step, this embodiment systematically evaluates the multi-flexible needle path planning system from three aspects: needle entry point quality, path geometry characteristics, and endpoint ablation effect.

[0100] S501: Set of needle entry points satisfying hard constraints This embodiment further defines a scoring function to measure each needle insertion point. Advantages and disadvantages:

[0101] (9);

[0102] In formula (9), , , , These are the constant coefficients for each term.

[0103] Path evaluation function Taking into account factors such as proximity of the needle entry point, consistency of direction, incident curvature, and safe distance from obstacles, after normalization... We perform a weighted summation between them to facilitate sorting and selection.

[0104] S502: At the path level, set the path For the first The three-dimensional path of the flexible needle, in which This represents a parameterized point on the path. In this implementation, the path evaluation function is written in the following form:

[0105] (10);

[0106] Path evaluation function The performance of the flexible needle path is evaluated by comprehensively considering three factors: path length, minimum safe distance from obstacles, and minimum distance between paths of multiple agents. The first term in the function... This is used to penalize path length; the shorter the path, the higher the score. For path length, The maximum length among all feasible paths, used for normalization. The second term... The minimum distance between the path and obstacles is measured; the greater the minimum distance, the higher the score. (Third item) The minimum distance between paths is used to evaluate the degree of mutual perturbation between multiple needles; the farther the path, the higher the score. Each of the three items is multiplied by a weight. , and The final score is normalized to the [0,1] interval, which facilitates path sorting and comparison.

[0107] S503: In terms of ablation effect evaluation, the ablation area formed by the endpoints of each needle can be approximated as an ellipsoid. Its center is the insertion endpoint, its major axis is aligned with the needle tip direction, and its semi-axis length is determined by the temperature conduction model. The total ablation area is then... The target area is The following endpoint ablation region coverage evaluation function is introduced:

[0108] (11);

[0109] In formula (11), For the collection of all voxels within the tumor target area, This is an indicator function. The evaluation function represents the proportion of the target area effectively ablated. Further supplementary indicators such as over-ablation ratio and non-coverage rate can be defined to assess whether there are significant over-ablation or missed areas.

[0110] After filtering through the above evaluation functions, the multi-agent training environment based on NII medical images, obstacle settings, alternative starting point settings, and dual-needle planning path are as follows: Figure 5 As shown, by Figure 5 It can be seen that by combining the nonholonomic motion model and the voxel interaction mechanism to ensure the feasibility of the path and providing a systematic evaluation method, the reliability and effectiveness of the present invention in practical applications have been verified. Through the above three-dimensional evaluation system of needle insertion point, path geometry and ablation prediction, the overall quality of the multi-agent flexible needle insertion path can be comprehensively and systematically judged, providing a solid foundation for the optimization and strategy comparison of multi-flexible needle path planning.

[0111] The proposed method for multi-flexible needle path planning based on deep reinforcement learning offers several advantages in terms of feasibility. Compared to traditional geometric modeling, mechanical modeling, and optimization-based path planning methods, traditional methods are often designed for single-needle scenarios and struggle to adapt to the complex surgical environment of multi-needle collaboration. They are prone to path intersections and collisions, and cannot handle dynamic tissue deformation when modeling the static environment. In contrast, this invention achieves two-stage collaborative planning through the MAPPO algorithm, combined with spatial safety constraints and conflict detection mechanisms, and introduces nonholonomic motion constraints and voxel interaction mechanisms. This allows for the generation of executable puncture paths even in multi-target and dynamically deformed environments. Regarding robustness, traditional methods rely on simplification assumptions, are sensitive to initial parameters, and cannot dynamically adapt to tissue deformation. The DQN algorithm, due to its imperfect multi-agent information sharing mechanism, is prone to policy oscillations during training and generalizes poorly to dynamic environments. While traditional methods have limited capabilities, this invention utilizes a dynamic deformation model of obstacles to respond in real time to the interaction between tissue and needle. The MAPPO algorithm, combined with a shear probability ratio-stabilized training process, exhibits higher tolerance for fluctuations in initial parameters and dynamic environmental changes. In terms of accuracy, traditional methods suffer from significant deviations between the path and target due to model simplification, making it difficult to ensure uniform coverage of the tumor region across multiple needles. The DQN algorithm, lacking a collaborative coverage mechanism in multi-needle scenarios, suffers from uneven target distribution, and path generation relies on discrete motion space, reducing positional accuracy. In contrast, this invention uses K-means clustering to ensure uniform distribution of multi-needle target points, and combines an endpoint ablation region coverage evaluation function to improve puncture endpoint accuracy. The three-dimensional rotation matrix and discrete path point generation mechanism ensure continuous and smooth path trajectories, further reducing the positional deviation between the needle tip and target. Therefore, this invention outperforms traditional methods in terms of feasibility, robustness, and accuracy.

[0112] To further verify the advantages of this invention in multi-flexible needle path planning, this embodiment set up the same six obstacles in the simulation experiment. Under this simulation environment, the path planning lengths of two-needle and three-needle methods with and without needle entry point optimization were compared. Figure 6 As shown, regarding the two-stage path planning (including needle entry point optimization) and path planning without needle entry point optimization proposed in this invention, through the following... Figure 6The box plots shown visually illustrate the differences: taking two-needle and three-needle path planning with and without needle entry point optimization as examples, in the Random group without needle entry point optimization, the path length of each two-needle path is concentrated in the 300-500 range, and the path length of each three-needle path is concentrated in the 200-600 range, with a large overall dispersion and high mean in path length. In contrast, in the Entry Optimized group with dual-stage needle entry point optimization, the path length of each two-needle path is concentrated in the 100-200 range, and the path length of each three-needle path is concentrated in the 100-200 range, with significantly shorter path lengths and a more concentrated distribution. This indicates that dual-stage path planning, through needle entry point optimization, can effectively constrain the path search space, reduce the length of flexible needle puncture paths, and reduce the complexity of needle-tissue interaction. While improving the feasibility of path planning and avoiding tissue damage and path conflicts caused by long paths, the more regular paths enhance robustness and stability in the face of tissue deformation and disturbance. This also supports the reduction of cumulative errors caused by shorter paths, further demonstrating the advantages of this invention in multi-flexible needle path planning.

[0113] Furthermore, this invention also evaluated the importance of traditional non-cooperative methods and the multi-needle cooperative method proposed in this invention in 2-needle, 3-needle, and 4-needle scenarios. The comparison results of the importance evaluation are as follows: Figure 7 As shown, without coordination, the success rate drops sharply with increasing needle count. Particularly in the four-needle case, the success rate falls to only 23%. When the coordination mechanism proposed in this invention is introduced, the system's performance is much more reliable. The success rate is significantly improved in all cases, and the negative impact of increasing the number of needles is greatly reduced. From Figure 7 As can be clearly seen, in the "No Collab." mode, as the number of puncture needles increases from 2 to 4, the surgical success rate drops sharply from 58.1% to 23.45%, showing a clear trend of rapid performance deterioration with increasing needle count. However, after enabling the "With Collab." mechanism proposed in this invention, the success rate remains at a high level for puncture scenarios with 2, 3, and 4 needles, reaching 93.3%, 90.12%, and 88.7% respectively. Compared to the "No Collab." mode, the success rate is significantly improved, and the negative impact of increasing the number of needles on the success rate is greatly reduced. This experimental result strongly verifies the superiority of the multi-agent reinforcement learning collaborative mechanism adopted in this invention in multi-needle puncture scenarios. Through information sharing and collaborative decision-making among multiple agents, it effectively avoids path conflicts caused by lack of coordination in traditional methods during multi-needle parallel punctures, significantly improving the overall performance of multi-flexible needle path planning. This provides a more reliable and efficient technical guarantee for complex multi-needle collaborative puncture surgery, further highlighting the advanced nature and practical value of the technical solution of this invention.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for multi-flexible needle path planning based on deep reinforcement learning, characterized in that, The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. Step 2.1: Model the obstacle as a solid sphere by the elastic deformation method, set the initial position of the obstacle and radius ; Step 2.2: Set the tissue elasticity coefficient and the influence radius , according to the distance between the agent and the obstacle , the tissue elasticity coefficient and the influence radius Set the deformation strength of the obstacle When the distance between the agent and the obstacle is less than the influence radius , the obstacle deforms; Step 2.3: Update the position of the obstacle according to the pushing distance of the agent L , the maximum tolerance distance between the agent and the obstacle Calculate the deformation strength of the obstacle during the agent's pushing process , according to the initial position of the obstacle , the influence radius , the unit vector of the agent pointing to the obstacle , the deformation strength of the obstacle and the deformation strength of the obstacle during the agent's pushing process Update the position of the obstacle; Step 2.4: Deformation intensity of the obstacle according to the obstacle shape Deformation intensity of the obstacle during the agent advancing process Calculate the overall deformation degree of the obstacle, and adjust the radius of the obstacle in real time according to the overall deformation degree of the obstacle; when the distance between the agent and the obstacle is greater than the influence radius , the obstacle returns to the initial state; The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization.

2. The method of claim 1, wherein, The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. For the tumor region, a voxel set of the tumor region is acquired , the number of intelligent agents is set to N , a K-means clustering algorithm is used to divide clusters in the voxel set of the tumor region N , and the centroid of each cluster is taken as the target endpoint of the corresponding intelligent agent; For the liver region, obtain the voxel set of the liver region. According to voxel sets Define the set of surface points of the liver region Set of surface points in the liver region In the middle, an equal spacing strategy is used to generate K A number of candidate needle insertion points were selected, and the puncture path length threshold parameter was used. Directional deviation threshold parameter Surface curvature threshold parameter and safe distance threshold parameters Set constraints and select the set of points that satisfy the constraints from the candidate needle entry points as the needle entry point set; The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization.

3. The method of claim 1, wherein, The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. Step 3.2: Set the current position of the agent , a direction vector , a circular arc radius , and a center direction angle , calculate a rotation angle according to the set circular arc radius and the agent's propulsion distance L ; Step 3.3: Determine the rotation axis based on the cross product of the current direction vector and the circle center vector , construct a three-dimensional rotation matrix based on the rotation axis ; The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization.

4. The method of claim 1, wherein, The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy optimization. The application relates to a multi-flexible-needle double-stage path collaborative planning method based on multi-agent near-end strategy Step 4.3: Introduce a spatial safety constraint mechanism among multiple agents to ensure that the flexible needles avoid trajectory intersection, interference and structural collision during the insertion process; Step 4.4: Divide the multi-flexible needle two-stage path collaborative planning into an entry point selection stage and a path planning stage. In the entry point stage, the agent selects the optimal position from the set of entry points; Step 4.5: In the path planning stage, a conflict detection and resolution mechanism is constructed, and a feasible multi-flexible needle two-stage path collaborative planning result is output.

5. The method of claim 1, wherein, Step 5 specifically includes: Step 5.1: Establish an entry point scoring function, a path evaluation function, and an endpoint ablation region coverage evaluation function; Step 5.2: Based on the entry point scoring function, consider the proximity, direction consistency, incident curvature and safety distance from obstacles of all entry points, and sort and optimize all entry points through normalized weighted scoring; Step 5.3: Through the path point scoring function, consider the path length, the minimum safety distance from obstacles and the minimum distance between multi-agent paths, quantify the geometric advantages and disadvantages of each feasible multi-flexible needle two-stage path collaborative planning, and sort and optimize the feasible paths through normalized weighted scoring; Step 5.4: Calculate the total ablation area by establishing a temperature conduction model-based needle tip ellipsoid ablation zone , calculate the total ablation area, and evaluate the effective coverage rate and over-ablation and uncovered areas by the endpoint ablation area coverage evaluation function, and output the optimal multi-flexible needle two-stage path collaborative planning result.

Images