A puncture path planning method based on improved SAC algorithm

Abstract

The present invention discloses a puncture path planning method based on an improved SAC algorithm, comprising the following steps: obtaining a three-dimensional organ model, uniformly sampling and voxelizing the three-dimensional organ model to obtain a reconstructed three-dimensional model; providing a training environment for the SAC algorithm based on the reconstructed three-dimensional model, and planning the puncture path based on the trained SAC algorithm; during the planning process, providing geometric constraints for the puncture path, and performing collision detection on the puncture path based on a fine-grained collision detection method combining an enclosing shape method and a KD tree; finally, optimizing the puncture path using an angle reward function and a state reward function to obtain an optimized puncture path. By improving the SAC algorithm, the present invention can gradually optimize the puncture path in a large number of simulation trainings, so that the path is more accurate and can avoid obstacles and accurately reach the target area.

Description

Technical Field

[0001] The present invention belongs to the technical field of puncture path planning, and in particular relates to a puncture path planning method based on an improved SAC algorithm. Background Art

[0002] The planning of puncture paths can affect the examination of multiple organs in the body. At present, medical puncture path planning algorithms are mainly divided into two categories: traditional search methods and deep methods based on reinforcement learning.

[0003] Traditional methods take into account the physical constraints of the needle and the interaction between the needle and soft tissue. For example, Petruska et al. introduced physical constraints to track the trajectory of a highly curved magnetic-tipped steerable needle without damaging the tissue in the article "Magnetic needle guidance for neurosurgery: initial design and proof of concept". Hamze et al. used the haystack (HST) algorithm in the article "Preoperative trajectory planning for percutaneous procedures indeformable environments" to solve effective preoperative curved trajectories in tissue deformable environments. Li et al. proposed a discrete potential field algorithm based on three-dimensional anatomical structure in the article "Modeling of path planning and needle steering with path tracking in anatomical soft tissues for minimally invasive surgery" for needle path planning in minimally invasive surgery. The above methods focus on the flexible needle trajectory constraints and generation speed, but ignore the distance between the puncture path and dangerous human organs.

[0004] Reinforcement learning algorithms are widely used in complex and unknown environments to make decisions by assigning different reward values ​​to different states. For example, Kong et al. (2021) proposed a path planning algorithm based on hierarchical reinforcement learning and artificial potential fields in the paper "Constrained policy optimization algorithm for autonomous driving via reinforcement learning", which greatly improved the convergence speed and learning efficiency of the algorithm. However, traditional reinforcement learning path planning is highly dependent on map accuracy. In surgical planning scenarios, the environmental state of human organs is complex and prone to dimensionality disasters, resulting in slow convergence of reinforcement learning networks.

[0005] Therefore, it is urgent to propose a puncture path planning method based on the improved SAC algorithm. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a puncture path planning method based on an improved SAC algorithm to solve the problems existing in the above prior art.

[0007] To achieve the above object, the present invention provides a puncture path planning method based on an improved SAC algorithm, comprising the following steps:

[0008] Acquire a three-dimensional organ model, perform uniform sampling and voxelization processing on the three-dimensional organ model, and obtain a reconstructed three-dimensional model;

[0009] Providing a training environment for the SAC algorithm based on the reconstructed 3D model, and planning the puncture path based on the trained SAC algorithm;

[0010] During the planning process, geometric constraints are provided for the puncture path, and collision detection is performed on the puncture path based on a fine-grained collision detection method combining the enclosing shape method and the KD tree;

[0011] Finally, the angle reward function and the state reward function are used to optimize the puncture path to obtain the optimized puncture path.

[0012] Optionally, the reconstructed three-dimensional model includes: a puncture target area, impenetrable obstacles and penetrable obstacles.

[0013] Optionally, the process of providing geometric constraints for the puncture path includes: obtaining the current position and target position of the puncture needle, and calculating the direction vector of the puncture needle based on the functional relationship between the current position and the target position; based on the direction vector, calculating the pitch angle and yaw angle of the target direction, and updating the direction vector of the puncture needle based on the pitch angle and yaw angle; presetting the moving distance of the puncture needle, and calculating the new position of the puncture needle based on the updated direction vector and moving distance.

[0014] Optionally, the process of performing collision detection on the puncture path includes: determining the radius of a sphere based on half of the maximum moving distance of the puncture needle, and decomposing the puncture needle into a number of spheres based on the sphere radius; inserting the position data of all obstacles in the training environment into a KD tree to construct a spatial index; and then performing fine-grained collision detection on each sphere of the puncture needle based on the KD tree.

[0015] Optionally, the angle reward function is as follows:

[0016]

[0017] Among them, reward angleis the angle reward function, d new ·d target is the dot product of two vectors, ||d new || and ||d target || is the norm of the vector and λ is the coefficient.

[0018] Optionally, the state reward function is as follows:

[0019]

[0020] Among them, r is the reward function, and the variable s t and a t Represent the state and action at time t, respectively. Action a t After that, the state s t Transform to state s at time t+1 t+1 , Set goal is the target area, Set bone For the bone area, Set spleen For the spleen area, Set artery For the arterial region, Set nerve For nerve and vein areas, S target is the target area state, α is the distance penalty coefficient, reward penalty Penalty for algorithm steps.

[0021] The present invention also provides a puncture path planning system based on an improved SAC algorithm, comprising: a model reconstruction module, a path planning module and a path optimization module connected in sequence;

[0022] The model reconstruction module is used to obtain a three-dimensional organ model, and perform uniform sampling and voxelization processing on the three-dimensional organ model to obtain a reconstructed three-dimensional model;

[0023] The path planning module is used to provide a training environment for the SAC algorithm based on the reconstructed three-dimensional model, and plan the puncture path based on the trained SAC algorithm;

[0024] The path optimization module is used to provide geometric constraints for the puncture path, perform collision detection on the puncture path, and finally optimize the puncture path using a reward function to obtain an optimized puncture path.

[0025] Optionally, the model reconstruction module includes a sampling unit and a voxelization unit;

[0026] The sampling unit is used to uniformly sample the three-dimensional organ model to obtain model point cloud data; the voxelization unit is used to voxelize the model point cloud data to obtain a reconstructed three-dimensional model.

[0027] Optionally, the path planning module includes an algorithm training unit and a path planning unit;

[0028] The algorithm training unit is used to provide a training environment for the SAC algorithm based on the reconstructed three-dimensional model to obtain a trained SAC algorithm; the path planning unit is used to plan a puncture path based on the trained SAC algorithm.

[0029] Optionally, the path optimization module includes a geometric constraint unit, a collision detection unit and a path optimization unit;

[0030] The geometric constraint unit is used to obtain the current position and target position of the puncture needle, calculate the direction vector of the puncture needle based on the functional relationship between the current position and the target position; calculate the pitch angle and yaw angle of the target direction based on the direction vector, and update the direction vector of the puncture needle based on the pitch angle and yaw angle; preset the moving distance of the puncture needle, and calculate the new position of the puncture needle based on the updated direction vector and moving distance;

[0031] The collision detection unit is used to determine the radius of the sphere based on half of the maximum moving distance of the puncture needle, and decompose the puncture needle into a plurality of spheres based on the sphere radius; insert the position data of all obstacles in the training environment into the KD tree to construct a spatial index; and then perform fine-grained collision detection for each sphere of the puncture needle based on the KD tree;

[0032] The path optimization unit is used to optimize the puncture path by using an angle reward function and a state reward function to obtain an optimized puncture path.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] (1) Improved path accuracy:

[0035] The improved SAC algorithm of the present invention can gradually optimize the puncture path through reinforcement learning in a large number of simulation trainings, making the path more accurate and able to avoid obstacles and accurately reach the target area.

[0036] (2) Efficiency of collision detection:

[0037] The present invention introduces the KD tree structure and the bounding sphere method for collision detection, which can quickly detect obstacles, reduce the amount of calculation, and improve the efficiency of path planning; and the use of the KD tree enables the algorithm to quickly find the nearest neighbor in a high-dimensional space, thereby performing accurate collision detection.

[0038] (3) Real-time feedback mechanism:

[0039] The present invention adopts a real-time reward and punishment mechanism to adjust the algorithm parameters in real time according to the quality of the puncture path, so that the optimal path planning strategy can be learned quickly.

[0040] (4) Efficient planning in complex environments:

[0041] The improved SAC algorithm can consider a variety of geometric and physiological constraints and perform effective path planning, especially the handling of a variety of obstacles, and can intelligently bypass obstacles to ensure safety and effectiveness.

[0042] (5) High fault tolerance:

[0043] Through continuous training and optimization, the SAC algorithm can have higher fault tolerance. Even if unknown situations occur in certain environments, it can better adjust the path and avoid major mistakes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 The figure is a flow chart of a puncture path planning method based on an improved SAC algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Embodiment 1

[0049] like Figure 1 As shown, this embodiment provides a puncture path planning method based on an improved SAC algorithm, comprising the following steps:

[0050] SAC (Soft Actor-Critic) is a model-based reinforcement learning algorithm that is particularly suitable for solving tasks in continuous action spaces. It is an Actor-Critic method that combines a value function in policy optimization, but introduces entropy regularization during training to improve the exploratory nature of the policy, thereby more effectively exploring the environment and learning more robust policies. For the puncture task, this embodiment improves the SAC algorithm from three aspects: state environment, safety constraints, and reward design.

[0051] State Environment:

[0052] In this embodiment, the three-dimensional organ model of the medical image is uniformly sampled to obtain model point cloud data, and then the point cloud is voxelized. In this embodiment, the three-dimensional organ is constructed by multi-scale voxelization, which can balance accuracy and computing performance.

[0053] The reconstructed 3D model has three main functions. First, it can provide clear and intuitive visual feedback to help observe the behavior of the agent and the state of the environment. Second, it is used to simulate the real environment and ensure that the path planning does not pass through obstacles by detecting the collision between the agent and the model. Finally, it helps define the range of motion of the agent and guides the agent to learn to find the optimal path to the target point in a complex environment. In this embodiment, according to clinical requirements, voxels can be divided into three categories:

[0054] Puncture target area: such as liver tumor area, which has positive feedback to the agent. These areas are the priority targets of the path planning algorithm, and the path design should try to reach these areas.

[0055] Impenetrable obstacles: such as bones, large blood vessels, etc., have a great negative feedback to the agent. These voxels are marked as high-risk areas in the 3D environment, and the path planning algorithm should avoid these areas.

[0056] Penetrable obstacles: such as capillaries, ordinary muscle tissue, etc., have little negative feedback on the agent.

[0057] Based on this, a complete training environment is obtained.

[0058] Physical constraints:

[0059] To ensure that the puncture needle moves along a safe and effective path, this embodiment imposes geometric constraints on its movement direction. Specifically, the forward direction of the puncture needle is limited to a cone-like range to prevent it from deviating from the predetermined trajectory and avoiding damage to key anatomical structures.

[0060] The movement of the puncture needle is controlled by the action in reinforcement learning, which consists of three parameters: pitch, ), yaw angle (yaw, ) and the moving distance (distance, d∈[a,b]). Pitch and yaw limit the forward direction to the cone range, while distance ensures that the movement of the puncture needle in each step is moderate to avoid excessive or small displacement.

[0061] To achieve geometric constraints, the motion direction of the puncture needle needs to be adjusted according to the current state and target position. First, according to the current position of the puncture needle P current and the target position P target , calculate the direction vector pointing to the target as:

[0062]

[0063] Then, according to the direction vector u target Calculate the pitch angle θ in the target direction target and the yaw angle ψ target for:

[0064] θ target =arcsin(u target,z ),ψ target =arctan2(u target,y ,u target,x );

[0065] Next, the pitch angle θ and yaw angle ψ of the motion parameters are used to update the direction vector d of the puncture needle. new . Construct the pitch angle adjustment matrix R θ and the yaw angle adjustment matrix R ψ :

[0066]

[0067] Calculate the direction vector d new =R ψ ·R θ ·u target .

[0068] Finally, according to the adjusted direction vector and the given moving distance d, the new position of the puncture needle is calculated. new for:

[0069]

[0070] Fine-grained collision detection based on the enclosing shape method:

[0071] In the puncture path planning, in order to ensure that the puncture needle does not collide with the three-dimensional organ during the movement, this embodiment adopts a fine-grained collision detection method combining the enclosing shape method and the KD tree.

[0072] First, the puncture needle is decomposed into multiple spheres, and the radius (r) of each sphere is set to the maximum moving distance (d max ) to ensure that the sphere covers not only the puncture needle itself but also its range of movement. Secondly, in order to perform efficient collision detection, a KD tree is used to spatially index the three-dimensional organs in the environment. The KD tree is an efficient spatial segmentation data structure that can quickly perform nearest neighbor search. The position data of all obstacles are inserted into the KD tree to build a spatial index. Finally, for each sphere of the puncture needle, a KD tree is used to perform fine-grained collision detection. This embodiment uses the radius of the sphere as the detection radius and uses the KD tree to perform nearest neighbor search to find all obstacles within the detection radius. That is:

[0073] N i =query_KD_Tree(KD_Tree,c i ,r)

[0074] Among them, N i =query_KD_Tree() represents the nearest neighbor search, KD_Tree represents the spatial index of the three-dimensional organ, and the center position of the sphere is c i , the radius of the sphere is r.

[0075] Reward design in complex three-dimensional space:

[0076] In the puncture path planning task, the design of the reward function is crucial when using the SAC (Soft Actor-Critic) algorithm. The reward function guides the agent to learn effectively by providing clear and timely feedback. Reasonable reward function design ensures that the agent learns to optimize the puncture path in a complex medical environment and improves the accuracy and safety of the surgery.

[0077] Angle Bonus:

[0078] The angle reward in this embodiment is calculated by cosine value. The use of cosine value avoids sudden reward changes caused by angle changes, reduces the risk of gradient explosion and disappearance, and makes the reward more continuous and smooth. The reward calculation formula based on cosine value is:

[0079]

[0080] Among them, d new ·d target is the dot product of two vectors, ||d new || and ||d target || is the norm of the vector and λ is the coefficient.

[0081] Status Rewards:

[0082] To further improve learning efficiency, an environment-scale adaptive reward function based on the risk levels of different organs in clinical practice is adopted.

[0083]

[0084] Among them, r is the reward function, variable s t and a t Represent the state and action at time t respectively. Action a t After that, the state s t Transform to state s at time t+1 t+1 . Set goal is the target area, Set bone For the bone area, Set spleen For the spleen area, Set artery For the arterial region, Set nerve is the nerve and vein area. The reinforcement learning action is initialized as follows: the action space set A = (a1,...,a a ), the state space set S = (s1,...,s s ). Action a t The reward value depends on the state s t+1 (target or obstacle). The target area includes the target area Set goal (Tumor voxels), obstacle areas include nerve and vein areas Set nerve (Venous and Nerve) areas of danger include bone areas Set bone (bone voxel), artery region Set artery (artery voxels) and spleen region Set spleen (spleen voxel). If s t+1 If the position does not belong to the positions of these voxels, it means that the current position may be the middle position of the puncture needle.

[0085] This embodiment also provides a puncture path planning system based on an improved SAC algorithm, comprising: a model reconstruction module, a path planning module and a path optimization module connected in sequence;

[0086] The model reconstruction module is used to obtain a three-dimensional organ model, and perform uniform sampling and voxelization processing on the three-dimensional organ model to obtain a reconstructed three-dimensional model;

[0087] The path planning module is used to provide a training environment for the SAC algorithm based on the reconstructed three-dimensional model, and plan the puncture path based on the trained SAC algorithm;

[0088] The path optimization module is used to provide geometric constraints for the puncture path, perform collision detection on the puncture path, and finally optimize the puncture path using a reward function to obtain an optimized puncture path.

[0089] It can be implemented that the model reconstruction module includes a sampling unit and a voxelization unit; the sampling unit is used to uniformly sample the three-dimensional organ model to obtain model point cloud data; the voxelization unit is used to voxelize the model point cloud data to obtain a reconstructed three-dimensional model.

[0090] It can be implemented that the path planning module includes an algorithm training unit and a path planning unit; the algorithm training unit is used to provide a training environment for the SAC algorithm based on the reconstructed three-dimensional model to obtain the trained SAC algorithm; the path planning unit is used to plan the puncture path based on the trained SAC algorithm.

[0091] It can be implemented that the path optimization module includes a geometric constraint unit, a collision detection unit and a path optimization unit;

[0092] The geometric constraint unit is used to obtain the current position and target position of the puncture needle, calculate the direction vector of the puncture needle based on the functional relationship between the current position and the target position; calculate the pitch angle and yaw angle of the target direction based on the direction vector, and update the direction vector of the puncture needle based on the pitch angle and yaw angle; preset the moving distance of the puncture needle, and calculate the new position of the puncture needle based on the updated direction vector and moving distance;

[0093] The collision detection unit is used to determine the radius of the sphere based on half of the maximum moving distance of the puncture needle, and decompose the puncture needle into a plurality of spheres based on the sphere radius; insert the position data of all obstacles in the training environment into the KD tree to construct a spatial index; and then perform fine-grained collision detection for each sphere of the puncture needle based on the KD tree;

[0094] The path optimization unit is used to optimize the puncture path by using an angle reward function and a state reward function to obtain an optimized puncture path.

[0095] This embodiment of the medical puncture path planning based on the improved SAC (Soft Actor-Critic) algorithm can achieve the following effects: (1) Improved path accuracy: The improved SAC algorithm can gradually optimize the puncture path through reinforcement learning in a large number of simulation trainings, making the path more accurate and able to accurately reach the target area while avoiding important organs and blood vessels. (2) High efficiency of collision detection: By introducing the KD tree structure and the bounding ball method for collision detection, obstacles can be detected quickly, the amount of calculation can be reduced, and the efficiency of path planning can be improved; the use of the KD tree enables the algorithm to quickly find the nearest neighbor in a high-dimensional space, thereby performing accurate collision detection. (3) Real-time feedback mechanism: During the training process, a real-time reward and punishment mechanism is used to adjust the algorithm parameters in real time according to the quality of the puncture path, so that the intelligent agent can quickly learn the optimal path planning strategy; the instant feedback mechanism can help the algorithm to quickly adjust in simulation and actual operation, thereby improving adaptability. (4) Efficient planning in complex environments: The improved SAC algorithm can consider multiple geometric and physiological constraints in complex anatomical environments and perform effective path planning, especially for the handling of multiple obstacles, and can intelligently bypass key tissues to ensure safety and effectiveness. (5) High fault tolerance: Through continuous training and optimization, the algorithm can have higher fault tolerance. Even if unknown situations occur in certain environments, it can better adjust the path and avoid major mistakes.

[0096] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A puncture path planning method based on an improved SAC algorithm, characterized in that: The following steps are involved: Acquire a three-dimensional organ model, perform uniform sampling and voxelization processing on the three-dimensional organ model, and obtain a reconstructed three-dimensional model; Providing a training environment for the SAC algorithm based on the reconstructed 3D model, and planning the puncture path based on the trained SAC algorithm; During the planning process, geometric constraints are provided for the puncture path, and collision detection is performed on the puncture path based on a fine-grained collision detection method combining the enclosing shape method and the KD tree; Finally, the puncture path is optimized using the angle reward function and the state reward function to obtain the optimized puncture path; The angle reward function is as follows: Among them, reward angle is the angle reward function, d new ·d target is the dot product of two vectors, ||d new || and ||d target || is the norm of the vector, and λ is the coefficient; The state reward function is as follows: Among them, r is the reward function, and the variable s t and a t Represent the state and action at time t, respectively. Action a t After that, the state s t Transform to state s at time t+1 t+1 , Set goal is the target area, Set bone For the bone area, Set spleen For the spleen area, Set artery For the arterial region, Set nerve For nerve and vein areas, S target is the target area state, α is the distance penalty coefficient, reward penalty Penalty for algorithm steps.

2. The puncture path planning method based on the improved SAC algorithm according to claim 1 is characterized in that: The reconstructed three-dimensional model includes: a puncture target area, impenetrable obstacles and penetrable obstacles.

3. The puncture path planning method based on the improved SAC algorithm according to claim 1 is characterized in that: The process of providing geometric constraints for the puncture path includes: obtaining the current position and target position of the puncture needle, and calculating the direction vector of the puncture needle based on the functional relationship between the current position and the target position; calculating the pitch angle and yaw angle of the target direction based on the direction vector, and updating the direction vector of the puncture needle based on the pitch angle and yaw angle; presetting the moving distance of the puncture needle, and calculating the new position of the puncture needle based on the updated direction vector and moving distance.

4. The puncture path planning method based on the improved SAC algorithm according to claim 1, characterized in that: The process of collision detection on the puncture path includes: determining the radius of the sphere based on half of the maximum moving distance of the puncture needle, and decomposing the puncture needle into several spheres based on the sphere radius; inserting the position data of all obstacles in the training environment into the KD tree to build a spatial index; and then performing fine-grained collision detection on each sphere of the puncture needle based on the KD tree.

5. A puncture path planning system based on an improved SAC algorithm, characterized in that: The puncture path planning method based on the improved SAC algorithm according to any one of claims 1 to 4 comprises: a model reconstruction module, a path planning module and a path optimization module connected in sequence; The model reconstruction module is used to obtain a three-dimensional organ model, and perform uniform sampling and voxelization processing on the three-dimensional organ model to obtain a reconstructed three-dimensional model; The path planning module is used to provide a training environment for the SAC algorithm based on the reconstructed three-dimensional model, and plan the puncture path based on the trained SAC algorithm; The path optimization module is used to provide geometric constraints for the puncture path, perform collision detection on the puncture path, and finally optimize the puncture path using a reward function to obtain an optimized puncture path.

6. The puncture path planning system based on the improved SAC algorithm according to claim 5, characterized in that: The model reconstruction module includes a sampling unit and a voxelization unit; The sampling unit is used to uniformly sample the three-dimensional organ model to obtain model point cloud data; the voxelization unit is used to voxelize the model point cloud data to obtain a reconstructed three-dimensional model.

7. The puncture path planning system based on the improved SAC algorithm according to claim 5, characterized in that: The path planning module includes an algorithm training unit and a path planning unit; The algorithm training unit is used to provide a training environment for the SAC algorithm based on the reconstructed three-dimensional model to obtain a trained SAC algorithm; the path planning unit is used to plan a puncture path based on the trained SAC algorithm.

8. The puncture path planning system based on the improved SAC algorithm according to claim 5, characterized in that: The path optimization module includes a geometric constraint unit, a collision detection unit and a path optimization unit; The geometric constraint unit is used to obtain the current position and target position of the puncture needle, calculate the direction vector of the puncture needle based on the functional relationship between the current position and the target position; calculate the pitch angle and yaw angle of the target direction based on the direction vector, and update the direction vector of the puncture needle based on the pitch angle and yaw angle; preset the moving distance of the puncture needle, and calculate the new position of the puncture needle based on the updated direction vector and moving distance; The collision detection unit is used to determine the radius of the sphere based on half of the maximum moving distance of the puncture needle, and decompose the puncture needle into a plurality of spheres based on the sphere radius; insert the position data of all obstacles in the training environment into the KD tree to construct a spatial index; and then perform fine-grained collision detection for each sphere of the puncture needle based on the KD tree; The path optimization unit is used to optimize the puncture path by using an angle reward function and a state reward function to obtain an optimized puncture path.

Images