Vascular navigation path planning method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of vascular navigation and deep neural network modeling, and provides a vascular navigation path planning method based on deep reinforcement learning. The method comprises the following steps: firstly, constructing a vascular navigation state action model by unifying a coordinate system, dividing a safety corridor and binding a state action index; configuring parameters of a strategy network, a value network, an experience playback buffer area and a soft target network to form a training preparation structure; training to obtain a convergent navigation model through multiple interaction rounds in combination with composite reward calculation and parameter updating; finally, candidate paths are generated, and deployable vascular navigation configuration is output through screening, instruction resampling and parameter packaging. According to the method, the safety, efficiency and control smoothness of path planning can be effectively improved, vessel wall collision is avoided, the method adapts to a complex vessel branch environment, and the reliability and adaptability of robot navigation are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of vascular navigation and deep neural network modeling technology, and particularly relates to a vascular navigation path planning method based on deep reinforcement learning. Background Technology

[0002] In the field of vascular navigation and deep neural network modeling, existing solutions for vascular simulation environment data and robot initial pose and target point configuration typically model and implement vascular image reconstruction, path planning, and robot control separately and step by step. This results in limitations such as a disconnect between the basic data structure of the vascular environment state and the representation structure of the vascular navigation state, a simplistic design of vascular navigation interaction rounds and composite reward calculations, and a lack of unified construction of the vascular navigation training convergence model structure. Existing methods often rely on static models, offline search, or simple rule control. In scenarios involving dynamic changes in the vascular environment, safety corridor constraints, and continuous motion control, these methods are prone to insufficient matching between state representation and action space, and unstable training processes. This makes it difficult to achieve stable generation of vascular navigation candidate paths and control sequences, leading to complex path design, lengthy verification cycles, and high system integration difficulties for clinical interventional procedures and related equipment development. For the joint processing of the vascular navigation training preparation structure and the basic data structure of the vascular environment state, existing technologies generally have shortcomings such as low coupling and incomplete links in the stages of vascular navigation interactive round acquisition, compound reward calculation, experience playback buffer management, and linkage update of policy network and value network. It is difficult to form a continuous process from environment modeling, state alignment, interactive judgment, policy control to deployment recording in application scenarios of vascular navigation candidate path and control sequence generation and navigation deployment parameter packaging. As a result, the vascular navigation training convergence model structure cannot simultaneously take into account safety boundary constraints and path efficiency requirements, which limits the integrated promotion of related production processing and system deployment. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a blood vessel navigation path planning method based on deep reinforcement learning, comprising: Acquire vascular simulation environment data and robot initial pose and target point configuration, perform coordinate system I, safety corridor division and state action index binding processing, and generate vascular navigation state action modeling structure; Based on the vascular navigation state-action modeling structure, policy network configuration and value network construction are performed, and experience playback buffer and soft target network parameter setting are executed to generate the vascular navigation training preparation structure. Based on the vascular navigation training preparation structure, the vascular navigation interactive rounds and compound reward calculation and parameter update processing are executed to generate the vascular navigation training convergence model structure. Based on the vascular navigation training convergence model structure, candidate paths and control sequences for vascular navigation are generated and processed. Path selection, control command resampling, and navigation deployment parameter packaging are then performed to generate the vascular navigation deployment configuration structure.

[0004] Furthermore, the process of establishing coordinate system one also includes: The coordinate system processing involves using the world coordinate system defined in the vascular simulation environment data as the navigation reference coordinate system, transforming the robot's initial pose and target point configuration to this navigation reference coordinate system, applying rigid body transformation to the position field in the robot's initial pose, and using a unified angle description reduction to convert the pose field into a unified pose expression. At the same time, the local coordinates in the target point configuration are mapped to the spatial position near the corresponding vascular centerline based on the vascular inlet mark and branch identifier, and the coordinate units, coordinate axis directions, and origin position are standardized to form a general spatial reference frame.

[0005] Furthermore, the process of defining safety corridors also includes: The safety corridor division process involves traversing the centerlines of each blood vessel from the vascular simulation environment data, generating a local cross-sectional description at each centerline point based on the corresponding radius and adjacent sensitive tissue labels, constructing several annular zones according to preset radial distance intervals, marking areas close to the centerline and far from the blood vessel wall and sensitive tissue mask as passable areas, and marking areas close to the blood vessel wall or entering the sensitive tissue mask as danger areas, connecting adjacent passable areas longitudinally to form continuous passable channel segments, and assigning a safety corridor number to each segment. At the same time, a mapping relationship is established between the corridor number and the branch node according to the vascular tree topology to record upstream and downstream connections.

[0006] Furthermore, the state-action index binding process also includes: The state-action index binding process includes mapping the local curvature and vessel diameter fields in the vascular navigation state representation structure to action channels related to steering amount, mapping the safety corridor boundary distance and danger zone distance fields to action channels related to radial offset amount, and mapping the target point orientation relationship and topological distance fields to action channels related to propulsion amount, according to the state-action correspondence rules. The index of each state field, the index of the corresponding action channel, and the fusion weight coefficient are recorded through an index table.

[0007] Furthermore, the process of configuring the strategy network and constructing the value network also includes: The policy network configuration and value network construction process includes configuring the number of input layer nodes of the policy network according to the length of the state vector of the vascular navigation state representation structure template to match the state dimension, and corresponding the input layer node indices according to the order of the state fields. At the same time, it configures the number of output layer nodes according to the number of action channels to correspond to the propulsion, offset, and steering control channels respectively. An output transformation unit is added to the output layer to perform amplitude pruning and range mapping to ensure that the control quantities fall within the action boundaries. Several hidden layers are selected for the policy network and the number of nodes is automatically given according to preset configuration rules. In the design of the hidden layers, nonlinear transformations are performed on key state fields through multi-branch feature extraction sub-layers, and feature fusion is performed in the middle and high layers. The value network construction process includes design state input branches and action input branches. The number of input layer nodes in the state input branch is consistent with the dimension of the state vector, and the number of input layer nodes in the action input branch is consistent with the number of action channels. After feature transformation, the state features and action features are concatenated at a certain hidden layer depth. Then, a scalar output is generated through a common sub-network. The state-action index relationship table is referenced to give the state fields describing safety and target proximity a higher connection priority.

[0008] Furthermore, the process of executing the experience playback buffer and soft target network parameter setting also includes: Perform experience playback buffer and soft target network parameter setting processing; The experience replay buffer configuration process includes configuring the storage layout of state fields, action fields, immediate reward fields, and subsequent state fields for a single record based on the state vector dimension and the number of action channels, setting the maximum buffer capacity and write index rolling rules, and maintaining sampling weights for each record and dynamically adjusting them based on the immediate reward magnitude, value estimation bias, and number of samplings; The soft target network parameter setting process includes configuring update step size parameters and update frequency parameters for the target policy network and the target value network, and adopting an update mechanism that is uniformly triggered according to the number of training steps or adaptively adjusting the update step size according to the changes in the loss function.

[0009] Furthermore, the process of performing the vascular navigation interactive rounds and calculating compound rewards also includes: The vascular navigation interactive round processing includes constructing a simulated navigation environment instance based on the robot's initial pose field and target point configuration field in the vascular environment state basic data structure. It supports reset and stepping operations. The reset operation generates a starting state near the vascular centerline based on the robot's initial pose. The stepping operation propels the robot's movement, updates its pose, and detects collision, boundary crossing, or endpoint events based on input control commands. It also starts multiple interactive rounds according to the number of training rounds and the maximum number of steps per round. At each time step, the state vector is extracted from the current vascular navigation state representation structure and input into the online policy network to obtain continuous control actions. The legality of the actions is checked, and out-of-bounds actions are pruned before being submitted to the stepping interface. After updating, the new state and event flags are returned. The composite reward calculation process includes calculating the target arrival reward component based on the termination flag and target point configuration field for each time step, calculating the boundary violation penalty component based on the collision type identifier and boundary violation flag, calculating the safety corridor deviation penalty component based on the degree of safety corridor deviation, calculating the path efficiency component based on the centerline information and path length statistics, and calculating the trajectory smoothness penalty component based on the difference in control actions in consecutive time steps. These components are then combined into an instant composite reward using preset weighting coefficients.

[0010] Furthermore, the parameter update process also includes: The parameter update process involves: when the number of valid records in the experience replay buffer reaches the minimum sampling threshold, a batch of state transition records is randomly selected from the buffer using weighted sampling. The current state and action are input into the online value network to obtain the current value estimate. The subsequent state is input into the target policy network and the target value network to obtain the long-term value estimate. The target value estimate is constructed by combining the immediate reward and the termination flag. The loss is calculated and the gradient of the online value network parameters is updated. Then, the current state is input into the online policy network with a fixed sample and the long-term value of the action is evaluated through the online value network to adjust the policy network parameters. Finally, the online network parameters are partially transferred to the target network according to the soft target network parameter settings.

[0011] Furthermore, the process of generating candidate blood vessel navigation paths and control sequences also includes: The process of generating candidate paths and control sequences for vascular navigation includes loading the robot's initial pose and target point configuration fields from the basic data structure of the vascular environment state, constructing the navigation start state and encoding it into a vascular navigation state representation structure, performing numerical normalization and field alignment according to the state preprocessing configuration, inputting it into a convergence strategy network to obtain continuous control actions, and performing amplitude clipping and constraint mapping on the actions. Then, in the simulation environment instance, the pose is updated step by step and collision, boundary crossing or target arrival events are detected. Through multiple rounds of independent operation, different perturbation seeds are applied to generate multiple candidate paths and control sequences, and the pose sequence, control command sequence and event flag sequence are temporarily stored.

[0012] Furthermore, the process of executing path filtering also includes: The path selection process involves verifying the safety of each pose point for each path in the candidate path set based on the safety corridor configuration field and obstacle occupancy mask. Paths that cross the safety corridor boundary or enter a danger zone are marked as unqualified. For qualified paths, the actual path length, topological redundancy, deviation from the shortest centerline path, and trajectory smoothness index are calculated. Path evaluation records are constructed and sorted by multiple indicators to generate a candidate path ranking list.

[0013] The key innovations of this invention include: (1) In the process of processing vascular simulation environment data and robot initial pose and target point configuration, the basic data structure of vascular environment state and the vascular navigation state representation structure are jointly organized into vascular navigation state action modeling structure by dividing the coordinate system and safety corridor and binding the state action index, so that vascular geometric information, safety constraint information and continuous action space are associated and described in the same modeling structure.

[0014] (2) Under the joint control of the vascular navigation training preparation structure and the vascular environment state basic data structure, through the vascular navigation interactive rounds and composite reward calculation and parameter update processing, the vascular navigation strategy network structure initialization result, vascular navigation strategy value network initialization structure, experience replay buffer and soft target network parameter setting fields are used in a coordinated manner, and factors such as target arrival, safety corridor deviation, collision overstepping, path efficiency and trajectory smoothing are uniformly incorporated into the composite reward calculation link to generate the vascular navigation training convergence model structure.

[0015] (3) Based on the vascular navigation training convergence model structure and the basic data structure of vascular environment state, the vascular navigation training convergence model structure obtained by training is converted into a vascular navigation deployment configuration structure by generating vascular navigation candidate paths and control sequences, filtering paths and resampling control instructions and packaging navigation deployment parameters. In the same data link, vascular navigation candidate paths and control sequences and navigation deployment parameters matching the robot platform are given at the same time.

[0016] The following are its main beneficial effects: (1) By storing the basic data structure of the vascular environment state and the vascular navigation state representation structure in the vascular navigation state action modeling structure at the same time, and by using the safety corridor division and state action index binding, the state information formed by the vascular simulation environment data, the robot's initial pose and the target point configuration is directly correlated with the continuous control action. Compared with the method of dispersing the environmental modeling and control variables, it is beneficial to directly call the unified vascular navigation state action modeling structure in the deep reinforcement learning model, reduce the intermediate links of state transition and constraint processing, and make the modeling process of vascular navigation task more coherent.

[0017] (2) By performing vascular navigation interactive rounds and compound reward calculation and parameter update processing under the support of the vascular navigation training preparation structure, and using the vascular navigation strategy network structure initialization results, vascular navigation strategy value network initialization structure, experience replay buffer and soft target network parameter setting fields to construct the vascular navigation training convergence model structure, the immediate reward is no longer limited to a single target, but simultaneously reflects multiple factors such as target arrival, safety corridor deviation, collision overrun, path efficiency and trajectory smoothness. Compared with training methods that rely only on simple rewards or single constraints, it is more conducive to forming a strategy update process that takes into account both safety boundaries and path efficiency in the continuous action control scenario in blood vessels, and reducing invalid paths and unstable control behaviors during training.

[0018] (3) By generating vascular navigation candidate paths and control sequences based on the vascular navigation training convergence model structure during the inference stage, and combining path screening, control command resampling and navigation deployment parameter packaging, the control results output by the vascular navigation training convergence model structure are reorganized into a vascular navigation deployment configuration structure. Compared with the method of directly using the model output for control, it is more conducive to forming a complete link from the training model to the vascular navigation candidate paths and control sequences to the navigation deployment parameters in vascular navigation applications. This allows the path geometry information, safety parameters and control commands to be uniformly organized before deployment, which is convenient for docking with the actual robot platform and for verification and application in clinical or engineering environments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a blood vessel navigation path planning method based on deep reinforcement learning, provided as an embodiment of this application. Detailed Implementation

[0020] Example 1: Refer to Figure 1 This is a flowchart illustrating a blood vessel navigation path planning method based on deep reinforcement learning provided in an embodiment of the present invention. The flowchart may include at least steps S100-S400: S100: Acquire vascular simulation environment data and robot initial pose and target point configuration, perform coordinate system I, safety corridor division and state action index binding processing, and generate vascular navigation state action modeling structure; S200: Based on the vascular navigation state-action modeling structure, perform policy network configuration and value network construction processing, and execute experience playback buffer and soft target network parameter setting processing to generate vascular navigation training preparation structure. S300: Based on the vascular navigation training preparation structure, perform vascular navigation interactive rounds and compound reward calculation, parameter update processing, and generate the vascular navigation training convergence model structure. S400: Based on the vascular navigation training convergence model structure, perform vascular navigation candidate path and control sequence generation processing, and perform path screening, control command resampling and navigation deployment parameter packaging processing to generate vascular navigation deployment configuration structure.

[0021] Specifically, in S100, the vascular simulation environment data and the robot's initial pose and target point configuration are acquired, and coordinate system I, safety corridor division and state action index binding processing are performed to generate the vascular navigation state action modeling structure. This step first calls the vascular simulation environment data output from the preceding image reconstruction and data preprocessing stages, and simultaneously reads the robot's initial pose and target point configuration recorded in the interventional planning configuration. These three types of inputs are then integrated into the processing flow through the same data channel. In this embodiment, the vascular simulation environment data is a three-dimensional structured description of the patient's target vascular region, containing vascular tree topology, vascular cavity mesh, centerline point set, local radius and curvature information at each centerline point, and region labels for adjacent sensitive tissues. It also includes a space occupancy mask describing organ boundaries and existing implants. The robot's initial pose is the starting position and attitude information in the navigation coordinate system, described by a set of position coordinates and attitude angles, along with an identification marker for the currently inserted vascular inlet. The target point configuration is the terminal position to be reached and a set of several intermediate reference points, accompanied by a priority marker and the identifier of the vascular branch to which each target point belongs. After loading the above inputs, the system checks the data integrity, filters records missing key geometric or topological fields and writes them to the log, and performs basic smoothing and interpolation operations on obviously abnormal radius sequences and centerline sampling point distributions to form a data foundation layer suitable for subsequent modeling.

[0022] In the coordinate system processing stage, this step selects the world coordinate system defined in the vascular simulation environment data as the navigation reference coordinate system, transforming the robot's initial pose and target point configuration recorded in the robot body coordinate system and the operating table coordinate system to this navigation reference coordinate system. Specifically, based on pre-calibrated extrinsic parameter relationships, the system applies rigid body transformation to the position field in the robot's initial pose, adopts a unified angle description reduction for the attitude field, converts the attitude descriptions output by different devices into a unified attitude expression, and maps the local coordinates in the target point configuration to the spatial position near the corresponding vascular centerline according to the vascular inlet mark and branch identifier. Coordinate system one also standardizes the coordinate units, coordinate axis directions, and origin position, realigning coordinate data from different sources to form a universal spatial reference framework throughout the navigation task. On this basis, the system recalculates the offset relationship between the robot's initial pose and the vascular cross-section, as well as the initial distance estimate between the robot and each target point, providing a unified spatial semantics for subsequent safety corridor division and state coding steps.

[0023] After completing coordinate system one, this step performs safe corridor partitioning under a unified coordinate system. The system sequentially traverses each vessel centerline from the vascular simulation environment data, generating a local cross-sectional description at each centerline point based on the corresponding radius and adjacent sensitive tissue labels. For each local cross-section, the system constructs several annular zones according to preset radial distance intervals, marking areas close to the centerline and far from the vessel wall and sensitive tissue mask as passable areas, and areas close to the vessel wall or entering the sensitive tissue mask as danger areas. Based on the order of the centerline points, adjacent passable areas are connected longitudinally to form several continuous passable channel segments, and a safe corridor number is assigned to each segment. For branch areas, the system establishes a mapping relationship between corridor numbers and branch nodes according to the vascular tree topology, so that each safe corridor segment simultaneously carries the branch identifier and the connection relationship with upstream and downstream corridors. After this processing, the system obtains a corridor configuration field containing the safe corridor geometric boundary, passable area mask, danger area mask, and safe corridor number sequence, and adds this field to the vascular environment state basic data structure to describe the safe navigation channel information corresponding to each spatial location.

[0024] In constructing the basic data structure for the vascular environment state, this step organizes multi-source information with unified coordinates and completed safety corridor division into a composite graph structure and a grid structure. Specifically, the system uses sampling points on the vascular centerline as graph nodes, the physical connectivity between adjacent sampling points as graph edges, and attaches attribute fields such as local radius, curvature, nearest distance to sensitive tissues, safety corridor number, and estimated topological distance to the target point to each node. In addition to the graph structure, the system maintains a sparse grid index, mapping the graph node indexes to voxel grids in 3D space for quick querying of local obstacle occupancy and safety corridor masks. The robot's initial pose field and target point configuration field are appended to the basic data structure for the vascular environment state in a dedicated record format, and are associated with the vascular tree graph structure through the entry node index and target node index. Through this organization, the basic data structure for the vascular environment state forms a comprehensive environmental description table that can be jointly accessed by deep neural networks and control strategies, providing a unified data source for subsequent state encoding, interactive sampling, and path reconstruction.

[0025] In the construction phase of the vascular navigation state representation structure, this step, targeting the state input format required by the deep reinforcement learning decision-making process, extracts a subset of information relevant to the current navigation moment from the basic data structure of the vascular environment state, and performs reduction and encoding. The system searches for the nearest graph nodes based on the robot's current position, and combines fields such as local radius, curvature, safety corridor number, distance to sensitive tissues, and topological distance to the target point of these nodes in a predetermined order. Simultaneously, it extracts the robot's current position, current orientation, and control command summary fields from the robot's initial pose and the latest executed control commands, forming supplementary information reflecting navigation dynamics. After range pruning, outlier suppression, and numerical normalization, these fields are written into fixed-length state vector slots to form the vascular navigation state representation structure. This structure not only contains the geometric and safety information of the current moment but also explicitly records the current safety corridor number and neighboring corridor numbers, guiding the subsequent policy network to refer to safety boundaries and possible turning paths during decision-making. After this step, the vascular navigation state representation structure serves as a state template during sample construction and is frequently referenced during subsequent experience replay buffer writing and policy network forward inference.

[0026] In the state-action index binding stage, this step addresses the continuous action control requirements by establishing explicit associations between different fields in the vascular navigation state representation structure and various control action channels. In this embodiment, the action channels in the navigation task consist of the advance along the vessel centerline, the offset in the radial direction towards the vessel cross-section, and the turning amount around the centerline. Each channel corresponds to a set of continuous control quantities output by the policy network. Based on expert experience and experimental analysis, the system internally constructs state-action correspondence rules, mapping fields describing local curvature and vessel diameter to action channels related to turning amounts, fields describing the current distance to the safety corridor boundary and the distance to the danger zone to action channels related to radial offsets, and fields describing the directional relationship and topological distance between the current position and the target point to action channels related to advance amounts. This mapping relationship is written into the vascular navigation state-action modeling structure in the form of an index table, recording the index of each state field, the index of the corresponding action channel, and the fusion weight coefficients involved in action generation. This is used to guide the selection of input / output connection methods and the setting of feature fusion strategies during the design of the policy network structure.

[0027] Through the above processing, this step ultimately generates a vascular navigation state-action modeling structure, which serves as the core data object in this embodiment for uniformly describing the vascular environment state, navigation state encoding, and action channel configuration. The vascular navigation state-action modeling structure includes a basic data structure for the vascular environment state, a template for the vascular navigation state representation structure, and a state-action index relationship table, playing a crucial role in subsequent steps. On one hand, this structure serves as the main input in the policy network and value network construction and initialization process (step S200), determining the network input layer morphology, output layer channel partitioning, and intermediate feature fusion methods. On the other hand, the basic data structure for the vascular environment state is invoked by the composite reward-driven experience sampling and parameter update process (step S300) to run interactive rounds and generate state transition samples. Simultaneously, it is used by the vascular navigation path and continuous control sequence generation process (step S400) to map the action sequence output by the policy network back to the real-world path and generate deployment configurations.

[0028] In summary, the technical effects of this step are as follows: This step involves unified modeling of vascular simulation environment data, robot initial pose, and target point configuration. It organizes complex 3D vascular geometry information and navigation control requirements into a basic data structure for vascular environment state and a vascular navigation state representation structure. Furthermore, it forms a vascular navigation state and action modeling structure through state-action index binding. This provides a clear and semantically consistent foundation for state and action descriptions in subsequent network construction, strategy training, and path generation, enabling continuous control strategies to have stable environmental representation support and clear control channel associations in vascular navigation scenarios.

[0029] S200: Based on the vascular navigation state-action modeling structure, perform policy network configuration and value network construction processing, and execute experience playback buffer and soft target network parameter setting processing to generate vascular navigation training preparation structure. In this step, the vascular navigation state-action modeling structure output from step S100 is used as the main input. This structure includes a basic data structure of the vascular environment state, a vascular navigation state representation structure template, and a state-action index relationship table. After loading the vascular navigation state-action modeling structure, the system first analyzes the scale of state information required for the deep reinforcement learning decision-making process based on the state vector length, field order, and value range of each field recorded in the vascular navigation state representation structure template. Simultaneously, the system reads the number of action channels defined in the state-action index relationship table, the physical meaning of each action channel, and the action value boundaries, thus forming a set of structural analysis results regarding the state input dimension, action output dimension, and the correspondence between state fields and action channels. These structural analysis results are used internally in this step as a basic description of the network topology design, written into the internal network structure design reference field, and provide input for the subsequent generation of the vascular navigation strategy network structure initialization results and the vascular navigation strategy value network initialization structure.

[0030] Specifically, in the initialization stage of the vascular navigation strategy network structure, the system configures the number of input layer nodes of the strategy network to be completely consistent with the dimension of the vascular navigation state representation structure based on the length of the parsed state vector, and maps the state fields to the input layer node indices in the order they appear in the template. Subsequently, the system selects several hidden layers for the strategy network based on the complexity of the vascular navigation scenario and the scale of the state space. The number of nodes in each hidden layer is automatically given according to preset structural configuration rules, empirical parameters, and available computing power constraints. In the hidden layer structure design, the system refers to the state-action index relationship table and performs nonlinear transformations on key state fields reflecting the safety corridor boundary distance, local vascular curvature, and relative direction to the target point through multi-branch feature extraction sub-layers. Then, feature fusion is performed at mid-to-high levels to form an intermediate representation that takes into account local geometric information, global target information, and safety constraint information. The number of output layer nodes of the strategy network is set according to the number of action channels, with each output node corresponding to control channels such as the advance amount along the vascular center direction, the radial offset of the cross-section, and the turning amount around the centerline. The system adds an output transformation unit after the output layer to perform amplitude pruning, sign constraint, and range mapping on the original network output, ensuring that the generated continuous control quantity falls within the action value boundaries recorded in the vascular navigation state-action modeling structure. After completing the above topology construction and connection configuration, the system records this topology description as the vascular navigation strategy network structure initialization result, indicating the storage location of parameters for each layer and their correspondence with the subsequent training process.

[0031] In the initialization phase of the vascular navigation strategy value network, the system designs a value network topology for evaluating the long-term rewards of states and actions, based on the aforementioned strategy network structure. The value network receives the state vector encoded by the vascular navigation state representation structure and its matching action vector. To accommodate the differentiated processing of state and action information, the system divides the value network into two paths: a state input branch and an action input branch. The number of input layer nodes in the state input branch matches the dimension of the vascular navigation state representation structure, while the number of input layer nodes in the action input branch matches the number of action channels. After several layers of feature transformations, both branches perform feature concatenation at a certain hidden layer depth, merging state and action features into a unified intermediate representation. Subsequently, the intermediate representation undergoes multiple nonlinear transformations in the common subnetwork, generating a scalar output representing the long-term reward estimate at the output layer. When constructing the value network topology, the system still references the state-action index relationship table, ensuring that state fields describing local safety and target proximity receive higher connection priority and richer feature transformation paths in the state input branch and common subnetwork, making the value estimation more closely reflect the key attributes of the vascular navigation task. After completing the topology design of the online policy network and online value network, the system replicates their structures to generate the target policy network structure and target value network structure, respectively. The topology descriptions, parameter buffer configurations, and update relationships of the online and target networks are recorded as the initialization structure of the vascular navigation policy value network. This initialization structure guides the parameter migration between the online and target networks during subsequent parameter update phases. At the end of this step, it is written into the vascular navigation training preparation structure along with the initialization results of the vascular navigation policy network structure.

[0032] Furthermore, after the policy network configuration and value network construction are completed, this step initiates the experience replay buffer configuration process. In this embodiment, the experience replay buffer is used to store data records such as states, actions, immediate rewards, and subsequent states generated during the vascular navigation interaction, supporting the batch extraction and reuse mechanism in deep reinforcement learning. Based on the state vector dimension and the number of action channels defined in the vascular navigation state representation structure template, the system configures the storage layout of state fields, action fields, immediate reward fields, and subsequent state fields for each record in the experience replay buffer, and sets the maximum capacity and write index rolling rules at the buffer level. During the interaction process, when a new state transition sample arrives, the system writes the record according to the current write index position, and sequentially increases the write index while the buffer is not full; when the buffer reaches its capacity limit, it replaces the earliest written record or replaces records with lower priority according to a preset coverage strategy to control the temporal distribution and diversity of samples in the buffer. Simultaneously, the system maintains a sampling weight value for each record. This sampling weight is dynamically adjusted during subsequent parameter updates based on the immediate reward magnitude, value estimation bias, and the number of times it is sampled. This ensures that when the experience replay buffer is accessed, it can focus on experience segments that have a greater impact on the current strategy changes. The capacity, field layout, coverage strategy, and sampling weight update rules of the experience replay buffer together constitute the experience management configuration field. This field is written into the vascular navigation training preparation structure at the end of this step and can be directly invoked in the composite reward-driven experience sampling and parameter update processing in step S300.

[0033] In the soft target network parameter setting process, this step defines the parameter synchronization method between the previously generated target policy network structure and target value network structure and the online policy network and online value network. The system configures update step size parameters and update frequency parameters for the target policy network and target value network respectively. These two types of parameters jointly describe the proportion of online network parameters migrating to target network parameters and the synchronization trigger time interval during each synchronization. In one implementation, the system adopts an update mechanism that is uniformly triggered according to the number of training steps. After completing several parameter update operations based on the experience replay buffer, the system performs weighted fusion between the online network parameters and target network parameters according to the update step size parameter, thereby gradually pushing the target network to follow the changes in the online network. In another implementation, the system adaptively adjusts the update step size parameter based on the recent changes in the loss function or policy stability indicators. When large training fluctuations are observed, the step size parameter is reduced; when training tends to be stable, the step size parameter is appropriately increased, making the response rhythm of the soft target network to changes in the online network smoother. The above soft target network parameter setting results are written into the vascular navigation training preparation structure as soft target network parameter setting fields. Together with the vascular navigation strategy network structure initialization results, the vascular navigation strategy value network initialization structure, and the experience playback buffer configuration data, they constitute a complete training preparation description.

[0034] Through the aforementioned policy network configuration, value network construction, experience replay buffer configuration, and soft target network parameter setting, this step ultimately generates the vascular navigation training preparation structure. This structure is used as direct input in step S300, the composite reward-driven experience sampling and parameter update process, to guide the sample writing method, batch sampling method, and the parameter update rhythm of the policy network and value network during interactive rounds. Simultaneously, in step S400, during the vascular navigation path and continuous control sequence generation process, it is used to load the network structure and parameter organization consistent with the training process, ensuring consistency in network morphology and control channel division between the inference and training phases.

[0035] The technical effect of this step can be summarized as follows: by generating the initialization results of the vascular navigation strategy network structure and the initialization structure of the vascular navigation strategy value network based on the vascular navigation state action modeling structure, and completing the setting of the experience playback buffer and soft target network parameters, a vascular navigation training preparation structure oriented towards the deep reinforcement learning training process is constructed. This provides a complete network and data management foundation with a clear operating mechanism for the subsequent parameter update based on interactive experience in step S300 and the output of vascular navigation candidate paths and continuous control sequences in step S400.

[0036] S300: Based on the vascular navigation training preparation structure, perform vascular navigation interactive rounds and compound reward calculation, parameter update processing, and generate the vascular navigation training convergence model structure. In this step, the system first loads the initialization results of the vascular navigation strategy network structure, the initialization structure of the vascular navigation strategy value network, the configuration data of the experience replay buffer, and the parameter setting fields of the soft target network from the vascular navigation training preparation structure output in the previous step S200. Simultaneously, it loads the vascular tree topology, the vascular centerline point set, local radius and curvature information, the safety corridor configuration fields, and the obstacle occupancy mask from the vascular environment state basic data structure output in step S100. Specifically, the system constructs the runtime context of the vascular navigation training task, binds the topology descriptions of the strategy network and value network recorded in the vascular navigation training preparation structure to the parameter storage buffer, initializes the capacity, field layout, and sampling weights of the experience replay buffer to the values ​​set in the configuration data, and maps the soft target network parameter setting fields to the target network update scheduling strategy. At the same time, based on the robot's initial pose field and target point configuration field in the vascular environment state basic data structure, the system constructs a simulated navigation environment instance, providing an environment interface for state generation and action execution in subsequent vascular navigation interaction rounds. This simulation environment instance supports reset and stepping operations. The reset operation generates the initial state near the centerline of the blood vessel based on the robot's initial pose field. The stepping operation propels the robot to move inside the blood vessel channel according to the input control commands, updates the robot's pose, and determines events such as collisions, boundary crossings, or reaching the endpoint. This step is called repeatedly during its execution.

[0037] During the vascular navigation interactive round execution phase, the system initiates multiple vascular navigation interactive rounds according to the number of training rounds and the maximum number of steps per round set in the vascular navigation training preparation structure. At the start of each interactive round, the system calls the reset interface of the simulation environment instance, reads the robot's initial pose and safety corridor configuration fields from the vascular environment state basic data structure, generates an initial vascular navigation state representation structure, and writes this state to a temporary state cache. Specifically, at each time step, the system extracts a state vector from the current vascular navigation state representation structure, inputs it into the online policy network corresponding to the policy network parameters and topology description recorded in the vascular navigation training preparation structure, and obtains the continuous control action for the current time step through forward propagation. This action consists of the propulsion along the vascular centerline, the radial offset within the cross section, and the turning amount around the centerline. The system performs an action validity check on this continuous action. When the action exceeds the action boundary recorded in the vascular navigation state action modeling structure, the action is pruned and the pruning event log is recorded. Subsequently, the system submits the trimmed control actions to the stepping interface of the simulation environment instance. The simulation environment updates the robot's position and orientation in the 3D blood vessel model based on the current robot pose, blood vessel centerline position, and safety corridor configuration fields. It detects whether the robot has crossed the blood vessel lumen, touched a danger zone, deviated from the safety corridor, reached the target point, or reached the maximum number of time steps set for this interaction round. When a collision or severe boundary violation event is detected, the simulation environment instance marks the time step as terminated, along with a collision type identifier. Similarly, when the robot enters the target area and meets the termination conditions, the time step is also marked as terminated, along with an arrival identifier. At the end of each time step, the simulation environment instance returns the updated blood vessel navigation status representation structure, termination flag, and auxiliary diagnostic fields. The system writes this information, along with the control actions used in the current time step, into the interaction buffer, awaiting subsequent composite reward calculations and experience playback buffer write operations.

[0038] In the composite reward calculation stage, the system calculates and weights the immediate reward components for each time step's data record in the interaction buffer, based on the composite reward definition designed for the vascular navigation task. Specifically, the system first determines whether the current time step has reached the target area based on the termination flag and target point configuration field. When an arrival event is detected, a target arrival reward component is generated for that time step; for time steps where the target has not been reached, this component is zero. Then, based on the collision type identifier, boundary crossing flag, and safe corridor deviation degree returned by the simulation environment instance, the system calculates the boundary crossing penalty component and the safe corridor deviation penalty component. When a collision occurs, the robot goes out of the vascular cavity, or contacts a sensitive area, the penalty component takes a larger negative value. When the robot approaches the safe corridor boundary but does not cross it, the safe corridor deviation penalty component increases proportionally according to the deviation distance. Furthermore, the system combines the centerline information and path length statistical buffer in the vascular environment state basic data structure to calculate the path efficiency component for each time step. This component reflects the degree to which the current path shrinks towards the target in a topological sense. When the robot advances along the direction closest to the shortest path, this component is larger; when the robot detours or retreats, this component decreases. Simultaneously, the system calculates a trajectory smoothness penalty component based on the differences in control actions across consecutive time steps. When the change in action between adjacent time steps is too large, the trajectory smoothness penalty component increases, thereby imposing constraints on overly drastic control behavior. After combining the aforementioned reward components with weighted coefficients, an instantaneous composite reward for that time step is formed. The system writes this instantaneous composite reward into the reward field of the corresponding record in the interaction buffer, realizing composite reward calculation at the time step granularity. Subsequently, according to the field layout set in the experience playback buffer configuration data, the system packages the vascular navigation state representation structure, control actions, instantaneous composite reward, and the subsequent vascular navigation state representation structure returned by the simulation environment for the current time step into a state transition record. This record is written to the current write index position in the experience playback buffer, and the write index and buffer fill status are updated according to the coverage strategy in the experience management configuration. When the experience playback buffer is not yet full, the system writes sequentially; when the buffer is full, the system replaces older or lower-priority records according to the preset coverage strategy and updates the sampling weight field, providing a basis for batch sampling in subsequent parameter update stages.

[0039] During the parameter update processing phase, when the number of valid records in the experience replay buffer reaches the minimum sampling threshold preset in the vascular navigation training preparation structure, the system initiates the network parameter update process. Specifically, the system first selects a batch of state transition records from the experience replay buffer using a weighted random sampling method, based on the sampling batch size and sampling weights set in the experience management configuration. The current vascular navigation state representation structure and control action vector from this batch of records are input into the online value network to obtain the current value estimate. The subsequent vascular navigation state representation structure from this batch of records is then input into the target policy network and the target value network. The target policy network generates subsequent control actions, and the target value network estimates the long-term value of the subsequent state and subsequent actions. A target value estimate is constructed by combining the immediate composite reward and the termination flag. Internally, the system constructs a loss metric based on the difference between the current value estimate and the target value estimate. This loss metric is used to calculate the gradient of each layer parameter of the online value network, and the value network parameters are fine-tuned according to the optimizer configuration and learning step size configuration recorded in the vascular navigation training preparation structure. After the value network parameters are updated, the system maintains the same sampling batch and inputs the current vascular navigation state representation structure into the online policy network. The long-term value corresponding to the actions output by the policy network is evaluated through the online value network. The policy network parameters are adjusted based on the direction of long-term value improvement, so that the control actions output by the policy network tend towards more suitable values ​​under the composite reward definition. After completing one round of value network parameter updates and policy network parameter updates, the system migrates some of the online policy network parameters and online value network parameters to the target policy network and target value network according to the update step size and update frequency recorded in the soft target network parameter setting field, performing a corresponding soft target network parameter update. When the number of training steps has not yet reached the maximum number of updates set in the vascular navigation training preparation structure, the system repeatedly executes the loop of experience sampling, value network update, policy network update, and soft target network update. In each training phase, it calculates the average composite reward, trajectory smoothness statistics, and collision count statistics for the most recent several rounds of interaction. When it is found that the change in these statistics is lower than a preset threshold or the number of training steps reaches the upper limit within several training phases, the current online policy network and online value network parameters, as well as the corresponding target network parameter states, are frozen and exported, encapsulated into a vascular navigation training convergence model structure. This vascular navigation training convergence model structure includes a policy network topology description, a value network topology description, weights of each network parameter, state preprocessing configuration, action postprocessing configuration, and constraint parameter descriptions related to the safety corridor. These are the output field names for this step and are used as input in step S400 to perform inference operations during the vascular navigation candidate path and continuous control sequence generation phase. Simultaneously, the safety constraint-related configuration and state preprocessing configuration work together with the vascular environment state basic data structure to guide the path generation and control command resampling process in step S400.

[0040] The technical effect of this step can be summarized as follows: by using the vascular navigation training preparation structure and the basic data structure of the vascular environment state to execute multiple rounds of vascular navigation interaction, calculating the immediate reward based on the composite reward design and writing it into the experience replay buffer, and then alternately updating the parameters of the value network and policy network on the basis of batch experience sampling and stabilizing the parameter transfer through the soft target network mechanism, a vascular navigation training convergence model structure is formed, which provides a trained and converged deep reinforcement learning model for subsequent path generation and control command output.

[0041] S400: Based on the vascular navigation training convergence model structure, perform vascular navigation candidate path and control sequence generation processing, and execute path filtering, control command resampling, and navigation deployment parameter packaging processing to generate vascular navigation deployment configuration structure. In this step, the system first loads the policy network topology description, value network topology description, layer parameter weights, state preprocessing configuration, and action postprocessing configuration from the vascular navigation training convergence model structure output in step S300. Simultaneously, it loads the vascular tree topology, vascular centerline point set, local radius and curvature fields, safety corridor configuration fields, obstacle occupancy mask, and robot initial pose and target point configuration fields from the vascular environment state basic data structure output in step S100. Understandably, the vascular navigation training convergence model structure serves as the model source for inference in this step, and the vascular environment state basic data structure serves as the environmental description source for path generation and evaluation. After loading these two types of structures, the system first constructs the navigation start state based on the robot initial pose field in the vascular environment state basic data structure, encoding it into a vascular navigation state representation structure consistent with the training phase. Then, based on the target point configuration field, it determines the position range of the target region in the vascular centerline parameter space and the corresponding safety corridor number, providing basic information for termination determination and candidate path classification in the subsequent path generation process.

[0042] Specifically, in the vascular navigation candidate path and control sequence generation stage, the system performs numerical normalization, missing field imputation, and field order alignment on the current vascular navigation state representation structure according to the state preprocessing configuration recorded in the vascular navigation training convergence model structure. The processed state vector is then input into the converged policy network for forward inference to obtain the continuous control actions at the current time step. These continuous control actions include a propulsion component along the vascular centerline, a radial offset component within the cross section, and a steering component around the centerline. The system performs amplitude pruning and constraint mapping on the policy network output according to the action value boundaries recorded in the vascular navigation state action modeling structure to obtain control action samples that meet safety constraints. In a candidate path generation process, the system starts with the robot's initial pose. At each time step, it inputs the current vascular navigation state representation structure into the policy network to obtain control actions. These control actions are then submitted to a simulation environment instance built on the vascular environment state data structure for step calculation. The environment instance updates the robot's pose based on the vascular centerline, vascular radius, safety corridor configuration, and obstacle occupancy mask, and returns a new vascular navigation state representation structure along with event flags such as whether a collision occurred, whether the robot crossed the safety corridor, whether it entered a sensitive area, and whether it reached the target area. When a collision or severe boundary violation event is detected, the system marks the current candidate path as invalid and terminates the generation of that candidate path. When the robot enters the target area and meets the termination condition, the system marks the candidate path as reached and ends the generation round. If the target area is not reached after reaching the preset maximum number of time steps, the system marks the candidate path as unreached. To obtain a diverse set of candidate paths, this step, under the premise of consistent initial state, can generate multiple vascular navigation candidate paths and control sequences in multiple independent runs by applying different perturbation seeds to the policy network output or using different random sampling strategies. The pose sequence, control command sequence, event flag sequence and path statistics field corresponding to each path are temporarily stored as a set of candidate paths to provide input for subsequent path selection processing.

[0043] During the path selection phase, the system verifies the safety and feasibility of each vascular navigation candidate path and control sequence in the candidate path set based on the safety corridor configuration field and obstacle occupancy mask in the vascular environment state basic data structure. Specifically, the system queries the corresponding vascular centerline position, safety corridor number, and distance from the vascular wall and sensitive area mask for each discrete pose point along the candidate path. When any pose point crosses the safety corridor boundary or enters a danger zone, the candidate path is determined to be a safety-unqualified path, and the corresponding position and reason fields are recorded. For safety-qualified candidate paths, the system further calculates the actual path length, the redundancy of the path in the topological space, and the deviation of the path from the shortest centerline path based on the vascular centerline point set and target point configuration field. It also calculates trajectory smoothing indicators based on the control command sequence recorded during the candidate path generation process, such as the statistical value of the change amplitude of control actions in adjacent time steps. The system comprehensively uses safety indicators, path length indicators, and trajectory smoothing indicators to construct a path evaluation record for each candidate path, eliminating candidate paths that do not meet preset threshold conditions, and generating a candidate path ranking list for candidate paths that meet the conditions according to a multi-indicator ranking strategy. In the ranking list, the top few highly-rated paths are marked as key candidate paths and written into an intermediate structure for subsequent control command resampling and navigation deployment parameter packaging. Thus, the path selection process maps the vascular navigation candidate paths and control sequence set into a set of vascular navigation screening paths and control sequence candidates that meet safety and geometric constraints, along with corresponding evaluation records.

[0044] During the control command resampling stage, the system resamples the control command sequence along the time axis and adjusts its amplitude based on the vascular navigation screening path and control sequence candidate set obtained during the path screening stage, according to the execution frequency, control bus bandwidth, and actuator response characteristics of the target robot platform. Specifically, the system first maps the time step information corresponding to the candidate path to a unified control time grid according to the standard control cycle of the robot control system. If the time resolution of the original control command sequence of the candidate path is higher than the target execution frequency, representative control commands are selected in several consecutive time steps according to a preset downsampling strategy, and the control trend at intermediate moments is compensated for when necessary through interpolation. If the time resolution of the original control command sequence is lower than the target execution frequency, supplementary commands are generated between adjacent commands through an interpolation strategy to make the control signal smoother and more continuous along the time axis. Subsequently, the system performs amplitude correction and rate of change constraint on the resampled control command sequence according to the physical capability constraints of the robot actuator. When the change in control quantity within adjacent control cycles exceeds the allowable change range of the actuator, the system prunes and gradually processes the corresponding command and records the pruning position in the control adjustment log. Based on this, the system can also quantize and encode the resampled control commands according to the mapping requirements of actual execution variables such as drive current, pressure, or flow rate, so as to conform to the command format of the robot controller. After the control commands are resampled, for each selected blood vessel navigation screening path and control sequence, the system obtains a control command resampled sequence that matches the target execution frequency, meets the actuator rate of change constraint, and has a clear timestamp. This sequence, along with the corresponding path geometry data and safety parameters, is stored in the deployment preparation buffer.

[0045] During the navigation deployment parameter packaging and processing phase, the system constructs a configuration description for the actual microrobot platform based on the geometric data of the vascular navigation screening path, the control command resampling sequence, and safety-related parameters stored in the deployment preparation buffer. Specifically, the system first converts the coordinates of the path points from the navigation coordinate system used in the simulation environment to the coordinate system used by the robot control system, according to the coordinate system rule in the basic data structure of the vascular environment state. At the same time, it confirms the attitude difference between the path start point and the robot's initial insertion position based on the target point configuration field and generates a path alignment offset parameter field. Subsequently, according to the robot control bus protocol and control command encoding specifications, the system maps the propulsion, offset, and steering control quantities in the control command resampling sequence to the underlying drive command sequence, constructs an instruction table containing timestamps, channel numbers, and amplitude codes, and generates a safety monitoring configuration field by combining the safety corridor number, safety threshold parameters, and abnormal event handling strategies. This field is used for real-time monitoring and emergency handling in the robot body or host computer during deployment. The system also maps the state preprocessing configuration and action postprocessing configuration in the vascular navigation training convergence model structure to preprocessing and postprocessing parameter fields in the online inference stage. This allows for the execution of policy network forward inference and necessary action postprocessing steps even under limited computing power in a real deployment environment. The aforementioned parameter fields, along with path geometry data and control command resampling sequences, constitute the vascular navigation deployment configuration structure. In this embodiment, this structure serves as the output field name for this step. It is written into the robot platform's configuration storage area through an interface with the surgical robot control system. Simultaneously, it is recorded within the system as a reference configuration source for the next round of vascular simulation environment data and the generation process of the robot's initial pose and target point configuration. This enables step S100 to read historical deployment configuration information when reconstructing the basic data structure of the vascular environment state, forming a closed-loop call relationship across steps.

[0046] The technical effect of this step can be summarized as follows: by performing vascular navigation candidate path and control sequence generation, path selection, control command resampling, and navigation deployment parameter packaging based on the vascular navigation training convergence model structure and the basic data structure of vascular environment state, the deep reinforcement learning model output obtained in the training stage is transformed into a vascular navigation deployment configuration structure that can be directly called on the micro-robot platform. The method as a whole realizes the connection from model training to online deployment, and maintains a coherent flow of path and control information between different steps through a unified data structure.

Claims

1. A blood vessel navigation path planning method based on deep reinforcement learning, characterized in that, include: Acquire vascular simulation environment data and robot initial pose and target point configuration, perform coordinate system I, safety corridor division and state action index binding processing, and generate vascular navigation state action modeling structure; Based on the vascular navigation state-action modeling structure, policy network configuration and value network construction are performed, and experience playback buffer and soft target network parameter setting are executed to generate the vascular navigation training preparation structure. Based on the vascular navigation training preparation structure, the vascular navigation interactive rounds and compound reward calculation and parameter update processing are executed to generate the vascular navigation training convergence model structure. Based on the vascular navigation training convergence model structure, candidate paths and control sequences for vascular navigation are generated and processed. Path selection, control command resampling, and navigation deployment parameter packaging are then performed to generate the vascular navigation deployment configuration structure.

2. The method according to claim 1, characterized in that, The process of implementing coordinate system one also includes: The coordinate system processing involves using the world coordinate system defined in the vascular simulation environment data as the navigation reference coordinate system, transforming the robot's initial pose and target point configuration to this navigation reference coordinate system, applying rigid body transformation to the position field in the robot's initial pose, and using a unified angle description reduction to convert the pose field into a unified pose expression. At the same time, the local coordinates in the target point configuration are mapped to the spatial position near the corresponding vascular centerline based on the vascular inlet mark and branch identifier, and the coordinate units, coordinate axis directions, and origin position are standardized to form a general spatial reference frame.

3. The method according to claim 1, characterized in that, The process of dividing safety corridors also includes: The safety corridor division process involves traversing the centerlines of each blood vessel from the vascular simulation environment data, generating a local cross-sectional description at each centerline point based on the corresponding radius and adjacent sensitive tissue labels, constructing several annular zones according to preset radial distance intervals, marking areas close to the centerline and far from the blood vessel wall and sensitive tissue mask as passable areas, and marking areas close to the blood vessel wall or entering the sensitive tissue mask as danger areas, connecting adjacent passable areas longitudinally to form continuous passable channel segments, and assigning a safety corridor number to each segment. At the same time, a mapping relationship is established between the corridor number and the branch node according to the vascular tree topology to record upstream and downstream connections.

4. The method according to claim 1, characterized in that, The process of state-action index binding also includes: The state-action index binding process includes mapping the local curvature and vessel diameter fields in the vascular navigation state representation structure to action channels related to steering amount, mapping the safety corridor boundary distance and danger zone distance fields to action channels related to radial offset amount, and mapping the target point orientation relationship and topological distance fields to action channels related to propulsion amount, according to the state-action correspondence rules. The index of each state field, the index of the corresponding action channel, and the fusion weight coefficient are recorded through an index table.

5. The method according to claim 1, characterized in that, The process of configuring the policy network and constructing the value network also includes: The policy network configuration and value network construction process includes configuring the number of input layer nodes of the policy network according to the length of the state vector of the vascular navigation state representation structure template to match the state dimension, and corresponding the input layer node indices according to the order of the state fields. At the same time, it configures the number of output layer nodes according to the number of action channels to correspond to the propulsion, offset, and steering control channels respectively. An output transformation unit is added to the output layer to perform amplitude pruning and range mapping to ensure that the control quantities fall within the action boundaries. Several hidden layers are selected for the policy network and the number of nodes is automatically given according to preset configuration rules. In the design of the hidden layers, nonlinear transformations are performed on key state fields through multi-branch feature extraction sub-layers, and feature fusion is performed in the middle and high layers. The value network construction process includes design state input branches and action input branches. The number of input layer nodes in the state input branch is consistent with the dimension of the state vector, and the number of input layer nodes in the action input branch is consistent with the number of action channels. After feature transformation, the state features and action features are concatenated at a certain hidden layer depth. Then, a scalar output is generated through a common sub-network. The state-action index relationship table is referenced to give the state fields describing safety and target proximity a higher connection priority.

6. The method according to claim 1, characterized in that, The process of executing the experience playback buffer and soft target network parameter setting also includes: Perform experience playback buffer and soft target network parameter setting processing; The experience replay buffer configuration process includes configuring the storage layout of state fields, action fields, immediate reward fields, and subsequent state fields for a single record based on the state vector dimension and the number of action channels, setting the maximum buffer capacity and write index rolling rules, and maintaining sampling weights for each record and dynamically adjusting them based on the immediate reward magnitude, value estimation bias, and number of samplings; The soft target network parameter setting process includes configuring update step size parameters and update frequency parameters for the target policy network and the target value network, and adopting an update mechanism that is uniformly triggered according to the number of training steps or adaptively adjusting the update step size according to the changes in the loss function.

7. The method according to claim 1, characterized in that, The process of performing the vascular navigation interactive round and compound reward calculation also includes: The vascular navigation interactive round processing includes constructing a simulated navigation environment instance based on the robot's initial pose field and target point configuration field in the vascular environment state basic data structure. It supports reset and stepping operations. The reset operation generates a starting state near the vascular centerline based on the robot's initial pose. The stepping operation propels the robot's movement, updates its pose, and detects collision, boundary crossing, or endpoint events based on input control commands. It also starts multiple interactive rounds according to the number of training rounds and the maximum number of steps per round. At each time step, the state vector is extracted from the current vascular navigation state representation structure and input into the online policy network to obtain continuous control actions. The legality of the actions is checked, and out-of-bounds actions are pruned before being submitted to the stepping interface. After updating, the new state and event flags are returned. The composite reward calculation process includes calculating the target arrival reward component based on the termination flag and target point configuration field for each time step, calculating the boundary violation penalty component based on the collision type identifier and boundary violation flag, calculating the safety corridor deviation penalty component based on the degree of safety corridor deviation, calculating the path efficiency component based on the centerline information and path length statistics, and calculating the trajectory smoothness penalty component based on the difference in control actions in consecutive time steps. These components are then combined into an instant composite reward using preset weighting coefficients.

8. The method according to claim 1, characterized in that, The parameter update process also includes: The parameter update process involves: when the number of valid records in the experience replay buffer reaches the minimum sampling threshold, a batch of state transition records is randomly selected from the buffer using weighted sampling. The current state and action are input into the online value network to obtain the current value estimate. The subsequent state is input into the target policy network and the target value network to obtain the long-term value estimate. The target value estimate is constructed by combining the immediate reward and the termination flag. The loss is calculated and the gradient of the online value network parameters is updated. Then, the current state is input into the online policy network with a fixed sample and the long-term value of the action is evaluated through the online value network to adjust the policy network parameters. Finally, the online network parameters are partially transferred to the target network according to the soft target network parameter settings.

9. The method according to claim 1, characterized in that, The process of generating candidate paths and control sequences for vascular navigation also includes: The process of generating candidate paths and control sequences for vascular navigation includes loading the robot's initial pose and target point configuration fields from the basic data structure of the vascular environment state, constructing the navigation start state and encoding it into a vascular navigation state representation structure, performing numerical normalization and field alignment according to the state preprocessing configuration, inputting it into a convergence strategy network to obtain continuous control actions, and performing amplitude clipping and constraint mapping on the actions. Then, in the simulation environment instance, the pose is updated step by step and collision, boundary crossing or target arrival events are detected. Through multiple rounds of independent operation, different perturbation seeds are applied to generate multiple candidate paths and control sequences, and the pose sequence, control command sequence and event flag sequence are temporarily stored.

10. The method according to claim 1, characterized in that, The process of executing path filtering also includes: The path selection process involves verifying the safety of each pose point for each path in the candidate path set based on the safety corridor configuration field and obstacle occupancy mask. Paths that cross the safety corridor boundary or enter a danger zone are marked as unqualified. For qualified paths, the actual path length, topological redundancy, deviation from the shortest centerline path, and trajectory smoothness index are calculated. Path evaluation records are constructed and sorted by multiple indicators to generate a candidate path ranking list.

Images