Action mask assisted obstacle avoidance target tracking method based on deep reinforcement learning

Abstract

The invention provides an action mask assisted obstacle avoidance target tracking method based on deep reinforcement learning, and the method comprises the steps: constructing simple to complex obstacle-target combinations based on an obstacle distribution type and a target motion type, and each combination corresponds to a training stage; in each training stage, according to an obstacle distribution type and a target motion type corresponding to the obstacle-target combination, determining an environment mode of the current stage, and further constructing a task execution map; the input of the main strategy network and the target network is robot observation information introduced with the action masks, and the optimal action output is generated after the action probability distribution output by the main strategy network and the target network is corrected by utilizing the action masks. By using the method, a reasonable motion strategy can be provided for the robot in an environment with a dynamic and static mixed obstacle and a variable-motion-mode target, the risk that the robot collides with the obstacle is reduced, and effective tracking of the target is realized and maintained.

Description

Technical Field

[0001] This invention relates to the field of single-robot target tracking technology, specifically to a target tracking method based on deep reinforcement learning with action mask-assisted obstacle avoidance. Background Technology

[0002] Robot target tracking in dynamic environments is one of the core challenges in the field of intelligent robots. Especially when the target's motion state is unknown and there are dense dynamic and static obstacles in the environment, traditional methods are difficult to achieve continuous and stable tracking results.

[0003] Current mainstream solutions are mainly divided into two categories: passive tracking and active tracking. Passive tracking methods (such as generative and discriminative models based on correlation filtering or deep learning) typically rely on fixed-viewpoint cameras, focusing on target detection and localization in image sequences. They cannot actively control sensor movement to cope with scenarios such as target occlusion or exiting the field of view, and the model's generalization ability is severely limited by the distribution of training data. In active tracking methods, visual servoing solutions adopt a decoupled architecture of perception and control, requiring separate optimization of the target localization module and motion controller. This results in problems such as large system latency and accumulation of hardware errors, making it difficult to meet real-time monitoring requirements.

[0004] In recent years, end-to-end active tracking methods based on deep reinforcement learning have significantly improved system response speed by directly mapping raw observation data to control commands. However, these methods still have key drawbacks: First, they lack effective obstacle avoidance constraints in environments with dense obstacles. Existing solutions rely on soft constraints such as reward function design or action space pruning, which are insufficient to completely avoid collision risks and result in poor training stability. Second, training efficiency is low; policies converge slowly in complex environments and are prone to getting trapped in local optima. Especially when the target's motion patterns are varied, existing single-stage training strategies struggle to balance exploration efficiency and policy generalization. These shortcomings lead to a significant decrease in the tracking success rate of existing methods in real dynamic environments, severely limiting their practical application value.

[0005] Ultimately, dynamic target tracking faces three major technical challenges: the coupling problem of real-time obstacle avoidance and motion planning, requiring robots to simultaneously handle multiple tasks such as target state estimation, obstacle avoidance, and path optimization within extremely short response cycles; the generalization requirement of environmental uncertainty, necessitating algorithms to adapt to uncontrollable factors such as the randomness of target motion, changes in obstacle spatial distribution, and sensor noise interference; and the dilemma of balancing training efficiency and policy robustness, as traditional reinforcement learning requires excessive interaction samples in complex tasks, resulting in poor training stability and difficulty in convergence. Therefore, an end-to-end solution that combines an efficient obstacle avoidance constraint mechanism with an adaptive training framework is urgently needed to improve the robustness of robot target tracking in open and dynamic environments. Summary of the Invention

[0006] This invention provides a target tracking method for obstacle avoidance based on action masking using deep reinforcement learning. It can provide a reasonable motion strategy for robots in environments with mixed dynamic and static obstacles and targets with varying motion patterns, reduce the risk of collisions between the robot and obstacles, and achieve and maintain effective tracking of the target.

[0007] A deep reinforcement learning-based action mask-assisted obstacle avoidance target tracking method includes: Based on obstacle distribution types and target motion types, obstacle-target combinations ranging from simple to complex are constructed, with each combination corresponding to a training phase. The training phases are implemented sequentially: In each training phase, the current environmental mode is determined based on the obstacle distribution type and target motion type corresponding to the obstacle-target combination; a task execution map is constructed based on the current environmental mode; and the robot is instructed to perform actions within the task execution map. The inputs to the main policy network and the target network are robot observation information with action masks incorporated. The action mask corresponds to each discrete action in the action space and is determined based on the obstacle situation in the region involved by the discrete action, indicating whether the discrete action is legal. After correcting the action probability distribution output by the main policy network and the target network using the action mask, the optimal action output is generated. After completing all training phases, the main policy network is directly deployed to the robot for target tracking.

[0008] Preferably, the action mask is obtained in the following way: A local polar coordinate system is established with the robot's center of mass as the origin, and detection sectors are divided at equal angular intervals in the radial direction; each discrete action corresponds to two adjacent detection sectors; If the distance between the nearest obstacle and the robot in two adjacent detection sectors is less than or equal to the safe distance, the corresponding discrete action is an unsafe action, and the corresponding action mask is marked as an illegal action; otherwise, the corresponding discrete action is a safe action, and the corresponding action mask is marked as a legal action. A stationary action is always considered a valid action, and the corresponding action mask is marked as valid.

[0009] Preferably, the step of modifying the action probability distributions output by the main policy network and the target network using action masks is as follows: If action Action mask represents action If illegal, then the actions output by the main policy network and the target network will be... The original action evaluation value was modified to negative infinity; If action Action mask represents action If the action is valid, then the actions output by the main policy network and the target network are valid. The original motion evaluation value remains unchanged.

[0010] Preferably, the action spaces of the main policy network and the target network include k The discrete actions that a robot can perform are denoted as . Each discrete action corresponds to a rotation angle. and a linear shift The combination of; the rotation angle The linear movement is selected within the range of angles that the robot can make in one time step; the linear movement includes three types of linear movement: forward, backward, and stationary.

[0011] Preferably, the rotation angle The selection within the range of angles that the robot can make in one time step is specifically as follows: taking the robot's forward direction as 0 degrees, the rotation angle is... Take 0, , .

[0012] Preferably, the robot is equipped with radar to continuously collect environmental obstacle information, and the radar outputs the distance value of the nearest obstacle in each sector in real time, which is used to determine the nearest obstacle in two adjacent detection sectors.

[0013] Preferably, the obstacle-target combination constructed from simple to complex based on obstacle distribution type and target motion type is as follows: The obstacle distribution types range from simple to complex and include three types: unobstructed, static obstacles, and static-dynamic obstacles. The target motion types range from simple to complex and include three types: movement along a fixed trajectory, navigation, and random movement.

[0014] Preferably, the obstacle-target combinations obtained from simple to complex are as follows: Accessibility + movement along a fixed trajectory, static obstacles + movement along a fixed trajectory, static obstacles + navigation, static and dynamic obstacles + navigation, static and dynamic obstacles + random movement.

[0015] Preferably, in each training phase, the current phase environment mode is determined based on the obstacle distribution type and target movement type corresponding to the obstacle-target combination: Step a1: The pre-set grid map is divided into free areas and obstacle areas. Free areas include three types: roads, activity areas, and grass. Obstacle areas include buildings and fixed obstacles. Step a2: Set up obstacles according to the obstacle distribution type: If the obstacle distribution type is unobstructed, then there is no need to generate random obstacles; If the obstacle distribution type is static obstacle type, the center position, size and shape of the static obstacle will be randomly generated within the activity area and grassland, and the corresponding map area will be set as the obstacle area. If the obstacle distribution type is dynamic or static, the center position, size, and shape of the static obstacles are randomly generated within the activity area and grassland, and the corresponding map area is set as the obstacle area; then, an indefinite number of dynamic obstacles are randomly generated within the free area of ​​the map and their initial positions are set as obstacle areas; the dynamic obstacle movement strategy includes two types, specifically: random movement type dynamic obstacles randomly select to move to any position in the nine-square grid centered on themselves at each time step; fixed trajectory movement type dynamic obstacles move at a constant speed along a preset path in the map; Step a3: Set the target based on the target motion type: If the target's movement type is to move along a fixed trajectory, the target's starting position is any point on the preset trajectory and the initial direction of movement is randomly determined. After that, the target moves at a constant speed along the preset trajectory. After completing one full rotation, it is randomly selected whether to change the direction of movement. If the target's motion type is navigation, then the target's initial position and navigation point are randomly generated within the free area of ​​the map, and the target moves towards the navigation point under the guidance of the artificial potential field; If the target's movement type is random movement, then at each time step, the target randomly selects to move to any position in the 3x3 grid centered on itself.

[0016] Preferably, the inputs to the main policy network and the target network are:

[0017] in, and Given the distance and angle of the target relative to the robot, This is the action mask for the current state. for k 3D data, corresponding to the action space k A discrete action.

[0018] Beneficial effects: (1) This invention provides a target tracking method for obstacle avoidance robots based on action masking using deep reinforcement learning, and innovatively constructs an end-to-end tracking decision framework. This framework directly maps the environmental perception state to robot control commands through a deep reinforcement learning model, integrating the perception and decision-making processes, and effectively solves the problems of poor real-time performance and weak generalization ability caused by the decoupling of target tracking and motion control in traditional methods. In particular, a highly realistic training environment is constructed by finely modeling the target motion pattern and complex obstacle distribution, which, together with the training mechanism, significantly improves the algorithm's adaptability to the uncertainty of the outdoor environment.

[0019] (2) This invention provides a target tracking method for obstacle avoidance robots based on action masking using deep reinforcement learning. It innovatively introduces an action masking mechanism to ensure the robot's operational safety in dense obstacle environments. By using multi-directional LiDAR to detect obstacle distances in real time and combining this with predefined safety thresholds, action masks are applied to movement directions that may lead to collisions, eliminating collision risks from the source of the strategy. Compared to traditional "soft constraint" methods, this mechanism avoids the drawbacks of complex obstacle avoidance rule design and poor generalization, reducing the collision rate to an extremely low level in dynamic obstacle scenarios while ensuring tracking continuity, providing hardware-level safety assurance for monitoring tasks.

[0020] (3) This invention provides a target tracking method for obstacle avoidance robots based on action masking using deep reinforcement learning. It innovatively proposes a multi-stage course learning strategy, significantly optimizing the algorithm's training efficiency and convergence performance. Addressing the difficulty in convergence caused by the coupling of target tracking and obstacle avoidance tasks, the task is decomposed into five progressive stages with seamless switching between them. This strategy enables the algorithm to quickly master basic tracking capabilities in the early stages and gradually adapt to more challenging scenarios. Compared to direct training, it significantly improves convergence speed and performance, effectively solving the problems of high training costs and susceptibility to local optima in complex tasks. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of the action mask-assisted obstacle avoidance robot target tracking method based on deep reinforcement learning provided by the present invention; Figure 2 This is an overall architecture diagram of the deep reinforcement learning model provided by the present invention; Figure 3 This is a schematic diagram of the radar detection sector and action space of the present invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0026] This invention provides a target tracking method for obstacle avoidance robots based on action masking using deep reinforcement learning. It can provide robots with reasonable motion strategies in environments with mixed dynamic and static obstacles and targets with varying motion patterns, reduce the risk of collisions between the robot and obstacles, and achieve and maintain effective target tracking.

[0027] This paper takes the robot target tracking problem in the supervision of minors in urban outdoor environments (city parks or public green spaces within residential communities) as an implementation example. A two-dimensional planar map is constructed. Its boundary is The map shows a rectangular area containing multiple static and dynamic obstacles, a tracking target, and a monitoring robot. The robot's maximum observation distance and maximum observation angle are also shown. and Constitutes the local target's perceptible range The 16-directional lidar array distributed around the chassis circumference forms the obstacle detection range. The information available includes: Real-time location information, target location information, and obstacle distance information.

[0028] like Figure 1 and Figure 2 As shown in the figure, an embodiment of the present invention provides a target tracking method for obstacle avoidance robots based on deep reinforcement learning and action mask assistance, comprising the following steps: S0: Based on obstacle distribution type and target motion type, construct obstacle-target combinations from simple to complex, with each combination corresponding to a training phase.

[0029] Obstacle distribution types range from simple to complex, including three types: no obstacles, static obstacles, and static-dynamic obstacles; target movement types range from simple to complex, including three types: movement along a fixed trajectory, navigation, and random movement.

[0030] By combining obstacle distribution types with target motion types, various obstacle-target combinations can be obtained. In practice, some combinations are not essential. After simplification and optimization, this invention employs five obstacle-target combinations, as shown in Table 1. Each obstacle-target combination corresponds to a training stage. Subsequent steps of this invention require the sequential implementation of each training stage. Each training stage determines the current stage's environmental mode based on the obstacle distribution type and target motion type corresponding to the obstacle-target combination, thereby constructing a targeted task execution map.

[0031] Table 1. Obstacle-target combinations for the 5 training phases

[0032] Step S1: Based on the current training phase, create a two-dimensional discrete environment containing multiple dynamic and static obstacles, a tracking target, and a robot as a task execution map.

[0033] Specifically, the task execution map is constructed as follows: S11: Construct an initial map based on the preset grid map information and distinguish different map area categories. The grid map includes free areas and obstacle areas. The preset grid map information includes three types of free areas: roads, activity areas, and grass, labeled 1, 2, and 3 respectively; obstacle areas include buildings and fixed obstacles, labeled 4 and 5 respectively.

[0034] S12: Set up the environment mode based on the training phase and generate a complete task execution map. Environment mode setup consists of two steps: obstacle setting and target setting. The specific steps are as follows: (1) The specific rules for setting up obstacles are as follows: If the obstacle distribution type is "unobstructed", then there is no need to generate random obstacles; If the obstacle distribution type is "static obstacle", then the center position, size and shape of the static obstacle will be randomly generated within the map activity area and grassland area, and the corresponding map area will be set as the obstacle area. If the obstacle distribution type is "dynamic and static obstacles", the center position, size, and shape of static obstacles are randomly generated within the activity area and grassland, and the corresponding map area is set as the obstacle area. Then, an indefinite number of dynamic obstacles are randomly generated within the free area of ​​the map and their initial positions are set as obstacle areas. The movement strategies of dynamic obstacles include two types: random movement type dynamic obstacles randomly select to move to any position in the nine-square grid centered on themselves at each time step; fixed trajectory movement type dynamic obstacles move at a constant speed along a preset rectangular path in the map.

[0035] (1) The specific rules for setting goals are as follows: If the target's movement type is "moving along a fixed trajectory", the target's starting position is any point on the preset trajectory and the initial direction of movement (clockwise or counterclockwise) is randomly determined. After that, the target moves at a constant speed along the preset trajectory. After completing one revolution, it is randomly selected whether to change the direction of movement. If the target's movement type is "random movement", then at each time step, the target randomly selects to move to any position in the 3x3 grid centered on itself; If the target's movement type is "navigation," then the initial position and navigation point of the target are randomly generated within the free area of ​​the map. The target moves towards the navigation point under the guidance of an artificial potential field. The specific process is as follows: based on the relative distance, obstacles in the map exert a repulsive force on the target, and the navigation point exerts an attractive force on the target. The target moves a fixed length along the direction of the resultant force and may use a random movement strategy to escape the local optimum with a certain probability. It is then determined whether the navigation point has been reached. If not, the movement under the guidance of the artificial potential field continues. If the navigation point has been reached, a new navigation point is randomly generated in the reachable area of ​​the map, and the movement under the guidance of the artificial potential field continues.

[0036] (3) Randomly generate the initial position of the monitoring robot within a certain radius of the target in the free area of ​​the complete task execution map, with the initial direction pointing towards the tracking target.

[0037] Step S2: Initialize the robot's main policy network and target network. Both receive robot observation information as input. The outputs are all discrete actions with the highest value based on the current observation information. Once the main policy network has been trained, it can be directly deployed to the robot for environmental interaction, while the target network is only used for training.

[0038] Step S3: The main policy network and the target network input the observation information. Calculate the probability distribution of actions And output the optimal action. The specific content is as follows: S31: In this embodiment, the robot's executable action space includes 11 discrete actions, denoted as... Each action corresponds to a predefined high-level motion instruction, which specifically defines the robot's three types of linear movement within a control cycle: forward, backward, and stationary. and the accompanying predefined rotation angle The above correspondence is stored through a predefined lookup table. Under the premise of ensuring that the robot's kinematic constraints and driving capabilities are met, the continuous control space is discretized into this finite set of actions, thus maintaining a balance between the necessary control accuracy and the ability to explore the environment.

[0039] Among them, rotation angle The selection is made within the range of angles that the robot can make in one time step. This embodiment presents a preferred method.

[0040] like Figure 3 As shown, a 16-axis lidar array mounted on a mobile robot continuously collects information about environmental obstacles. The lidar establishes a local polar coordinate system with the robot's center of mass as the origin, and coordinates the axes in the radial direction as follows: The system is divided into 16 detection sectors at equal angular intervals, and the distance to the nearest obstacle in each sector is output in real time. , The sectors are numbered and distributed clockwise based on the orientation of the first radar on the left front side of the robot.

[0041] So the rotation angle You can take 0, , See the blue, green, and red arrows in the diagram. There are 5 rotation angles. With 3 linear moves After combining, 11 discrete actions can be obtained as shown in Table 2 below.

[0042] Table 2 Eleven discrete actions in the action space

[0043] S32: Input of observation information The specific components are:

[0044] in, and The distance and angle of the target relative to the robot have been normalized. This is the action mask for the current state, which is 11-dimensional data corresponding to 11 discrete actions.

[0045] Specifically, the action mask in the current state The generation method is as follows: ①Represent any discrete action Associate two adjacent radar sectors according to their direction of motion. and ,cover Azimuth range, stationary motion Processed independently, not associated with any specific sector. For example... Figure 3 As shown, the action corresponding to the first green arrow in the left front direction corresponds to the two adjacent radar sectors #1 and #2.

[0046] In an example of this invention, the specific relationship is as follows:

[0047] in, Number the action. This is the associated sector number.

[0048] ② Set a safe distance threshold For any action If the distance between the robot and the nearest obstacle in either of its two associated sectors is less than or equal to the safe distance threshold. That is, based on the distance to the nearest obstacle within the sector obtained by radar, it satisfies Then the action is determined. An illegal action is denied the judgment as a legal action; a stationary action is always judged as a legal action.

[0049] Generate a binary mask vector based on the judgment result. ,in

[0050] In this context, 1 and 0 are both indicators of whether the action is legal or not. In practice, they can be expressed in other ways, and are not limited to these.

[0051] S33: After introducing the action masking mechanism, the main policy network and the target network output action evaluation values. This invention uses action masking to correct the action evaluation values ​​and uses the corrected action probability distribution to generate the optimal action output.

[0052] In this step, a masking function is used to correct the original action evaluation values ​​output by the main policy network and the target network. The specific masking function is as follows:

[0053] In the above formula, This is the corrected motion evaluation value. The above formula indicates that if the motion... Action mask represents action If illegal, then the actions output by the main policy network and the target network will be... The original action evaluation value was modified to negative infinity. If the action Action mask represents action If the action is valid, then the actions output by the main policy network and the target network are valid. The original motion evaluation value remains unchanged.

[0054] Then, the action probability distribution is renormalized. The specific calculation method for the probability distribution corresponding to a certain action is as follows:

[0055] Finally, the softmax function is used to filter out illegal actions and output the optimal action. .

[0056] Step S4: Calculate the reward function in state S4 using the set reward function. Next action Corresponding rewards And obtain the state at the next moment. And record the termination marker. and the main policy network output The six-tuple Stored in experience replay pool B.

[0057] In this embodiment, the designed reward function is as follows:

[0058] in, and These represent the mainline sparsity penalties when a collision occurs and when the target is lost, respectively. For single-step dense reward with amplitude limitations during normal tracking:

[0059] in, and The actual distance and angle of the target relative to the robot. and Given the target's desired tracking distance and angle relative to the robot, and The robot's furthest observation distance and maximum observation angle; is a coefficient.

[0060] Step S5: Update the map dynamic variables for the next moment based on the above information and environmental infrastructure settings, including robot state, target state, and dynamic obstacle state.

[0061] Step S6: Sample multiple six-tuples from the experience replay pool B in batches as training samples, adopt the course learning mode, calculate the KL divergence loss through the distributed value function, and use the Rainbow algorithm to update the parameters of the main policy network and the target network.

[0062] Step S7: Determine if the overall training process has ended. If it has, proceed to step S8; otherwise, determine if the current training phase has ended. If it has, switch training phases, return to step S1, and rebuild the task execution map based on the environment mode corresponding to the new training phase to begin training for the new phase. If the current training phase has not yet ended, return to step S3 to continue training for the current phase.

[0063] Among them, determining whether the overall training process has ended can be done by checking whether the current iteration count has reached the upper limit of the total number of training generations. To determine whether the current training phase has ended, it is possible to determine whether the maximum number of algebraic steps in the current course training phase has been reached. ( Number the current training phase. (Total number of training phases).

[0064] As mentioned above, the course learning and training model proposed in this invention is divided into 5 stages, namely... The environmental complexity increases progressively, with the same number of training iterations per stage. The complexity of obstacle distribution types is ranked as follows: none - static - dynamic; the complexity of target movement types is ranked as follows: movement along a fixed trajectory - navigation - random movement. Initial stage ( The simplest type combination is (obstacle-free + target moving along a fixed trajectory). Each switch alternately changes one dimension setting, with the first switch changing the obstacle distribution type, until the final training phase. The most difficult combination is (dynamic obstacles + randomly moving targets). Step S8: After training is complete, output the robot's main policy network as the robot target tracking model.

[0065] Step S9: During the execution phase, before each action, the robot will call the target tracking model to obtain the optimal safe action in the current state.

[0066] To further illustrate the effectiveness of the provided method, the target tracking method provided by this invention is tested under different target motion modes and different obstacle distribution types. Simulation experiments were conducted on a two-dimensional map. The algorithm ran for 100 rounds in each environment, with a maximum running length of 500 steps per round. The agent achieved superior and stable performance in terms of tracking success rate, average reward, and average round length, and can successfully complete the tracking task while effectively avoiding dynamic obstacles in the environment.

[0067] Based on the above experiments, the action mask-assisted obstacle avoidance robot target tracking method based on deep reinforcement learning provided by this invention can provide robots with efficient and robust motion strategies in dynamic environments, ensuring the safety and reliability of target tracking.

[0068] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A target tracking method for obstacle avoidance based on action masking using deep reinforcement learning, characterized in that, include: Based on obstacle distribution types and target motion types, obstacle-target combinations ranging from simple to complex are constructed, with each combination corresponding to a training phase. Implement each training phase sequentially: In each training phase, determine the current phase's environmental pattern based on the obstacle distribution type and target movement type corresponding to the obstacle-target combination; and construct a task execution map based on the current environmental pattern. Instruct the robot to move within the task execution map; The inputs to the main policy network and the target network are robot observation information with action masks incorporated. The action mask corresponds to each discrete action in the action space and is determined based on the obstacle situation in the region involved by the discrete action, indicating whether the discrete action is legal. After correcting the action probability distribution output by the main policy network and the target network using the action mask, the optimal action output is generated. After completing all training phases, the main policy network is directly deployed to the robot for target tracking.

2. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 1, characterized in that, The method for obtaining the action mask is as follows: A local polar coordinate system is established with the robot's center of mass as the origin, and detection sectors are divided at equal angular intervals in the radial direction; each discrete action corresponds to two adjacent detection sectors; If the distance between the nearest obstacle and the robot in two adjacent detection sectors is less than or equal to the safe distance, the corresponding discrete action is an unsafe action, and the corresponding action mask is marked as an illegal action; otherwise, the corresponding discrete action is a safe action, and the corresponding action mask is marked as a legal action. A stationary action is always considered a valid action, and the corresponding action mask is marked as valid.

3. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 1, characterized in that, The modification of the action probability distribution output by the main policy network and the target network using action masks is as follows: If action Action mask represents action If illegal, then the actions output by the main policy network and the target network will be... The original action evaluation value was modified to negative infinity; If action Action mask represents action If the action is valid, then the actions output by the main policy network and the target network are valid. The original motion evaluation value remains unchanged.

4. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in any one of claims 1-3, characterized in that, The action spaces of the main policy network and the target network include k The discrete actions that a robot can perform are denoted as . Each discrete action corresponds to a rotation angle. and a linear shift The combination of; the rotation angle The linear movement is selected within the range of angles that the robot can make in one time step; the linear movement includes three types of linear movement: forward, backward, and stationary.

5. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in any one of claims 4, characterized in that, The rotation angle The selection within the range of angles that the robot can make in one time step is specifically as follows: taking the robot's forward direction as 0 degrees, the rotation angle is... Take 0, , .

6. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 2, characterized in that, The robot is equipped with radar to continuously collect information on environmental obstacles. The radar outputs the distance value of the nearest obstacle in each sector in real time, which is used to determine the nearest obstacle in two adjacent detection sectors.

7. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 1, characterized in that, Based on obstacle distribution type and target motion type, obstacle-target combinations ranging from simple to complex are constructed as follows: The obstacle distribution types range from simple to complex and include three types: unobstructed, static obstacles, and static-dynamic obstacles. The target motion types range from simple to complex and include three types: movement along a fixed trajectory, navigation, and random movement.

8. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 7, characterized in that, The resulting obstacle-target combinations, from simple to complex, are as follows: Accessibility + movement along a fixed trajectory, static obstacles + movement along a fixed trajectory, static obstacles + navigation, static and dynamic obstacles + navigation, static and dynamic obstacles + random movement.

9. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 7, characterized in that, In each training phase, the current environmental mode is determined based on the obstacle distribution type and target motion type corresponding to the obstacle-target combination: Step a1: The pre-set grid map is divided into free areas and obstacle areas. Free areas include three types: roads, activity areas, and grass. Obstacle areas include buildings and fixed obstacles. Step a2: Set up obstacles according to the obstacle distribution type: If the obstacle distribution type is unobstructed, then there is no need to generate random obstacles; If the obstacle distribution type is static obstacle type, the center position, size and shape of the static obstacle will be randomly generated within the activity area and grassland, and the corresponding map area will be set as the obstacle area. If the obstacle distribution type is dynamic or static, the center position, size, and shape of the static obstacles are randomly generated within the activity area and grassland, and the corresponding map area is set as the obstacle area; then, an indefinite number of dynamic obstacles are randomly generated within the free area of ​​the map and their initial positions are set as obstacle areas; the dynamic obstacle movement strategy includes two types, specifically: random movement type dynamic obstacles randomly select to move to any position in the nine-square grid centered on themselves at each time step; fixed trajectory movement type dynamic obstacles move at a constant speed along a preset path in the map; Step a3: Set the target based on the target motion type: If the target's movement type is to move along a fixed trajectory, the target's starting position is any point on the preset trajectory and the initial direction of movement is randomly determined. After that, the target moves at a constant speed along the preset trajectory. After completing one full rotation, it is randomly selected whether to change the direction of movement. If the target's motion type is navigation, then the target's initial position and navigation point are randomly generated within the free area of ​​the map, and the target moves towards the navigation point under the guidance of the artificial potential field; If the target's movement type is random movement, then at each time step, the target randomly selects to move to any position in the 3x3 grid centered on itself.

10. The action mask-assisted obstacle avoidance target tracking method based on deep reinforcement learning as described in claim 1, characterized in that, The inputs to the main policy network and the target network are: in, and Given the distance and angle of the target relative to the robot, This is the action mask for the current state. for k 3D data, corresponding to the action space k A discrete action.

Images