Efficient strategy transfer multi-robot cooperative tracking method and system based on enhanced action space

Abstract

The invention discloses an efficient strategy transfer multi-robot cooperative tracking method and system based on an enhanced action space. Aiming at the problems of low strategy migration efficiency, complex credit distribution and insufficient long decision chain performance in a dynamic environment in the existing method, the method simplifies the learning process by modeling an expert strategy into a duration-adjustable macro action and fusing the duration-adjustable macro action with an original action space to form an enhanced action space. On the basis of a hierarchical reinforcement learning framework (HRL), an inter-macro-action intra-learning rule (IMALR) is introduced, and multiple training samples are extracted during macro action execution so as to improve learning efficiency; an internal reward mechanism is designed to encourage an intelligent agent to preferentially select a long-time effective macro action, so that strategy multiplexing is enhanced; and meanwhile, dynamic strategy switching is supported, and a better action is allowed to be interrupted and selected according to environment change in macro action execution. According to the method, the cooperative efficiency of multiple robots in a dynamic tracking task is remarkably improved, and the problems of obstacle avoidance, target offset and credit distribution of a traditional method in a complex environment are solved.

Description

Technical Field

[0001] The present invention relates to an efficient policy transfer multi-robot collaborative tracking method and system based on an enhanced action space, especially applied to multi-agent collaborative tracking tasks, and particularly to the technical field of optimizing policy transfer and action selection efficiency in reinforcement learning (RL). Background Art

[0002] With the rapid development of artificial intelligence and unmanned aerial vehicle (UAV) technology, a single UAV faces great challenges in performing tasks due to limitations in battery life, payload capacity, and sensing range, and it is difficult to efficiently complete complex tasks. To overcome these limitations, multi-UAV systems (UAS) have emerged, which can improve task execution efficiency and system robustness through the collaborative work of multiple UAVs.

[0003] Existing UAV swarm control methods can be mainly divided into two categories: centralized and distributed. Centralized control relies on a central node for decision-making, but when the central node fails, the entire system will collapse, affecting the stability of the system. Distributed control, on the other hand, adopts a decentralized strategy, and each UAV makes autonomous decisions based on local information, thereby enhancing the adaptability and stability of the system. During collaborative flight and task execution, existing technologies rely on known environmental information, and in the face of unknown environments, traditional methods are difficult to provide an efficient balance between obstacle avoidance and task target tracking. The execution of multi-robot collaborative tasks is also affected by problems such as environmental dynamic changes, task target offsets, and poor swarm coordination, resulting in low task completion efficiency and unstable system performance.

[0004] In addition, existing reinforcement learning (RL) methods still have some deficiencies in dealing with multi-task and multi-policy transfer problems. Especially in complex environments, how to efficiently transfer policies from multiple expert policies and how to efficiently search for the optimal policy in a wide state space are still the current research difficulties. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes an efficient policy transfer multi-robot collaborative tracking method based on an enhanced action space, aiming to improve the learning efficiency, adaptability, and execution ability of multi-robot systems in complex collaborative tracking tasks through macro-action and multi-policy transfer technologies.

[0006] Method design idea: The present invention simplifies the learning process of robots by converting expert policies into macro-actions and fusing these macro-actions with the original action space. The execution time of a macro-action can cover multiple time steps, which enables the agent to avoid making decisions at each time step, thereby reducing the credit assignment problem in long decision chains. The selection of macro-actions is based on the reuse of expert policies, enabling the robot to dynamically select the optimal policy according to the current task stage.

[0007] Assumptions and Simplifications: When the expert strategy is effective in a certain state, it also has good effects in the neighboring states of that state. Therefore, the robot does not need to evaluate the effectiveness of the expert strategy at every moment, but can judge in stages which expert strategies are the most valuable in the current stage.

[0008] This method adopts a hierarchical reinforcement learning framework (HRL), models the expert strategy as a macro-action, allows the robot to execute the same expert strategy within multiple time steps, and does not decide whether to switch to another macro-action until the macro-action is executed.

[0009] Core Technical Solutions: Macro-action Modeling and Action Space Enhancement: The present invention transforms each expert strategy into multiple macro-actions with different durations. For example, the artificial potential field method and the "wall-following" rule can be used as macro-actions, each with a different execution time. In this way, the agent can explore the environment more efficiently when executing the macro-action, reduce the decision-making frequency, and improve the decision-making quality.

[0010] Intrinsic Reward Mechanism: To encourage the agent to use long-duration macro-actions in the task, the present invention introduces an intrinsic reward proportional to the execution time of the macro-action. This reward mechanism enables the robot to more frequently select effective long-duration macro-actions, thereby obtaining better cooperation effects in the task.

[0011] Inter-Macro-Action Learning Rule (IMALR): The present invention introduces an inter-macro-action learning rule, which solves the limitation of the traditional method that only a single training sample can be extracted during the execution of the macro-action. By adjusting the temporal difference target of the macro-action, the method can extract multiple training samples during the execution of each macro-action, significantly improving the learning efficiency.

[0012] Action Selection and Policy Switching Mechanism: In the multi-robot cooperative tracking task, the agent needs to judge whether the current macro-action has been executed. If the macro-action is executed, the robot will select the next optimal macro-action from the enhanced action space; if the macro-action is not executed, the current macro-action will continue. In particular, the present invention introduces a "macro-action interruption" mechanism. When a higher Q value is found for other actions during the execution of the macro-action, the current macro-action can be immediately terminated and the optimal action can be reselected. This mechanism can effectively respond to environmental changes and improve the adaptability of the robot in a dynamic environment.

[0013] Specific Execution Process: The robot first obtains the current environmental state through local perception and selects an appropriate macro-action according to this state.

[0014] At each time step, the robot evaluates the values of all actions (including macro actions and primitive actions) in the action space according to the deep Q-network and selects the optimal action to execute.

[0015] During the execution process, if it is found that the execution time of the current macro action is too long and the effect is not good, then through the macro action interruption mechanism, switch to other strategies.

[0016] During the execution of macro actions by the robot, the intrinsic reward mechanism guides it to more frequently select macro actions that are effective for a long time, improving the reuse efficiency of strategies.

[0017] Advantages of the method: Efficient strategy reuse and migration: By converting expert strategies into macro actions, the robot can efficiently reuse existing strategies, avoiding the limitations of relying on a single expert strategy in traditional methods. Multi-strategy migration enhances the robot's ability to handle complex environments and improves the success rate of task completion.

[0018] Enhanced exploration efficiency: The introduction of macro actions reduces the decision-making frequency, enabling the robot to find the optimal strategy more quickly in complex environments, especially in long-term tasks.

[0019] Improved learning efficiency and generalization ability: By enhancing the action space, the robot can not only converge to the optimal strategy faster but also effectively cope with different environmental changes, achieving better generalization ability.

[0020] Specific application scenarios: The present invention is particularly suitable for handling complex tasks in dynamic environments, such as encirclement, obstacle avoidance, and dynamic target tracking, in multi-robot cooperative tracking tasks. In a multi-robot cluster, each robot selects the most appropriate macro action for cooperation according to the current task stage and environmental conditions, thereby achieving efficient encirclement of the target.

[0021] The beneficial effects of the method, system, and device of the present invention are as follows: By converting expert strategies into macro actions and integrating them into the primitive action space, the efficiency of multi-robot cooperative tracking tasks is significantly improved. The introduced intrinsic reward mechanism encourages the use of macro actions that are effective for a long time, enhancing the frequency of strategy reuse, improving learning efficiency, and the success rate of tasks. Through multi-expert strategy migration and the in-macro-action learning rule (IMALR), this method improves the generalization ability of tasks, enabling the robot to adapt to dynamic and complex environments. In addition, the strategy switching mechanism and attention mechanism based on the enhanced action space make the robot more flexible and efficient during cooperation, avoiding collisions and optimizing task execution. Overall, the present invention demonstrates excellent learning ability, adaptability, and stability in multi-robot cooperation and can complete efficient cooperative tracking tasks in complex environments. Brief Description of the Drawings

[0022] Figure 1 It is a flowchart of an efficient policy transfer multi-robot cooperative tracking method based on an enhanced action space. The deep Q-network outputs action values for all original actions and macro actions.

[0023] Figure 2 It is a comparison of the action space structures of each method.

[0024] Figure 3 It is an efficient policy transfer multi-robot cooperative tracking method based on an enhanced action space.

[0025] Figure 4 It is the different macro-action termination methods of proposed and CAPS.

[0026] Figure 5 It is the training environment of an efficient policy transfer multi-robot cooperative tracking method based on an enhanced action space.

[0027] Figure 6 It is a test scenario for verifying the generalization ability of each method to different obstacles. Specific implementation manners

[0028] The following further elaborates the present invention in detail with reference to the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of elaboration and explanation, and no limitation is imposed on the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0029] The present invention provides an efficient policy transfer multi-robot cooperative tracking method based on an enhanced action space, and the method includes the following steps: In this embodiment, for the multi-robot cooperative tracking task, an efficient policy transfer method based on an enhanced action space is adopted. This method models the expert policy as macro actions and integrates them into the original action space, and uses a hierarchical reinforcement learning framework to improve the learning efficiency and generalization ability.

[0030] The system of this embodiment includes the following key components: Deep Q-network: It is used to output the action values of all original actions and macro actions within the enhanced action space. The red nodes represent the original actions, and the green and yellow nodes respectively represent the macro actions of the "wall-following" rule and the artificial potential field method.

[0031] Inner learning rule of macro actions (IMALR): This rule is used to adjust the temporal difference target of macro actions to ensure that multiple training samples can be effectively extracted during the execution of macro actions, thereby improving the learning efficiency.

[0032] Macro-action Reward: This reward mechanism is related to the execution time of macro-actions, aiming to enhance the usage frequency of useful long-execution-time macro-actions and promote the exploration of the structured environment by the agent.

[0033] Method Steps

[0034] Action Space Enhancement: According to the task requirements, multiple expert policies are modeled as macro-actions, and these macro-actions are fused with the original actions to form an enhanced action space. The execution time of each macro-action is uncertain and can cover multiple time steps.

[0035] Training Process: At each time step, the agent evaluates all actions in the enhanced action space through a deep Q-network and selects the optimal macro-action for execution. The execution time of the macro-action is an adjustable parameter, and the agent adjusts the execution duration of the macro-action according to the environmental feedback. Through the IMALR rule, the system extracts multiple training samples at each time step and optimizes the policy.

[0036] Policy Optimization: During the training process, for each macro-action, the system increases the usage frequency of beneficial long-execution-time macro-actions through the macro-action reward mechanism. This reward mechanism ensures that the agent can use effective expert policies more frequently when performing tasks.

[0037] Reinforcement Learning: The agent uses the SMDP Q-learning method for learning, selects the optimal macro-action and its execution duration in the enhanced action space, and optimizes the policy according to the real-time feedback. Through the in-macro-action learning rule, the agent can extract multiple training samples during the execution of the same macro-action, improving the learning efficiency.

[0038] Experiments and Results

[0039] The algorithm of this embodiment has been verified in multiple multi-robot cooperative tracking tasks. The experimental results show that the efficient policy transfer method based on the enhanced action space can significantly improve the learning efficiency compared with the traditional algorithm based on the option-critic framework, and can better handle the credit assignment problem in the long decision chain in complex environments.

[0040] Experimental Setup

[0041] In the experiment, multiple robots were used to cooperate in the target tracking task. Each robot was trained using the efficient policy transfer method based on the enhanced action space, and the -greedy policy was adopted for exploration during the training process.

[0042] Result Analysis

[0043] By comparing with traditional methods (such as CAPS and DACOOP-A algorithms), the efficient policy transfer method based on the enhanced action space can achieve a high task completion rate in a short time and demonstrate good generalization ability in the test environment. Especially when dealing with long-execution macro actions, the method can effectively improve the structured exploration of the task environment and significantly reduce the complexity of credit assignment.

[0044] The present invention provides an effective multi-robot cooperative tracking method, which can optimize the policy transfer process based on the enhanced action space, improve the cooperative operation efficiency of the multi-robot system, solve some core technical problems in traditional methods, and has remarkable innovation and practicability.

[0045] The content in the above method embodiments is applicable to the present system embodiment. The functions specifically implemented by the present system embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0046] The content in the above method embodiments is applicable to the present device embodiment. The functions specifically implemented by the present device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0047] The above is a specific description of the preferred embodiment of the present invention, but the present invention is not limited to the described embodiment. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An efficient policy transfer multi-robot cooperative tracking method based on an enhanced action space, characterized in that It includes the following steps: Construct an enhanced action space: Model multiple expert policies as macro-actions with different execution durations respectively, and fuse the macro-actions with the original action space to form an enhanced action space that includes the original single-step actions and macro-actions with multiple durations; Hierarchical reinforcement learning decision-making: Evaluate the value of actions in the enhanced action space through a deep Q-network, and select the current optimal macro-action or original action; Macro-action execution and sample extraction: During the execution of the macro-action, continuously control the robot based on the expert policy, and extract multiple training samples at each time step through the in-macro-action learning rule (IMALR), and adjust the temporal difference target to optimize the action value function; Macro-action reward mechanism: According to the execution duration of the macro-action, add an intrinsic reward that is positively correlated with time to the macro-action to increase the usage frequency of effective long-duration macro-actions; Dynamic termination mechanism: During the execution of the macro-action, if it is detected that there is a higher-value action in the enhanced action space, terminate the current macro-action in advance and re-select the optimal action.

2. The method according to claim 1, wherein The construction of the enhanced action space includes: Model each expert policy as a set of macro-actions with multiple fixed execution durations, where the execution duration range of each macro-action is from 1 to the preset maximum duration; Map the set of macro-actions and the original single-step actions to the enhanced action space together to form a hybrid action selection mechanism.

3. The method according to claim 1, characterized in that The in-macro-action learning rule (IMALR) specifically includes: At each time step of the macro-action execution, generate multiple training samples corresponding to different execution start times; By adjusting the temporal difference target, use the training samples to update the action value function and improve the learning efficiency.

4. The method according to claim 1, wherein The reward function of the macro-action reward mechanism is expressed as: where c is a preset coefficient, and is the execution time step of the macro-action.

5. The method according to claim 1, characterized in that, The condition of the dynamic termination mechanism is: Among them, To enhance the maximum value of other actions in the action space, is the macro action currently being executed, is the preset threshold value.

6. The method according to claim 1, wherein The method further includes: Extract local observation information through an attention mechanism, calculate the importance weights of teammates, and fuse them into the artificial potential field method; Regularize the KL divergence of the attention weights to optimize the extraction of local observation features.

7. An efficient policy transfer multi-robot cooperative tracking system based on an enhanced action space, characterized in that, It includes: An action space enhancement module for modeling expert policies as multi-duration macro-actions and fusing them with the original actions; A hierarchical decision-making module for evaluating the action value in the enhanced action space based on a deep Q-network and selecting the optimal action; A sample extraction and learning module for executing the IMALR rule to generate multiple training samples during the macro-action; A reward allocation module for dynamically adding intrinsic rewards according to the macro-action execution duration; A dynamic interruption module for real-time monitoring of the action value and triggering the early termination of the macro-action.

8. The system according to claim 7, wherein The system further includes: An attention mechanism unit for calculating the relative importance weights among pursuers and optimizing the embedded representation of local observation information; An artificial potential field controller for generating repulsive and attractive forces by combining attention weights to achieve multi-robot cooperative obstacle avoidance and encirclement.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1-6.

10. A multi-robot collaborative tracking device, comprising a processor and a memory, characterized in that, The memory stores a computer program, and when the processor executes the program, it implements the method described in any one of claims 1-6.

Images