Multi-agent formation cooperative control method based on task decomposition reinforcement learning

Abstract

The invention relates to the technical field of information, discloses a multi-agent formation cooperative control method based on task decomposition reinforcement learning, and aims to solve the problems of low task weight distribution efficiency and weak algorithm generalization ability in existing multi-agent formation cooperative control. The scheme comprises the following steps: establishing a multi-agent motion model and an obstacle avoidance model; designing a state, an action space and a return function; constructing a strategy network and an evaluation function based on task decomposition; designing a strategy model updating method of deep reinforcement learning; and designing a multi-task dynamic weight updating method. For a complex environment, a task is decomposed into a plurality of parallel subtasks by means of task planning through a multi-agent formation cooperative control method, coupling between the tasks is reduced, interference between the tasks is reduced, optimal task weight distribution under different environments is achieved through dynamic weights, and the adaptability of agents to different environments is improved. The method is suitable for unmanned aerial vehicle formation control, robot cooperation and the like.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent collaborative control and reinforcement learning, and in particular to a multi-agent formation collaborative control method based on task decomposition reinforcement learning. Background Art

[0002] Multi-agent formation cooperative control technology is one of the important issues in the study of multi-agent systems and has been widely studied. In recent years, the rapid development of robotics, artificial intelligence and other fields has not only promoted the intelligence of various fields of industrial information, but also brought a feasible intelligent learning method to the study of multi-agent formation cooperative control. Reinforcement learning belongs to the category of unsupervised machine learning methods and has been successfully applied to multi-agent formation cooperative control. It has a unique learning mechanism, that is, by interacting with the environment, accumulating corresponding experience, and realizing the gradual evolution of strategies based on the mechanism of delayed rewards. Compared with traditional control methods, the strategies learned by reinforcement learning show significant advantages. It can ensure that the entire control process reaches the optimal state and has better adaptability when facing complex and changing environments. Due to these advantages of reinforcement learning, reinforcement learning has become one of the mainstream methods to solve the problem of environmental adaptability in multi-agent formation cooperative control tasks.

[0003] However, the current multi-agent formation collaborative control algorithm based on reinforcement learning still has problems such as low efficiency in task weight allocation and poor algorithm transferability. In particular, the highly coupled sub-task relationship directly affects the algorithm's adaptability in different environments, weakening the control strategy learning effect of reinforcement learning and the generalization effect of the algorithm. Therefore, how to improve the adaptive ability of agents in complex environments is an important issue that needs to be solved to promote the practical application of formation collaborative control algorithms based on reinforcement learning. To solve this problem, the present invention provides a multi-agent formation collaborative control method based on task decomposition reinforcement learning, targeting the problems of low efficiency in task weight allocation and poor transferability in the existing multi-agent formation collaborative control algorithm based on reinforcement learning. Summary of the invention

[0004] The purpose of this invention is to solve the problems of low efficiency in task weight allocation and weak algorithm generalization in multi-agent formation collaborative control in complex environments, so as to enhance the adaptability of agents to different environments and improve the effect of multi-agent formation collaborative control in complex environments.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A multi-agent formation collaborative control method based on task decomposition reinforcement learning includes the following steps:

[0007] Step S1, establishing a multi-agent motion model and an obstacle detection model, including constructing a multi-agent particle model, a leader-follower model and an obstacle detection model, and the constructed model provides an interactive environment for offline training of multi-agent formation collaborative control based on task decomposition reinforcement learning;

[0008] Step S2: construct the state space, action space and reward return function to meet the requirements and environmental constraints of the multi-agent task decomposition reinforcement learning formation collaborative control task, as follows:

[0009] State design: Based on the multi-agent motion model and obstacle detection model constructed in step S1, the local observation o of agent i is constructed by combining the relative position and speed information of the communicable neighboring agents and the position and speed information of the obstacles within the agent detection range. i , construct the global state s = (o1, o2, ..., o i ,…,o n );

[0010] Action design: Define the acceleration of the agent as the action a of the agent i , construct a joint action a=(a1, a2, ..., a i , …, a n );

[0011] Reward function design: Decompose the task based on task planning, and use the control error generated by the multi-agent formation collaborative control task as the reward value of the agent control. With the help of task planning, three subtasks of formation, obstacle avoidance and target point search are planned in the multi-agent formation.

[0012] The control error of the formation subtask includes the position and speed consistency control error between agents, the position and speed control error between the following agent and the leading agent, that is, the speed reward and distance reward between the leader and the follower.

[0013] The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range;

[0014] The target point search subtask includes a straight-line bonus item and a distance bonus item for the leader’s shortest search distance;

[0015] Step S3: Design a reward module based on task decomposition, and build a strategy network and an evaluation network in combination with a multi-layer perceptron to transform the observed state o in step S2 into iAs the input of the policy network to obtain the acceleration control amount of agent i, the global state s and joint action a in step S2 are used as the input of the evaluation network to complete the state-behavior evaluation;

[0016] Step S4, updating the parameters of the strategy network of each subtask in step S3 by a stochastic strategy optimization method of multi-agent reinforcement learning based on an information sharing mechanism;

[0017] Step S5: retain the reward updated each time in step S4, and design a dynamic weight update scheme based on the reward, weight each subtask, and complete the parameter update of the evaluation network in step S3;

[0018] Step S6: The control process of the multi-agent formation collaborative control method based on task decomposition reinforcement learning includes the following steps:

[0019] Step S6.1, constructing complete observation information using the observation state information obtained from the motion model and obstacle detection model of step S1;

[0020] Step S6.2, reconstructing the agent observation state using the state construction method of step S2 and the complete observation information constructed in step S6.1;

[0021] Step S6.3, sending the agent observation state reconstructed in step S6.2 to the strategy network model designed in step S3 to obtain the agent behavior;

[0022] Step S6.4, sending the intelligent agent behavior obtained in step S6.3 to the intelligent agent particle motion model in step S1 to complete the motion control of the intelligent agent.

[0023] In the above technical solution, step S1 comprises:

[0024] Step S1.1, establish a multi-agent particle motion model:

[0025] Consider each agent as a circular entity with radius r. Use the second-order differential system to describe the motion state of the center of the agent i:

[0026]

[0027] in, and Respectively represent the position, velocity and acceleration vector of agent i, and Respectively and The differential of , n represents the number of agents in the formation;

[0028] Step S1.2: Establish the following leader-follower model:

[0029] The first agent i1 is selected as the leader agent, and the rest are follower agents. The distance between the follower agent and the leader agent is:

[0030]

[0031] in, and Respectively represent the position information of the follower agent and the leader agent;

[0032] Step S1.3, establish the following obstacle detection model:

[0033] Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i is expressed as:

[0034]

[0035] Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r d The confidence radius of the detection sensor equipped by the agent, within which the detected obstacle motion information is credible.

[0036] In the above technical solution, step S2 comprises:

[0037] Design state space: Agent i establishes communication with neighboring agents to obtain the neighbor's position p and velocity information v. The internal observation state of agent i is constructed based on its own motion information and the neighbor's motion information obtained through communication, which can be expressed as:

[0038]

[0039] in, is the set of neighboring agents, represents the relative motion state of neighbor agent j relative to agent i, which is expressed as follows:

[0040]

[0041] in, and and Represents the position and speed information of agent j and agent i respectively, r c represents the normalized parameter of the relative position of neighbor agent j relative to agent i, ||v max || represents the modulus of the agent’s maximum speed;

[0042] The observation state of agent i to the target point is expressed as:

[0043]

[0044] Among them, p goal and r goal Respectively represent the normalized parameters of the position of the target point and the relative distance to the target point;

[0045] The obstacle observation information obtained by the agent through the obstacle detection model is set as

[0046] Based on the observation state information described above, the observation state of agent i is defined as: The system state of multi-agent consensus cooperative control is defined as the joint state s = (o1, o2, ..., o i ,…,o n );

[0047] Design action space: According to the motion model of the agent in step S1, the motion control of the agent can be achieved by providing the acceleration control amount, so the action a of the agent i is i Defined as the acceleration control quantity:

[0048] Design reward function: The formation coordination control error of the multi-agent system is used as the agent reward r(s, a). Through task planning, three subtasks of formation, obstacle avoidance and target point search are planned, which are expressed as follows:

[0049] r(s,a)=-(r f +r o +r g )

[0050] where r f Represents the reward item of the formation subtask, r o represents the reward item of the obstacle avoidance subtask, r g Represents the reward item for the target point search subtask;

[0051] The control error of the formation subtask includes the relative position and speed consistency control errors between agents, the control errors between the follower agent and the leader agent in position and speed, that is, the speed reward and distance reward between the leader and the follower:

[0052] The distance rewards are as follows:

[0053]

[0054] Where d is the relative distance between the follower agent and the leader agent, and the aggregation distance range is (d a , d c ); c1 and c2 represent formation reward factors;

[0055] The relative speeds are as follows:

[0056] Δv=|v leader -v other |

[0057] where v leader represents the speed of the leader agent, v other represents the speed of the following agent;

[0058] The speed bonus between agents is as follows:

[0059]

[0060] The total rewards for the formation subtask are as follows:

[0061] r f =r a +r v

[0062] where r a Indicates distance bonus item;

[0063] The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range;

[0064]

[0065] Among them, d represents the relative distance between the agent and other agents, d a represents the relative distance normalization parameter;

[0066] The target point search subtask is a straight-line reward item and a distance reward item for the leader's shortest search distance. The straight-line reward item is represented by the slope change of each agent's relative position before and after, as follows:

[0067]

[0068] in and Indicates the position of the agent before and after;

[0069] Distance bonus items are represented as follows:

[0070] r g =|k i -k|+Δd

[0071] Among them, k is the fixed slope between the initial position of the leader point and the target point, and Δd represents the relative distance between the leader agent and the target point before and after.

[0072] In the above technical solution, step S3 includes:

[0073] Design a reward module based on task decomposition, and combine a multi-layer perceptron to build a policy network and an evaluation network:

[0074] Local observation of agent i: And use it as the input of the policy network to obtain the acceleration control value of agent i;

[0075] Evaluation network model: The evaluation network includes an encoder and a multi-layer perceptron. The global state s and the joint action a of step S2 are encoded by the encoder and input into the evaluation network model. The evaluation network model encodes each observation state in s to obtain the encoded features, and concatenates the encoded features to obtain the feature h. s At the same time, the joint behavior a is mapped by the multi-layer perceptron to obtain the behavior feature h a ,h s With h a After splicing and passing through a multi-layer perceptron, the evaluation value Q is obtained;

[0076] Policy network model: The policy network includes an encoder and a multi-layer perceptron. The observed state of the agent is passed through the encoder to obtain aggregated features, and then the aggregated features are sent to the multi-layer perceptron to obtain the mean and variance of the agent's behavior ai.

[0077] In the above technical solution, step S4 includes:

[0078] Policy update method of deep reinforcement learning: In the multi-agent formation collaborative control task, the state space and behavior space attributes of the agents are exactly the same, and all agents are equivalent and share the same policy network and experience pool data. The policy network of the task is denoted as π, and two independent evaluation networks are used to evaluate each subtask;

[0079] Evaluation network parameter update: The evaluation network of the agent takes the global state s and the joint action a as input, and optimizes the parameters by truncating the double Q. The loss function of the evaluation network is defined as follows:

[0080]

[0081] in, For the experience replay pool, Indicates the sample pool for multiple samples. is the state-action value function, which depends on the parameter φ i, represents the estimated value of taking action a in state s;

[0082] φ i is the parameter of the i-th evaluation network, and the target fitting value y is defined as follows:

[0083]

[0084] Among them, r represents the reward obtained by the agent when taking action a in state s, represents the next action resampled from the policy, and the s agent takes action in state s After transferring to the next state, Y represents the discount factor, represents the minimum value selected from two mixed Q networks, is the Q-value function of the j-th hybrid Q network;

[0085] Policy network parameter update: The policy network of agent i will observe state o i as input and maximizes the expected future reward plus the expected future entropy V π (s) is the optimization target, which is expressed as follows:

[0086]

[0087] Among them, α represents the entropy regularization coefficient, The expected symbol indicates that the action a is averaged according to the strategy π, Q π (s, a) represents the state-action value function under strategy π, that is, the value after executing action a from state s, and π(a|s) represents the probability of executing action a in state s;

[0088] The policy optimization uses the reparameter trick to rewrite the expectation of the action as the expectation of the noise and take the minimum Q value between the two Q approximators to represent Q πθ , the strategy is updated as follows:

[0089]

[0090] In the formula, ξ represents the random factor, Indicates the expectation symbol, is the action sampled from the policy, It means to select the smallest one from two Q functions;

[0091] Finally, the target network is updated by the following loss:

[0092]

[0093] In the formula, τ t represents the soft update coefficient, represents the entropy term, logπ t (a t |τ t ) represents that at time step t, according to the strategy π t Select action a t The logarithmic probability of .

[0094] In the above technical solution, step S5 comprises:

[0095] When updating the strategy, the weights of different subtasks are normalized according to the size of the reward. The specific expression of the weight factor is as follows:

[0096]

[0097] Among them, r k represents the reward of the kth task;

[0098] The policy loss update of the overall task after weighting is expressed as follows:

[0099]

[0100] In the formula, represents the Q value of the kth task, σ k is the normalized weight of each Q-approximator, the entropy regularization coefficient α and the update of the target network remain unchanged.

[0101] Because the present invention adopts the above technical solution, it has the following beneficial effects:

[0102] First, because the present invention adopts a multi-agent formation collaborative control method based on deep reinforcement learning and dynamic weights, it solves the adaptability problem of the multi-agent formation collaborative control algorithm based on reinforcement learning in different complex environments. It uses the task decomposition model to cleverly decompose complex and comprehensive tasks into multiple parallel subtasks, and with the help of dynamic weights, the intelligent agent can flexibly adjust the weight distribution of each subtask according to the real-time changes of the environment, so that the intelligent agent can better adapt to the different environments it is in. Thereby achieving the effect of high stability, transferability and strong robustness of multi-agent formation collaborative control in complex environments.

[0103] Second, the task decomposition model reduces the dimensions of the state space and action space when applying reinforcement learning, avoiding the curse of dimensionality and enabling the agent to learn to optimize behavior strategies more efficiently;

[0104] 3. The present invention breaks the limitation of highly coupled subtasks in traditional methods. The relatively independent characteristics of each subtask make it possible for the algorithm based on the present invention to only adaptively adjust some subtask-related parameters or strategies when facing different application environments and scenario changes, without the need to make large-scale changes to the entire algorithm framework. This greatly enhances the ability of the algorithm to migrate from one environment to another and broadens its scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 is a flow chart of the method of the present invention;

[0106] Figure 2 A control flow diagram of a multi-agent formation collaborative control method based on task decomposition reinforcement learning;

[0107] Figure 3 The figure is a schematic diagram of the overall algorithm flow that combines the evaluation network and the strategy network structure. DETAILED DESCRIPTION

[0108] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0109] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0110] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0111] like Figure 1 As shown, a multi-agent formation collaborative control method based on task decomposition reinforcement learning includes the following steps:

[0112] Step S1, establishing a multi-agent motion model and an obstacle detection model, including constructing a multi-agent particle model, a leader-follower model and an obstacle detection model, and the constructed model provides an interactive environment for offline training of multi-agent formation collaborative control based on task decomposition reinforcement learning;

[0113] Step S2: construct the state space, action space and reward return function to meet the requirements and environmental constraints of the multi-agent task decomposition reinforcement learning formation collaborative control task, as follows:

[0114] State design: Based on the multi-agent motion model and obstacle detection model constructed in step S1, the local observation o of agent i is constructed by combining the relative position and speed information of the communicable neighboring agents and the position and speed information of the obstacles within the detection range. i , construct the global state s = (o1, o2, ..., o i ,…,o n );

[0115] Action design: Define the acceleration of the agent as the action a of the agent i , construct a joint action a=(a1, a2, ..., a i , …, a n );

[0116] Reward function design: The task is decomposed based on task planning, and the control error generated by the multi-agent formation collaborative control task is used as the reward value of the agent control. With the help of task planning in the multi-agent formation, three subtasks of formation, obstacle avoidance and target point search are planned.

[0117] The control error of the formation subtask includes the position and speed consistency control error between agents, the position and speed control error between the following agent and the leading agent, that is, the speed reward and distance reward between the leader and the follower.

[0118] The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range;

[0119] The target point search subtask includes a straight-line bonus item and a distance bonus item for the leader’s shortest search distance;

[0120] Step S3, design a reward module based on task decomposition, and build a strategy network and evaluation function in combination with a multi-layer perceptron; convert the observed state o in step S2 into i As the input of the policy network to obtain the acceleration control amount of the agent i; the global state s and the joint action a in step S2 are used as the input of the evaluation network to complete the state-behavior evaluation;

[0121] Step S4, updating the parameters of the strategy network of each subtask in step S3 by a random strategy optimization method of multi-agent reinforcement learning based on a sharing mechanism;

[0122] Step S5, retaining the reward updated each time in step S4, and designing a dynamic weight update scheme based on the reward, weighting each subtask, and completing the parameter update of the evaluation function in step S3;

[0123] Step S6: The control process of the multi-agent formation collaborative control method based on task decomposition reinforcement learning includes the following steps:

[0124] Step S6.1, constructing complete observation information using the observation state information obtained from the motion model and obstacle detection model of step S1;

[0125] Step S6.2, reconstructing the agent observation state using the state construction method of step S2 and the complete observation information constructed in step S6.1;

[0126] Step S6.3, sending the agent observation state reconstructed in step S6.2 to the strategy network model designed in step S3 to obtain the agent behavior;

[0127] Step S6.4, sending the intelligent agent behavior obtained in step S6.3 to the intelligent agent particle motion model in step S1 to complete the motion control of the intelligent agent.

[0128] In the above technical solution, step S1 comprises:

[0129] Step S1.1, establish the intelligent body particle motion model:

[0130] Consider each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i:

[0131]

[0132] in, and Respectively represent the position, velocity and acceleration vector of agent i, and Respectively and The differential of , n represents the number of agents in the formation;

[0133] Step S1.2: Establish the following leader-follower model:

[0134] The first agent i1 is selected as the leader agent, and the rest are follower agents. The distance between the follower agent and the leader agent is:

[0135]

[0136] in, and Respectively represent the position information of the follower agent and the leader agent.

[0137] Step S1.3, establish the following obstacle detection model:

[0138] Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i can be expressed as:

[0139]

[0140] Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r d The confidence radius of the detection sensor equipped by the agent, within which the detected obstacle motion information is credible.

[0141] In the above technical solution, step S2 comprises:

[0142] Design state space: Agent i establishes communication with neighboring agents to obtain the neighbor's position p and velocity information v. Based on its own motion information and the neighbor's motion information obtained through communication, the internal observation state of agent i can be constructed, which is expressed as:

[0143]

[0144] in, is the set of neighboring agents. represents the relative motion state of neighbor agent j relative to agent i, which is expressed as follows:

[0145]

[0146] in, and and Represents the position and speed information of agent j and agent i respectively, r c represents the normalized parameter of the relative position of neighbor agent j relative to agent i, ||v max || represents the modulus of the agent’s maximum speed;

[0147] The observation state of agent i to the target point can be expressed as:

[0148]

[0149] Among them, p goal and r goal They represent the normalized parameters of the position of the target point and the relative distance to the target point respectively.

[0150] In addition, the obstacle observation information scanned by the agent through the radar is set as

[0151] Based on the observation state information described above, the observation state of agent i can be defined as: The system state of multi-agent consensus cooperative control is defined as the joint state s = (o1, o2, ..., o i ,…,o n );

[0152] Design action space: According to the intelligent body motion model in step S1, only the acceleration control

[0153] The motion control of the intelligent agent can be realized by measuring the amount, so the action a of the intelligent agent i is i Defined as the acceleration control quantity:

[0154] Design reward function: Take the formation coordination control error of the multi-agent system as the agent reward r(s, a), and plan three subtasks of formation, obstacle avoidance, and target point search through task planning. It is expressed as follows:

[0155] r(s,a)=-(r f +r o +r g )

[0156] where r f Represents the reward item of the formation subtask, r o represents the reward item of the obstacle avoidance subtask, r g Represents the reward item for the target point search subtask;

[0157] The control error of the formation subtask includes the relative position and speed consistency control errors between agents, and the control errors between the follower agent and the leader agent in position and speed, namely the leader-follower speed reward and distance reward:

[0158] The distance rewards are as follows:

[0159]

[0160] Where d is the relative distance between the follower agent and the leader agent, and the aggregation distance range is (d a , d c ). c1 and c2 represent formation reward factors.

[0161] The relative speeds are as follows:

[0162] Δυ=||υ leader -υ other ||

[0163] The speed bonus between agents is as follows:

[0164]

[0165] The total rewards for the formation subtask are as follows:

[0166] r f =r a +r v

[0167] The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range;

[0168]

[0169] Among them, d represents the relative distance between the agent and other agents, d a Represents the relative distance normalization parameter.

[0170] The target point search subtask is a straight-line reward item and a distance reward item for the leader's shortest search distance. The straight-line reward item is expressed as the slope change of each agent's relative position before and after as follows:

[0171]

[0172] Distance bonus items are represented as follows:

[0173] r g =||k i -k||+Δd

[0174] Among them, k is the fixed slope between the initial position of the leader point and the target point, and Δd represents the relative distance between the leader agent and the target point before and after.

[0175] In the above technical solution, step S3 includes:

[0176] Design a reward module based on task decomposition, and combine it with a multi-layer perceptron to build a policy network and evaluation function:

[0177] Local observation of agent i And use it as the input of the policy network to obtain the acceleration control value of agent i;

[0178] Evaluation network model: The evaluation network is mainly composed of an encoder and a multi-layer perceptron. The global state s and the joint action a of step S2 are encoded by the encoder and input into the evaluation network model. The evaluation network model encodes each observation state in s to obtain the encoded features, and concatenates the encoded features to obtain the feature h. s At the same time, the joint behavior a is mapped by the multi-layer perceptron to obtain the behavior feature h a ,h s With h a After splicing and passing through a multi-layer perceptron, the evaluation value Q is obtained;

[0179] Policy network model: The policy network consists of an encoder and a multi-layer perceptron. The observed state of the agent is passed through the encoder to obtain aggregated features, and then the aggregated features are sent to the multi-layer perceptron to obtain the agent's behavior a i The mean and variance of .

[0180] In the above technical solution, step S4 includes:

[0181] Policy update method of deep reinforcement learning: In the multi-agent formation collaborative control task, the state space and behavior space attributes of the agents are exactly the same, and all agents are equivalent and share the same policy network and experience pool data. The policy network of the task is denoted as π, and two independent evaluation networks are used to evaluate each subtask;

[0182] Evaluation network parameter update: The evaluation network of the agent takes the system state s and the joint action a as input, and optimizes the parameters by truncating the double Q. The loss function of the evaluation network is defined as follows:

[0183]

[0184] in, is the experience replay pool, E D Indicates finding the mean of multiple samples, φ i is the parameter of the i-th evaluation network, and the target fitting value y is defined as follows:

[0185]

[0186] Among them, r represents the reward obtained by the agent when taking action a in state s, represents the next action resampled from the policy, the next state that the s agent moves to after taking action a in state s, and γ represents the discount factor.

[0187] Policy network parameter update: The policy network of agent i will observe state o i as input and maximizes the expected future reward plus the expected future entropy V π (s) is the optimization target, which is expressed as follows:

[0188]

[0189] Where α represents the entropy regularization coefficient.

[0190] The policy optimization uses the reparameter trick to rewrite the expectation of the action as the expectation of the noise and take the minimum Q value between the two Q approximators to represent Q πθ , the strategy is updated as follows

[0191]

[0192] In the formula, ξ represents the random factor.

[0193] Finally, the target network is updated by the following loss:

[0194]

[0195] In the formula, τ t represents the soft update coefficient, represents the entropy term.

[0196] In the above technical solution, step S5 comprises:

[0197] When updating the strategy, we use the weights of different subtasks normalized according to the size of the reward. The specific expression of the weight factor is as follows:

[0198]

[0199] The policy loss update of the overall task after weighting is expressed as follows:

[0200]

[0201] In the formula, σ k is the normalized weight for each Q approximator. The entropy regularization coefficient α and the update of the target network remain unchanged.

[0202] In summary, the method provided by the present invention is based on task decomposition reinforcement learning technology, aiming to solve the problems of low efficiency of task weight allocation and weak algorithm generalization ability in the prior art. The following are the main technical features and effects of the present invention:

[0203] Construction of task decomposition framework: By designing a task decomposition framework, complex tasks are decomposed into multiple parallel subtasks. This method improves the efficiency of task processing and enhances the algorithm's adaptability to complex environments.

[0204] Implementation of dynamic weight strategy: The present invention adopts a dynamic weight strategy to achieve optimal task weight allocation. This strategy can automatically adjust the weight according to the real-time environment and task requirements, thereby improving the flexibility and efficiency of task allocation.

[0205] Design of multi-agent motion model and obstacle avoidance model: The present invention establishes accurate multi-agent motion model and obstacle avoidance model, which can effectively handle the coordination and obstacle avoidance problems among multiple agents and ensure the stability and safety of the formation.

[0206] Design and update of deep reinforcement learning strategy model: Through deep reinforcement learning technology, the present invention designs a strategy model and implements its update method. This method enables the intelligent agent to learn from the environment and continuously optimize its behavior strategy.

[0207] Design of multi-task dynamic weight updating method: This invention introduces a new multi-task dynamic weight updating method, which can adjust the weights in real time according to the importance of the task and environmental changes, thereby improving the flexibility and efficiency of task processing.

[0208] In summary, the present invention effectively solves the problems in the prior art through task decomposition reinforcement learning technology, and improves the efficiency and generalization ability of multi-agent formation collaborative control. Compared with the prior art, the present invention has higher flexibility and adaptability, and can effectively cope with complex and changing task environments.

Claims

1. A multi-agent formation collaborative control method based on task decomposition reinforcement learning, characterized in that: The following steps are involved: Step S1, establishing a multi-agent motion model and an obstacle detection model, including constructing a multi-agent particle model, a leader-follower model and an obstacle detection model, and the constructed model provides an interactive environment for offline training of multi-agent formation collaborative control based on task decomposition reinforcement learning; Step S2: construct the state space, action space and reward return function to meet the requirements and environmental constraints of the multi-agent task decomposition reinforcement learning formation collaborative control task, as follows: State design: Based on the multi-agent motion model and obstacle detection model constructed in step S1, the local observation o of agent i is constructed by combining the relative position and speed information of the communicable neighboring agents and the position and speed information of the obstacles within the agent detection range. i , construct the global state s = (o1, o2, ..., o i ,…,o n ); Action design: Define the acceleration of the agent as the action a of the agent i , construct a joint action a=(a1, a2, ..., a i ,…,a n ); Reward function design: Decompose the task based on task planning, and use the control error generated by the multi-agent formation collaborative control task as the reward value of the agent control. The multi-agent formation includes task planning, which plans three subtasks: formation, obstacle avoidance, and target point search. The control error of the formation subtask includes the position and speed consistency control error between agents, the position and speed control error between the following agent and the leading agent, that is, the speed reward and distance reward between the leader and the follower. The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range; The target point search subtask includes a straight-line bonus item and a distance bonus item for the leader’s shortest search distance; Step S3: Design a reward module based on task decomposition, and build a strategy network and an evaluation network in combination with a multi-layer perceptron to transform the observed state o in step S2 into i As the input of the policy network to obtain the acceleration control amount of agent i, the global state s and joint action a in step S2 are used as the input of the evaluation network to complete the state-behavior evaluation; Step S4, updating the parameters of the strategy network of each subtask in step S3 by a stochastic strategy optimization method of multi-agent reinforcement learning based on an information sharing mechanism; Step S5: retain the reward updated each time in step S4, and design a dynamic weight update scheme based on the reward, weight each subtask, and complete the parameter update of the evaluation network in step S3; Step S6: The control process of the multi-agent formation collaborative control method based on task decomposition reinforcement learning includes the following steps: Step S6.1, constructing complete observation information using the observation state information obtained from the motion model and obstacle detection model of step S1; Step S6.2, reconstructing the agent observation state using the state construction method of step S2 and the complete observation information constructed in step S6.1; Step S6.3, sending the agent observation state reconstructed in step S6.2 to the strategy network model designed in step S3 to obtain the agent behavior; Step S6.4, sending the intelligent agent behavior obtained in step S6.3 to the intelligent agent particle motion model in step S1 to complete the motion control of the intelligent agent.

2. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 1 is characterized by: The step S1 comprises: Step S1.1, establish a multi-agent particle motion model: Consider each agent as a circular entity with radius r. Use the second-order differential system to describe the motion state of the center of the agent i: in, and Respectively represent the position, velocity and acceleration vector of agent i, and Respectively and The differential of , n represents the number of agents in the formation; Step S1.2: Establish the following leader-follower model: Select the first agent i l is the leader agent, and the rest are follower agents. The distance between the follower agent and the leader agent is: in, and Respectively represent the position information of the follower agent and the leader agent; Step S1.3, establish the following obstacle detection model: Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i is expressed as: Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r d The confidence radius of the detection sensor equipped by the agent, within which the detected obstacle motion information is credible.

3. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 2 is characterized by: The step S2 comprises: Design state space: Agent i establishes communication with neighboring agents to obtain the neighbor's position p and velocity information v. The internal observation state of agent i is constructed based on its own motion information and the neighbor's motion information obtained through communication, which can be expressed as: in, is the set of neighboring agents, represents the relative motion state of neighbor agent j relative to agent i, which is expressed as follows: in, and and Represents the position and speed information of agent j and agent i respectively, r c represents the normalized parameter of the relative position of neighbor agent j relative to agent i, ||v max || represents the modulus of the agent's maximum speed; The observation state of agent i to the target point is expressed as: Among them, p goal and r goal Respectively represent the normalized parameters of the position of the target point and the relative distance to the target point; The obstacle observation information obtained by the agent through the obstacle detection model is set as Based on the observation state information described above, the observation state of agent i is defined as: The system state of multi-agent consensus cooperative control is defined as the joint state s = (o1, o2, ..., oi, ..., o n ); Design action space: According to the motion model of the agent in step S1, the motion control of the agent can be achieved by providing the acceleration control amount, so the action a of the agent i is i Defined as the acceleration control quantity: Design reward function: The formation coordination control error of the multi-agent system is used as the agent reward r(s, a). Through task planning, three subtasks of formation, obstacle avoidance and target point search are planned, which are expressed as follows: r(s,a)=-(r f +r o +r g ) where r f Represents the reward item of the formation subtask, r o represents the reward item of the obstacle avoidance subtask, r g Represents the reward item for the target point search subtask; The control error of the formation subtask includes the relative position and speed consistency control errors between agents, the control errors between the follower agent and the leader agent in position and speed, that is, the speed reward and distance reward between the leader and the follower: The distance rewards are as follows: Where d is the relative distance between the follower agent and the leader agent, and the aggregation distance range is (d a , d c ); c1 and c2 represent formation reward factors; The relative speeds are as follows: Δv=|v leader -v other | where v leader represents the speed of the leader agent, v other represents the speed of the following agent; The speed bonus between agents is as follows: The total rewards for the formation subtask are as follows: r f ＝r a +r v where r a Indicates distance bonus item; The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range; Among them, d represents the relative distance between the agent and other agents, d a represents the relative distance normalization parameter; The target point search subtask is a straight-line reward item and a distance reward item for the leader's shortest search distance. The straight-line reward item is represented by the slope change of each agent's relative position before and after, as follows: in and Indicates the position of the agent before and after; Distance bonus items are represented as follows: r g =|k i -k|+Δd Among them, k is the fixed slope between the initial position of the leader point and the target point, and Δd represents the relative distance between the leader agent and the target point before and after.

4. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 3 is characterized by: The step S3 comprises: Design a reward module based on task decomposition, and combine a multi-layer perceptron to build a policy network and an evaluation network: Local observation of agent i: And use it as the input of the policy network to obtain the acceleration control value of agent i; Evaluation network model: The evaluation network includes an encoder and a multi-layer perceptron. The global state s and the joint action a of step S2 are encoded by the encoder and input into the evaluation network model. The evaluation network model encodes each observation state in s to obtain the encoded features, and concatenates the encoded features to obtain the feature h. s At the same time, the joint behavior a is mapped by the multi-layer perceptron to obtain the behavior feature h a ,h s With h a After splicing and passing through a multi-layer perceptron, the evaluation value Q is obtained; Policy network model: The policy network includes an encoder and a multi-layer perceptron. The observed state of the agent is passed through the encoder to obtain aggregate features, and then the aggregate features are sent to the multi-layer perceptron to obtain the agent's behavior a i The mean and variance of .

5. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 4 is characterized by: The step S4 comprises: Policy update method of deep reinforcement learning: In the multi-agent formation collaborative control task, the state space and behavior space attributes of the agents are exactly the same, and all agents are equivalent and share the same policy network and experience pool data. The policy network of the task is denoted as π, and two independent evaluation networks are used to evaluate each subtask; Evaluation network parameter update: The evaluation network of the agent takes the global state s and the joint action a as input, and optimizes the parameters by truncating the double Q. The loss function of the evaluation network is defined as follows: in, For the experience replay pool, Indicates the sample pool for multiple samples. is the state-action value function, which depends on the parameter φ i , represents the estimated value of taking action a in state s; φ i is the parameter of the i-th evaluation network, and the target fitting value y is defined as follows: Among them, r represents the reward obtained by the agent when taking action a in state s, represents the next action resampled from the policy, and the s agent takes action in state s After transferring to the next state, γ represents the discount factor, represents the minimum value selected from two mixed Q networks, is the Q-value function of the j-th hybrid Q network; Policy network parameter update: The policy network of agent i will observe state o i as input and maximizes the expected future reward plus the expected future entropy V π (s) is the optimization target, which is expressed as follows: Among them, α represents the entropy regularization coefficient, The expected symbol indicates that the action a is averaged according to the strategy π, Q π (s, a) represents the state-action value function under strategy π, that is, the value after executing action a from state s, and π(a|s) represents the probability of executing action a in state s; The policy optimization uses the reparameter trick to rewrite the expectation of the action as the expectation of the noise and take the minimum Q value between the two Q approximators to represent Q πθ , the strategy is updated as follows: In the formula, ξ represents the random factor, Indicates the expectation symbol, is the action sampled from the policy, It means to select the smallest one from two Q functions; Finally, the target network is updated by the following loss: In the formula, τ t represents the soft update coefficient, represents the entropy term, logπ t (a t |τ t ) represents that at time step t, according to the strategy π t Select action a t The logarithmic probability of .

6. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 5 is characterized by: The step S5 comprises: When updating the strategy, the weights of different subtasks are normalized according to the size of the reward. The specific expression of the weight factor is as follows: Among them, r k represents the reward of the kth task; The policy loss update of the overall task after weighting is expressed as follows: In the formula, represents the Q value of the kth task, σ k is the normalized weight of each Q-approximator, the entropy regularization coefficient α and the update of the target network remain unchanged.

Images