A multi-agent formation cooperative control method based on task decomposition reinforcement learning

Abstract

The present application relates to the field of information technology, and discloses a kind of multi-agent formation cooperative control method based on task decomposition reinforcement learning, which aims to solve the problem of low efficiency of task weight distribution and weak algorithm generalization ability in existing multi-agent formation cooperative control.The scheme includes establishing multi-agent motion model and obstacle avoidance model;Design state, action space and reward function;Construct strategy network and evaluation function based on task decomposition;Design strategy model updating method of deep reinforcement learning;Design multi-task dynamic weight updating method.For complex environment, through multi-agent formation cooperative control method, task is decomposed into multiple parallel subtasks by means of task planning, reducing the coupling between tasks, thereby reducing the interference between tasks, and using dynamic weight to achieve optimal task weight distribution in different environments, improve the adaptability of agent to different environments.It is suitable for, such as unmanned aerial vehicle formation control, robot collaboration, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-agent cooperative control and reinforcement learning, and particularly relates to a multi-agent formation cooperative control method based on task decomposition reinforcement learning. BACKGROUND

[0002] Multi-agent formation cooperative control technology is one of the important problems in multi-agent system research and has been widely studied. In recent years, the rapid development of robot technology, artificial intelligence and other fields not only promotes the intelligentization of various fields of industrial information, but also brings an intelligent learning method to the research of multi-agent formation cooperative control. Reinforcement learning belongs to the category of unsupervised machine learning methods and has been successfully applied to multi-agent formation cooperative control. It has a unique learning mechanism, that is, through interaction with the environment, corresponding experience is accumulated, and the mechanism of delayed reward is used to realize the gradual evolution of the strategy. Compared with traditional control methods, the strategy learned by reinforcement learning shows significant advantages. It can ensure that the whole control process reaches the optimal state, and has better adaptability when facing complex and variable environment. Due to these advantages of reinforcement learning, reinforcement learning has become one of the mainstream methods to solve the environmental adaptability problem in multi-agent formation cooperative control tasks.

[0003] However, the current multi-agent formation cooperative control algorithm based on reinforcement learning still has problems such as low task weight distribution efficiency and poor algorithm transferability, especially the highly coupled subtask relationship directly affects the adaptability of the algorithm in different environments, and weakens the control strategy learning effect of reinforcement learning and the generalization effect of the algorithm. Therefore, how to improve the self-adaptability of agents in complex environments is an important problem to be solved in promoting the practical application of formation cooperative control algorithm based on reinforcement learning. In order to solve this problem, the present application provides a multi-agent formation cooperative control method based on task decomposition reinforcement learning to solve the problems of low task weight distribution efficiency and poor transferability of the existing multi-agent formation cooperative control algorithm based on reinforcement learning. SUMMARY

[0004] The purpose of the present application is to solve the problems of low task weight distribution efficiency and weak algorithm generalization ability in multi-agent formation cooperative control in complex environments. To improve the adaptability of agents to different environments and improve the effect of multi-agent formation cooperative control in complex environments.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A multi-agent formation cooperative control method based on task decomposition reinforcement learning, comprising the following steps:

[0007] Step S1, a multi-agent motion model and an obstacle detection model are established, including constructing a multi-agent particle model, a leader-follower model and an obstacle detection model, and the constructed model provides an interactive environment for offline training of the multi-agent formation cooperative control based on task decomposition reinforcement learning;

[0008] Step S2, a state space, an action space and a reward return function are constructed to meet the needs and environmental constraints of the multi-agent task decomposition reinforcement learning formation cooperative control task, and the method is as follows:

[0009] State design: based on the multi-agent motion model and the obstacle detection model constructed in step S1, the relative position and speed information of the communicable neighbor agent, and the position and speed information of the obstacle within the detection range of the agent are combined to construct the local observation o i of the agent i i ; n The global state s=(o1, o2, …, o

[0010] Action design: the acceleration of the agent is defined as the action a i of the agent, and the joint action a=(a1, a2, …, a i , …, a n ) is constructed based on the action of each agent;

[0011] Reward function design: based on task planning, the task is decomposed, and the control error generated by the multi-agent formation cooperative control task is taken as the reward value of the agent control, and in the multi-agent formation, three subtasks of formation, obstacle avoidance and search target point are planned out with the help of task planning;

[0012] The control error of the formation subtask includes the position and speed consistency control error between the agents, and the control error between the follower and the leader in position and speed, i.e. the speed reward item and the distance reward item between the leader and the follower;

[0013] The control error of the obstacle avoidance subtask includes the relative distance reward item between the agent and other objects within the detection range;

[0014] The search target point subtask includes the straight line reward item and the distance reward item of the shortest search distance of the leader;

[0015] Step S3, a reward module based on task decomposition is designed, and a strategy network and an evaluation network are constructed in combination with a multi-layer perception mechanism, and the observation state o iThe global state s and joint action a in step S2 are taken as inputs of the evaluation network to complete state-action evaluation as an acceleration control amount of the agent i;

[0016] Step S4, parameter updating of the policy network of each sub-task in step S3 is performed by a random policy optimization method based on multi-agent reinforcement learning of an information sharing mechanism;

[0017] Step S5, the reward of each update in step S4 is retained, a dynamic weight updating scheme based on the reward is designed, each sub-task is weighted, and parameter updating of the evaluation network in step S3 is completed;

[0018] Step S6, the control flow of the multi-agent formation cooperative control method based on task decomposition reinforcement learning includes the following steps:

[0019] Step S6.1, complete observation information is constructed by using observation state information obtained from the motion model and obstacle detection model in step S1;

[0020] Step S6.2, the agent observation state is reconstructed by using the state construction method in step S2 and the complete observation information constructed in step S6.1;

[0021] Step S6.3, the agent behavior is obtained by sending the agent observation state reconstructed in step S6.2 into the policy network model designed in step S3;

[0022] Step S6.4, the motion control of the agent is completed by sending the agent behavior obtained in step S6.3 into the agent particle motion model in step S1.

[0023] In the above technical solution, step S1 includes:

[0024] Step S1.1, a multi-agent particle motion model is established:

[0025] Each agent is regarded as a circular entity with a radius r a The motion state of the center of the agent i is described by using a second-order differential system:

[0026]

[0027] wherein, and denote the position, velocity and acceleration vectors of the agent i, and denote the differentials of and , and n denotes the number of agents in the formation;

[0028] Step S1.2, the following leader-follower model is established:

[0029] The first intelligent agent i1 is selected as the leader agent, and the rest are follower agents. The distance between the follower agent and the leader agent is:

[0030]

[0031] Wherein, and represent the position information of the follower agent and the leader agent, respectively;

[0032] Step S1.3, the following obstacle detection model is established:

[0033] Based on the detection distance of the intelligent agent, the set of obstacles observed by the intelligent agent i in the environment is represented as:

[0034]

[0035] Wherein, V o is a set of all obstacles in the environment, is the position of obstacle k, r d is the confidence radius of the detection sensor equipped by the intelligent agent. Within this range, the detected obstacle motion information is reliable.

[0036] The above technical solution, the step S2 includes:

[0037] Design state space: the intelligent agent i communicates with the neighbor intelligent agent to obtain the position p and the speed information v of the neighbor, and constructs the internal observation state of the intelligent agent i according to the motion information of itself and the motion information of the neighbor obtained by communication, which is represented as:

[0038]

[0039] Wherein, is a set of neighbor intelligent agents, represents the relative motion state of the neighbor intelligent agent j relative to the intelligent agent i, which is represented as follows:

[0040]

[0041] Wherein, and and represent the position and speed information of the intelligent agent j and the intelligent agent i, respectively, r c represents the relative position normalization parameter of the neighbor intelligent agent j relative to the intelligent agent i, ||v max || represents the modulus value of the maximum speed of the intelligent agent;

[0042] The observation state of the target point of the intelligent agent i is represented as:

[0043]

[0044] where p goal and r goal represent the position of the target point and the normalized parameter of the relative distance to the target point, respectively;

[0045] The obstacle observation information obtained by the intelligent agent through the obstacle detection model is set as

[0046] Based on the above described observation state information, the observation state of the intelligent agent i is defined as: The system state of the multi-agent consensus collaborative control is defined as the joint state s=(o1, o2, …, o i , …, o n );

[0047] Design the action space: according to the intelligent agent motion model in the step S1, the motion control of the intelligent agent can be realized by providing the acceleration control quantity, therefore the action a i of the intelligent agent i is defined as the acceleration control quantity:

[0048] Design the reward function: the formation collaborative control error of the multi-agent system is taken as the intelligent agent reward r(s, a), and through task planning, the three sub-tasks of formation, obstacle avoidance and target point search are planned, and are represented as follows:

[0049] r(s, a) = -(r f +r o +r g )

[0050] where r f represents the reward item of the formation sub-task, r o represents the reward item of the obstacle avoidance sub-task, and r g represents the reward item of the target point search sub-task;

[0051] where the control error of the formation sub-task includes the relative position and speed consistency control error between the intelligent agents, and the control error of the position and speed between the follower intelligent agent and the leader intelligent agent, i.e. the speed reward item and the distance reward item between the leader and the follower:

[0052] where the distance reward item is as follows:

[0053]

[0054] where d is the relative distance between the follower and the leader, the gathering distance range is (d a , d c ); c1 and c2 represent the formation reward factor;

[0055] The relative velocity is as follows:

[0056] Δv = |v leader -v other |

[0057] where v leader represents the velocity of the leader, and v other represents the velocity of the follower;

[0058] The velocity reward term between the agents is as follows:

[0059]

[0060] The total reward of the formation subtask is as follows:

[0061] r f = r a + r v

[0062] where r a represents the distance reward term;

[0063] The control error of the obstacle avoidance subtask includes the relative distance reward term between the agent and other objects within the detection range;

[0064]

[0065] where d represents the relative distance between the agent and other agents, and d a represents the relative distance normalization parameter;

[0066] The search target point subtask is the straight line reward term of the shortest search distance of the leader and the distance reward term, and the straight line reward term is represented by the change of the slope of the relative position of each agent before and after, and is specifically as follows:

[0067]

[0068] where and represent the positions of the agent at different times;

[0069] The distance reward term is represented as follows:

[0070] r g = |k i -k| + Δd

[0071] Wherein, k is the fixed slope of the leader point initial position and the target point, and Delta d represents the relative distance between the leader agent and the target point at the front and rear time.

[0072] In the technical solution, the step S3 comprises:

[0073] A reward module based on task decomposition is designed, and a strategy network and an evaluation network are constructed based on a multilayer perception mechanism.

[0074] The local observation of the agent i: And the acceleration control amount of the agent i is obtained by taking the local observation as the input of the strategy network.

[0075] The evaluation network model: the evaluation network comprises an encoder and a multilayer perception mechanism, and the global state s and the joint action a of the step S2 are input into the evaluation network model after being encoded by the encoder. The evaluation network model encodes each observation state in s to obtain a coded feature, and the features obtained by encoding are spliced to obtain a feature h s , and the joint action a is mapped by the multilayer perception mechanism to obtain a behavior feature h a , h s is spliced and input into the multilayer perception mechanism to obtain an evaluation value Q. a

[0076] The strategy network model: the strategy network comprises an encoder and a multilayer perception mechanism. The observation state of the agent is input into the encoder to obtain an aggregated feature, and then the aggregated feature is input into the multilayer perception mechanism to obtain the mean and variance of the behavior a i of the agent.

[0077] In the technical solution, the step S4 comprises:

[0078] The strategy updating method of deep reinforcement learning: in the multi-agent formation cooperative control task, the state space and the behavior space attribute of the agent are completely same, and all the agents are equivalent and share the same strategy network and experience pool data. The strategy network of the task is denoted as pi. Two independent evaluation networks are used to evaluate each subtask.

[0079] The evaluation network parameter updating: the evaluation network of the agent takes the global state s and the joint action a as the input, and optimizes the parameters by truncating double Q. The loss function of the evaluation network is defined as follows:

[0080]

[0081] Wherein, is an experience replay pool, represents a sample pool of multiple samples, is a state-action value function, which depends on the parameter ​an estimate of the value of taking action a in state s;

[0082] For the parameters of the i-th evaluation network, the target fit value y is defined as follows:

[0083]

[0084] where r represents the reward obtained by the agent in state s by taking action a, represents the next action resampled from the policy, s' is the next state the agent transitions to by taking action in state s, Y represents the discount factor, represents the minimum value selected from the two mixed Q networks, is the Q value function of the j-th mixed Q network;

[0085] Policy network parameter update: the policy network of the agent i takes the observation state o i as input, and takes the maximum expected future return plus the expected future entropy V π (s) as the optimization target, which is expressed as follows:

[0086]

[0087] where α represents the entropy regularization coefficient, the expected symbol, represents the average of action a according to the policy π, Q π (s, a) represents the state-action value function under the policy π, i.e., the value after performing action a from state s, π(a|s) represents the probability of performing action a in state s;

[0088] The policy optimization uses the reparameterization trick to rewrite the expectation of action as the expectation of noise, and takes the minimum Q value between the two Q approximators to represent Q πθ , and the policy update is as follows:

[0089]

[0090] where ξ represents a random factor, represents the expected symbol, is the action sampled from the policy, represents the minimum selected from the two Q functions;

[0091] Finally, the target network is updated by the following loss:

[0092]

[0093] where represents the soft update coefficient, represents the entropy term, representing at time step t, according to policy t selecting action a t log probability.

[0094] In the technical solution, the step S5 comprises:

[0095] At the time of policy updating, the weights of different sub-tasks are normalized according to the size of the reward, and the specific representation of the weight factor is as follows:

[0096]

[0097] wherein, r k represents the reward of the kth task;

[0098] The policy loss of the overall task after weighting is updated as follows:

[0099]

[0100] In the formula, represents the Q value of the kth task, and sigma k is the normalization weight of each Q approximator, the entropy regularization coefficient alpha and the update of the target network are unchanged.

[0101] Because the application adopts the technical solution, the following beneficial effects are achieved:

[0102] I. Because the application adopts the deep reinforcement learning multi-agent formation cooperative control method based on task decomposition and dynamic weight, the adaptability problem of the multi-agent formation cooperative control algorithm based on reinforcement learning in different complex environments is solved, the complex and comprehensive task is ingeniously decomposed into multiple parallel sub-tasks by using the task decomposition model, and the agent can flexibly adjust the weight distribution of each sub-task according to the real-time change of the environment with the help of dynamic weight, so that the agent can better adapt to different environments. Thus, the effects of high stability, migration and strong robustness of multi-agent formation cooperative control in complex environments are achieved.

[0103] II. The task decomposition model reduces the state space and action space dimensions when the reinforcement learning is applied, avoids the dimension disaster, and enables the agent to more efficiently learn and optimize the behavior strategy.

[0104] III. The application breaks the limitation of high coupling of sub-tasks in the traditional method, and the relatively independent characteristics of each sub-task enable the algorithm based on the application to only need to adaptively adjust part of the sub-task related parameters or strategy when facing different application environments and scene changes, without the need to greatly change the entire algorithm framework, greatly enhancing the ability of the algorithm to migrate from one environment to another, and widening its application range. BRIEF DESCRIPTION OF DRAWINGS

[0105] Figure 1 is a flow chart of the method of the present invention;

[0106] Figure 2 This is a control flow diagram of the multi-agent formation collaborative control method based on task decomposition reinforcement learning;

[0107] Figure 3 A schematic diagram of the overall algorithm flow that combines the evaluation network and the policy network structure. DETAILED DESCRIPTION

[0108] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.

[0109] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0110] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0111] like Figure 1 As shown in FIG, a multi-agent formation cooperative control method based on task decomposition reinforcement learning includes the following steps:

[0112] Step S1: Establish a multi-agent motion model and obstacle detection model, including constructing a multi-agent particle model, a leader-follower model, and an obstacle detection model. The constructed model provides an interactive environment for offline training of multi-agent formation collaborative control based on task decomposition reinforcement learning;

[0113] Step S2: Construct the state space, action space, and reward function to meet the requirements and environmental constraints of the multi-agent task decomposition reinforcement learning formation collaborative control task. The method is as follows:

[0114] State design: Based on the multi-agent motion model and obstacle detection model constructed in step S1, the local observation o of agent i is constructed by combining the relative position and speed information of the communicable neighboring agents and the position and speed information of the obstacles within the detection range. i , construct the global state s = (o1, o2, ..., o i ,…,o n );

[0115] Action design: Define the acceleration of the agent as the action of the agent a i , construct the joint action a = (a1, a2, …, a i , …, a n ) based on the action of each agent;

[0116] Reward function design: Based on the task planning, the control error generated by the multi-agent formation cooperative control task is taken as the reward value of the agent control. In the multi-agent formation, with the help of task planning, three sub-tasks of formation, obstacle avoidance and search target point are planned.

[0117] The control error of the formation sub-task includes the error of the position and speed consistency control between the agents, and the control error of the position and speed between the follower and the leader, that is, the speed reward term and the distance reward term between the leader and the follower;

[0118] The control error of the obstacle avoidance sub-task includes the relative distance reward term between the agent and other objects within the detection range;

[0119] The search target point sub-task includes the straight line reward term and the distance reward term of the shortest search distance of the leader;

[0120] Step S3, design a reward module based on task decomposition, and combine a multi-layer perception mechanism to build a strategy network and an evaluation function; take the observation state o i in step S2 as the input of the strategy network to obtain the acceleration control amount of the agent i; take the global state s and the joint action a in step S2 as the input of the evaluation network to complete the state-action evaluation;

[0121] Step S4, update the parameters of the strategy network of each sub-task in step S3 by a random strategy optimization method based on multi-agent reinforcement learning with a sharing mechanism;

[0122] Step S5, retain the reward updated each time in step S4, and design a dynamic weight update scheme based on the reward, weight each sub-task, and complete the parameter update of the evaluation function in step S3;

[0123] Step S6, the control flow of the multi-agent formation cooperative control method based on task decomposition reinforcement learning includes the following steps:

[0124] Step S6.1, construct complete observation information using the observation state information obtained from the motion model and obstacle detection model in step S1;

[0125] Step S6.2, reconstructing the agent observation state by using the state construction method of step S2 and the complete observation information constructed in step S6.1;

[0126] Step S6.3, sending the agent observation state reconstructed in step S6.2 into the policy network model designed in step S3 to obtain the agent behavior;

[0127] Step S6.4, sending the agent behavior obtained in step S6.3 into the agent particle motion model in step S1 to complete the motion control of the agent.

[0128] In the above technical solution, step S1 includes:

[0129] Step S1.1, establishing an agent particle motion model:

[0130] Each agent is regarded as a circular entity with a radius r a , and a second-order differential system is used to describe the motion state of the center of the agent i:

[0131]

[0132] wherein, and represent the position, velocity and acceleration vectors of the agent i respectively, and represent the differentials of and , and n represents the number of agents in the formation;

[0133] Step S1.2, establishing the following leader-follower model:

[0134] The first agent i1 is selected as the leader agent, and the rest are follower agents. The distance between the follower agent and the leader agent is:

[0135]

[0136] wherein, and represent the position information of the follower agent and the leader agent respectively.

[0137] Step S1.3, establishing the following obstacle detection model:

[0138] Based on the detection distance of the agent, the set of obstacles observed by the agent i in the environment can be represented as:

[0139]

[0140] wherein, V oA set of all obstacles in the environment, A position of the obstacle k, r d A confidence radius of the detection sensor equipped by the intelligent agent, within which range the detected obstacle motion information is reliable.

[0141] In the technical solution, the step S2 comprises:

[0142] Designing a state space: the intelligent agent i obtains the position p and the velocity information v of the neighbors by establishing communication with the neighbor intelligent agents, and constructs the internal observation state of the intelligent agent i according to the motion information of the intelligent agent i and the motion information of the neighbors obtained through the communication, and the internal observation state is represented as:

[0143]

[0144] wherein, A set of neighbor intelligent agents. The relative motion state of the neighbor intelligent agent j relative to the intelligent agent i is represented as:

[0145]

[0146] wherein, and and The position and velocity information of the intelligent agent j and the intelligent agent i are represented as r c The relative position normalization parameter of the neighbor intelligent agent j relative to the intelligent agent i is represented as ||v max || represents the modulus value of the maximum speed of the intelligent agent;

[0147] The observation state of the intelligent agent i on the target point can be represented as:

[0148]

[0149] wherein, p goal and r goal represent the position of the target point and the relative distance normalization parameter of the target point, respectively.

[0150] In addition, the obstacle observation information scanned by the intelligent agent through the radar is set as

[0151] Based on the observation state information described above, the observation state of the intelligent agent i can be defined as: The system state of the multi-intelligent agent consistency collaborative control is defined as a joint state s=(o1, o2, …, o i , …, o n );

[0152] Design action space: According to the agent motion model in step S1, only the acceleration control quantity needs to be provided to realize the motion control of the agent, so the action a i is defined as the acceleration control quantity:

[0153] Design reward function: The formation cooperative control error of the multi-agent system is defined as the agent reward r(s, a), and three sub-tasks of formation, obstacle avoidance and search target point are planned through task planning. The expression is as follows:

[0154] r(s, a) = -(r f + r o + r g )

[0155] Wherein r f represents the reward item of the formation sub-task, r o represents the reward item of the obstacle avoidance sub-task, and r g represents the reward item of the search target point sub-task.

[0156] The control error of the formation sub-task includes the relative position and speed consistency control error between agents, and the control error of the position and speed between the follower agent and the leader agent, that is, the leader-follower speed reward item and the distance reward item:

[0157] The distance reward item is as follows:

[0158]

[0159] In the formula, d is the relative distance between the follower agent and the leader agent, the aggregation distance range is (d a , d c ). c1 and c2 represent the formation reward factors.

[0160] The relative speed is as follows:

[0161] Δv = ||v leader -v other ||

[0162] The speed reward item between agents is as follows:

[0163]

[0164] The total reward of the formation sub-task is as follows:

[0165] r f = r a + r v

[0166] The control error of the obstacle avoidance subtask includes a relative distance reward term of the agent and other objects within the detection range;

[0167]

[0168] wherein d represents the relative distance between the agent and other agents, d a represents a relative distance normalization parameter.

[0169] The search target point subtask is a straight line reward term of the shortest search distance of the leader and a distance reward term. The straight line reward term is represented by the change in the slope of the relative position of each agent before and after as follows:

[0170]

[0171] The distance reward term is represented as follows:

[0172] r g =||k i -k||+Δd

[0173] wherein k is a fixed slope of the initial position of the leader point and the target point, and Δd represents the relative distance between the leader agent at different time points and the target point.

[0174] In the technical solution, the step S3 comprises:

[0175] A reward module based on task decomposition is designed, and a strategy network and an evaluation function are constructed in combination with a multilayer perception mechanism:

[0176] The local observation of the agent i is taken as the input of the strategy network to obtain the acceleration control amount of the agent i;

[0177] The evaluation network model: the evaluation network mainly comprises an encoder and a multilayer perception mechanism, and the global state s and the joint action a of the step S2 are input into the evaluation network model after being encoded by the encoder. The evaluation network model encodes each observation state in s to obtain a coded feature, and the features obtained by encoding are spliced to obtain a feature h s , and the joint action a is mapped by the multilayer perception mechanism to obtain a behavior feature h a , h s and h a are spliced and input into the multilayer perception mechanism to obtain an evaluation value Q;

[0178] The strategy network model: the strategy network comprises an encoder and a multilayer perception mechanism. The observation state of the agent is input into the encoder to obtain an aggregated feature, and then the aggregated feature is input into the multilayer perception mechanism to obtain the mean and variance of the behavior a i of the agent.

[0179] In the technical solution, the step S4 comprises:

[0180] The strategy updating method of deep reinforcement learning: in the multi-agent formation cooperative control task, the state space and action space attributes of the agent are completely same, and all agents are equivalent and share the same strategy network and experience pool data, the strategy network of the task is denoted as pi, and two independent evaluation networks are used to evaluate each subtask;

[0181] Evaluation network parameter updating: the evaluation network of the agent takes the system state s and joint action a as input, and optimizes the parameters through truncated double Q, and the loss function of the evaluation network is defined as follows:

[0182]

[0183] Wherein, is the experience replay pool, E D represents the mean value of multiple samples, is the parameter of the i-th evaluation network, and the target fitting value y is defined as follows:

[0184]

[0185] Wherein, r represents the reward obtained by the agent in the state s by taking action a, represents the next action resampled from the strategy, s' is the next state to which the agent in the state s takes action , and γ represents the discount factor.

[0186] Strategy network parameter updating: the strategy network of the agent i takes the observation state o i as input, and takes the maximum expected future reward plus the expected future entropy V π (s) as the optimization target, and is represented as follows:

[0187]

[0188] Wherein, α represents the entropy regularization coefficient.

[0189] The strategy optimization uses the reparameterization trick, rewrites the expectation of the action as the expectation of the noise, and takes the minimum Q value between the two Q approximators to represent Q πθ , and the strategy updating is as follows

[0190]

[0191] In the formula, ξ represents a random factor.

[0192] Finally, the target network is updated by the following loss:

[0193]

[0194] wherein, represents a soft update coefficient, represents an entropy term.

[0195] In the technical solution, the step S5 comprises:

[0196] In the policy update, we normalize the weights of different sub-tasks according to the size of the reward, and the specific representation of the weight factor is as follows:

[0197]

[0198] The policy loss update of the overall task after weighting is updated as follows:

[0199]

[0200] wherein, σ k is the normalized weight of each Q approximator. The entropy regularization coefficient α and the update of the target network are unchanged.

[0201] In summary, the method provided by the present application is based on task decomposition reinforcement learning technology, aiming to solve the problems of low efficiency of task weight allocation and weak algorithm generalization ability in the prior art. The main technical features and effects of the present application are as follows:

[0202] Construction of task decomposition framework: a task decomposition framework is designed to decompose complex tasks into multiple parallel sub-tasks. This method improves the efficiency of task processing and enhances the adaptability of the algorithm to complex environments.

[0203] Implementation of dynamic weight strategy: the present application adopts a dynamic weight strategy to achieve optimal task weight allocation. This strategy can automatically adjust the weight according to the real-time environment and task requirements, thereby improving the flexibility and efficiency of task allocation.

[0204] Design of multi-agent motion model and obstacle avoidance model: the present application establishes accurate multi-agent motion model and obstacle avoidance model, which can effectively handle the coordination and obstacle avoidance problems between multiple agents, ensuring the stability and safety of the formation.

[0205] Design and update of deep reinforcement learning policy model: through deep reinforcement learning technology, the present application designs a policy model and realizes its update method. This method enables the agent to learn from the environment and continuously optimize its behavior strategy.

[0206] Design of multi-task dynamic weight update method: the present application introduces a new multi-task dynamic weight update method, which can adjust the weight in real time according to the importance of the task and the changes in the environment, thereby improving the flexibility and efficiency of task processing.

[0207] In summary, the present application effectively solves the problems in the prior art through task decomposition reinforcement learning technology, improves the efficiency and generalization ability of multi-agent formation cooperative control. Compared with the prior art, the present application has higher flexibility and adaptability, and can effectively cope with complex and variable task environment.

Claims

1. A multi-agent formation collaborative control method based on task decomposition reinforcement learning, characterized in that: The following steps are involved: Step S1: Establish a multi-agent motion model and obstacle detection model, including constructing a multi-agent particle model, a leader-follower model, and an obstacle detection model. The constructed model provides an interactive environment for offline training of multi-agent formation collaborative control based on task decomposition reinforcement learning; Step S2: Construct the state space, action space, and reward function to meet the requirements and environmental constraints of the multi-agent task decomposition reinforcement learning formation collaborative control task. The method is as follows: State design: Based on the multi-agent motion model and obstacle detection model constructed in step S1, the local observation o of agent i is constructed by combining the relative position and speed information of the communicable neighboring agents and the position and speed information of the obstacles within the agent's detection range. i , construct the global state s = (o1, o2, ..., o i ,…,o n ); Action design: Define the acceleration of the agent as the action a of the agent i , construct joint action a=(a1, a2, ..., a i ,…,a n ); Reward function design: Tasks are decomposed based on task planning, and the control error generated by the multi-agent formation collaborative control task is used as the reward value of the agent control. The multi-agent formation includes task planning, which plans three subtasks: formation, obstacle avoidance, and target point search. The control error of the formation subtask includes the position and speed consistency control error between the agents, the position and speed control error between the following agent and the leader agent, that is, the speed reward and distance reward between the leader and the follower. The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range; The target point search subtask includes a straight-line reward item and a distance reward item for the leader's shortest search distance; Step S3, design a reward module based on task decomposition, and combine the multi-layer perceptron to build a strategy network and an evaluation network, and transform the observation state o in step S2 into i As the input of the policy network to obtain the acceleration control amount of agent i, the global state s and joint action a in step S2 are used as the input of the evaluation network to complete the state-behavior evaluation; Step S4: updating the parameters of the policy network of each subtask in step S3 by a stochastic policy optimization method of multi-agent reinforcement learning based on an information sharing mechanism; Step S5: retain the reward updated in step S4 each time, and design a dynamic weight update scheme based on the reward, weight each subtask, and complete the parameter update of the evaluation network in step S3; Step S6: The control process of the multi-agent formation collaborative control method based on task decomposition reinforcement learning includes the following steps: Step S6.1, constructing complete observation information using the observation state information obtained from the motion model and obstacle detection model in step S1; Step S6.2: Reconstruct the agent's observation state using the state construction method of step S2 and the complete observation information constructed in step S6.1; Step S6.3: Send the agent observation state reconstructed in step S6.2 into the strategy network model designed in step S3 to obtain the agent behavior; Step S6.4: Send the intelligent agent behavior obtained in step S6.3 into the intelligent agent particle motion model in step S1 to complete the motion control of the intelligent agent.

2. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 1 is characterized by: The step S1 comprises: Step S1.1: Establish a multi-agent particle motion model: Consider each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i: in, and Represent the position, velocity and acceleration vector of agent i respectively, and Respectively and The differential of , n represents the number of agents in the formation; Step S1.2: Establish the following leader-follower model: Select the first agent i l is the leader agent, and the rest are follower agents. The distance between the follower agent and the leader agent is: in, and Represents the position information of the follower agent and the leader agent respectively; Step S1.3: Establish the following obstacle detection model: Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i is expressed as: Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r d The confidence radius of the detection sensor equipped by the intelligent agent, within which the detected obstacle motion information is credible.

3. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 2 is characterized by: The step S2 comprises: Design state space: Agent i establishes communication with neighboring agents to obtain the neighbor's position p and velocity information v. Based on its own motion information and the neighbor's motion information obtained through communication, it constructs the internal observation state of agent i, which can be expressed as: in, is the set of neighboring agents, represents the relative motion state of neighbor agent j relative to agent i, which is expressed as follows: in, and and Represents the position and speed information of agent j and agent i respectively, r c represents the normalized parameter of the relative position of neighbor agent j relative to agent i, ||v max || represents the modulus of the agent's maximum speed; The observation state of agent i to the target point is expressed as: Among them, p goal and r goal Respectively represent the normalized parameters of the target point position and the relative distance to the target point; The obstacle observation information obtained by the agent through the obstacle detection model is set as Based on the observation state information described above, the observation state of agent i is defined as: The system state of multi-agent consensus cooperative control is defined as the joint state s = (o1, o2, ..., oi, ..., o n ); Design action space: According to the motion model of the agent in step S1, the motion control of the agent can be achieved by providing the acceleration control amount, so the action a of the agent i is i Defined as the acceleration control quantity: Design the reward function: The formation coordination control error of the multi-agent system is used as the agent reward r(s, a). Through task planning, three subtasks are planned: formation, obstacle avoidance, and target point search. They are expressed as follows: r(s,a)=-(r f +r o +r g ) where r f Represents the reward item of the formation subtask, r o represents the reward item of the obstacle avoidance subtask, r g Represents the reward item for the target point search subtask; The control error of the formation subtask includes the errors in the relative position and speed consistency control between the agents, and the control errors in the position and speed between the following agent and the leader agent, namely the speed bonus and distance bonus between the leader and the follower: The distance bonus items are as follows: Where d is the relative distance between the follower agent and the leader agent, and the aggregation distance range is (d a , d c ); c1 and c2 represent formation reward factors; The relative speeds are as follows: Δv=|v leader -v other | where v leader represents the speed of the leader agent, v other represents the speed of the following agent; The speed bonus between agents is as follows: The total rewards for the formation subtask are as follows: r f ＝r a +r v where r a Indicates distance bonus items; The control error of the obstacle avoidance subtask includes the relative distance reward between the agent and other objects within the detection range; Among them, d represents the relative distance between the agent and other agents, d a represents the relative distance normalization parameter; The target point search subtask is a linear reward item and a distance reward item for the leader's shortest search distance. The linear reward item is represented by the slope change of each agent's relative position before and after, as follows: in and Indicates the position of the agent before and after the moment; Distance bonus items are represented as follows: r g =|k i -k|+Δd Where k is the fixed slope between the initial position of the leader point and the target point, and Δd represents the relative distance between the leader agent and the target point before and after.

4. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 3 is characterized by: The step S3 comprises: Design a reward module based on task decomposition and combine it with a multi-layer perceptron to build a policy network and an evaluation network: Local observation of agent i: And use it as the input of the policy network to obtain the acceleration control value of agent i; Evaluation network model: The evaluation network includes an encoder and a multi-layer perceptron. The global state s and the joint action a of step S2 are encoded by the encoder and then input into the evaluation network model. The evaluation network model encodes each observation state in s to obtain the encoded features, and concatenates the encoded features to obtain the feature h. s At the same time, the joint behavior a is mapped by the multi-layer perceptron to obtain the behavior feature h a , h s With h a After splicing and passing through the multi-layer perceptron, the evaluation value Q is obtained; Strategy network model: The strategy network includes an encoder and a multi-layer perceptron. The observed state of the agent is passed through the encoder to obtain aggregated features, and then the aggregated features are sent to the multi-layer perceptron to obtain the agent's behavior a i The mean and variance of .

5. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 4 is characterized by: The step S4 comprises: Deep reinforcement learning policy update method: In multi-agent formation collaborative control tasks, the state space and behavior space properties of the agents are exactly the same, and all agents are equivalent and share the same policy network and experience pool data. The task's policy network is denoted as π, and each subtask is evaluated using two independent evaluation networks; Evaluation network parameter update: The evaluation network of the agent takes the global state s and the joint action a as input and optimizes the parameters by truncating the double Q. The loss function of the evaluation network is defined as follows: in, For the experience replay pool, Indicates the sample pool for multiple samples. is the state-action value function, which depends on the parameter represents the estimated value of taking action a in state s; is the parameter of the i-th evaluation network, and the target fitting value y is defined as follows: Among them, r represents the reward obtained by the agent when taking action a in state s, represents the next action resampled from the policy, s′ agent takes action in state s After transferring to the next state, γ represents the discount factor, Indicates selecting the minimum value from two mixed Q networks, is the Q-value function of the j-th hybrid Q network; Policy network parameter update: The policy network of agent i will observe state o i as input and maximize the expected future reward plus the expected future entropy V π (s) is the optimization target, which is expressed as follows: Among them, α represents the entropy regularization coefficient, The expected symbol indicates that the action a is averaged according to the strategy π, Q π (s, a) represents the state-action value function under policy π, that is, the value of executing action a from state s, and π(a|s) represents the probability of executing action a in state s; The policy optimization uses the reparameter trick to rewrite the expectation of the action as the expectation of the noise and take the minimum Q value between the two Q approximators to represent Q πθ , the policy is updated as follows: Where ξ represents the random factor, Indicates the expectation symbol, is the action sampled from the policy, Indicates selecting the smallest of the two Q functions; Finally, the target network is updated by the following loss: Where, represents the soft update coefficient, represents the entropy term, Represents that at time step t, according to the strategy π t Select action a t The logarithmic probability of .

6. The multi-agent formation collaborative control method based on task decomposition reinforcement learning according to claim 5 is characterized by: The step S5 comprises: When updating the strategy, the weights of different subtasks are normalized according to the size of the reward. The specific expression of the weight factor is as follows: Among them, r k represents the reward of the kth task; The policy loss update of the overall task after weighting is expressed as follows: Where, represents the Q value of the kth task, σ k The normalized weights of each Q-approximator, the entropy regularization coefficient α and the update of the target network remain unchanged.

Images