Multi-task multi-agent cooperative control method based on knowledge distillation
Through the multi-task multi-agent collaborative control method based on knowledge distillation, the problem of weak interference and strategy sharing capabilities between tasks is solved, and the learning speed and collaborative control effect of the agent are improved, which is suitable for a variety of task scenarios.
Patent Information
- Application Number
- CN202510579680.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-01
- Publication Date
- 2025-08-15
AI Technical Summary
In the continuous action space, there are problems of serious interference between tasks and weak strategy sharing capabilities in the collaborative control of multi-tasks, resulting in low training efficiency and weak agent perception capabilities.
Using a multi-task multi-agent collaborative control method based on knowledge distillation, a strategy network model that adapts to the changes in the state dimensions of multi-tasks is designed and evaluated network models are evaluated. Expert strategies are introduced for each task using knowledge distillation technology, high-quality samples are shared, and inter-task interference is reduced through multi-task hierarchical playback pool.
It significantly improves the learning speed and sample utilization rate of the agent in a multi-task environment, enhances the knowledge sharing ability between tasks, reduces inter-task interference, and improves the training efficiency and the collaborative control effect of the agent.
Smart Images
Figure CN120493977A_ABST
Abstract
Description
Technical Field
[0001] This patent belongs to the field of multi-task multi-agent systems and reinforcement learning, and provides a multi-task multi-agent collaborative control method based on knowledge distillation. Background Art
[0002] Multi-task multi-agent collaborative control technology is one of the important issues in the study of multi-agent systems and has been widely studied. In recent years, the vigorous development of artificial intelligence technology has promoted the intelligence of various fields of industrial information, and has also brought a feasible intelligent learning method to the study of multi-task multi-agent collaborative control. Reinforcement learning, as an unsupervised machine learning method, has been successfully applied to multi-task multi-agent collaborative control. Since reinforcement learning is a learning method that uses the experience of interaction with the environment and the mechanism of delayed rewards to achieve strategy evolution, compared with traditional control methods, the learned strategy has the characteristics of optimality of the entire control process and better environmental adaptability. Due to these advantages of reinforcement learning, reinforcement learning has become an important method to solve multi-task multi-agent collaborative control tasks.
[0003] However, the current multi-task multi-agent collaborative control algorithm based on reinforcement learning still has problems such as discretization of action space, serious interference between tasks and low training efficiency. This interference between tasks directly affects the agent's ability to perceive the external environment and weakens the control strategy learning effect of reinforcement learning. Therefore, how to improve the agent's ability to share knowledge between tasks and reduce interference between tasks in a continuous action space is an important problem that needs to be solved urgently. To solve this problem, the present invention addresses the problems of serious interference between tasks in a continuous action space and poor strategy sharing ability between similar tasks. This paper proposes a multi-task multi-agent collaborative control method based on knowledge distillation in a continuous action space. Summary of the Invention
[0004] The purpose of this invention is to address the problems of severe inter-task interference and weak policy sharing capabilities in multi-task, multi-agent systems in continuous action spaces. A multi-task, multi-agent collaborative control method based on knowledge distillation is proposed. This method alleviates the problems of inter-task interference and low training efficiency during the policy update process, enhances the information perception capabilities of the agents in a multi-task environment, and improves the effectiveness of collaborative control of the agents in a multi-task environment. This objective of the invention is achieved through the following technical solutions:
[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical means:
[0006] The present invention provides a multi-task multi-agent collaborative control method based on knowledge distillation, comprising the following steps:
[0007] Step S1: Establish an intelligent body particle motion model and obstacle detection model to provide an interactive environment for reinforcement learning;
[0008] Step S2: Design a state space, an action space, and a reward function. The state space includes the agent's local observations and global state. The action space is defined as the agent's acceleration. The reward function includes different reward and penalty terms depending on the task type.
[0009] Step S3: Design the input state dimension unification module and the text semantic information differentiation task module, and build the policy network model and the evaluation network model;
[0010] Step S4: Design a multi-task and multi-agent hierarchical replay pool to store experience data for different tasks;
[0011] Step S5: Update the parameters of the policy network and the evaluation network based on the policy optimization method based on knowledge distillation;
[0012] Step S6: Execute the control process, including processing input state, reconstructing observation state, generating agent behavior and controlling movement.
[0013] In the above solution, step S1 includes:
[0014] Step S1.1: Establishing the intelligent body particle motion model:
[0015] Consider each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i:
[0016]
[0017] in, and Represent the position, velocity and acceleration vector of agent i respectively, and Respectively and The differential of , n represents the number of agents;
[0018] Step S1.2: Establish the following obstacle detection model:
[0019] Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i is expressed as:
[0020]
[0021] Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r dThe confidence radius of the detection sensor equipped by the intelligent agent, within which the detected obstacle motion information is credible.
[0022] In the above solution, step S2 includes the following sub-steps:
[0023] Step S2.1: State Design:
[0024] Based on the intelligent body particle model and obstacle detection model constructed in step S1,
[0025] In the obstacle avoidance task, the local observation o of agent i is constructed using the speed information of agent i, the relative position and speed information of the communicable neighboring agents, the position information of the target point, and the position information of the detected obstacles. i ;
[0026] In the formation task, the local observation o of agent i is constructed using the speed information of agent i, the relative position and speed information of the communicable neighboring agents, and the position information of the target point. i ;
[0027] In the cluster task, the leader agent constructs a local observation based on its own speed information, the relative position and speed information of the communicable neighbor agents, and the position information of the target point. i , follower agent i uses the speed information, the relative position and speed information of the communicable neighboring agents to construct the local observation o of agent i i ;
[0028] In each task, by combining the local observations of all agents and the textual semantic information, text To construct the global state s=(o1,o2,…,o i ,…,o n , o text );
[0029] Step S2.2: Action design: Define the agent's acceleration as the agent's action a i , construct a joint action a=(a1,a2,…,a i ,…,a n );
[0030] Step S2.3: Reward function design: For obstacle avoidance tasks, formation tasks, and cluster tasks, a polynomial reward function including target point reward, obstacle avoidance penalty, acceleration penalty, and path penalty is defined respectively, where the total reward value is the weighted sum of the control errors of all agents.
[0031] In the above scheme, step S2.1 includes the following steps:
[0032] Agent i establishes communication with neighboring agents to obtain neighbor position and speed information and builds internal observation state:
[0033]
[0034] Among them, the relative motion state of neighbor agent j is expressed as:
[0035]
[0036] in, represents the relative motion state of neighbor agent j relative to agent i, The set of neighboring agents, ||v max represents the modulus of the agent’s maximum velocity, represents the position of agent i, represents the position of agent j, represents the velocity vector of agent j, represents the velocity vector of agent i, r c is the normalized parameter of the relative position to the neighboring agent;
[0037] The observation state of agent i to the target point can be expressed as
[0038]
[0039] where p g Indicates the location of the target point;
[0040] The observation state of some agents at the target point due to incomplete received information can be expressed as
[0041]
[0042] The radar observation information of obstacles received by agent i is expressed as
[0043] Build global state based on task type:
[0044] Obstacle avoidance task: s = (o1, o2, ..., o i ,…,o n , o text ),in
[0045] Formation task: s = (o1, o2, ..., o i ,…,o n , o text ),in
[0046] Cluster task: s=(o leader , o1,…, oi ,…,o n-1 , o text ), where the leader observes Follower Observation
[0047] In the above solution, the reward function is designed in step S2.3:
[0048] The system control error of the multi-agent cooperative control system is taken as the agent return r(s, a), and the control error includes the control error e of each agent. i , which is expressed as follows:
[0049]
[0050] Before the agent reaches the target point, the change in the distance between the agent and the target point is used as the intermediate reward value. The target point reward term in the reward function is designed as follows:
[0051]
[0052] Where c1 is a positive proportional constant, ε represents the allowable target distance error; R goal Indicates the reward value for reaching the target point; The distance between the agent and the target point at the previous moment t-1 and the current moment t, respectively, p des，i is the target point coordinate of agent i, is the coordinate of agent i at the current time t;
[0053]
[0054] In order to make the agent reach the target point as quickly as possible, the acceleration penalty term r is introduced acc (s t ) and the straight path penalty term, expressed as follows:
[0055]
[0056] Where c2 and c3 are positive proportional constants, is the acceleration of agent i at the current time t, is the distance between agent i and the straight line path between the starting point and the target point at the current time t;
[0057] In order to enable the agent to learn autonomous obstacle avoidance behaviors between agents and between agents and obstacles in the strategy, the obstacle avoidance penalty term is defined as follows:
[0058]
[0059] Where c4, c5 are positive proportional constants, d min，i represents the minimum distance between agent i and other agents and obstacles, d o is the radius of the obstacle avoidance buffer zone;
[0060]
[0061] The penalty for the follower to keep the distance and angle unchanged when following the leader is:
[0062] r angle (s t )=-c6·||ag i -ags i ||
[0063]
[0064] Where c6 and c7 are positive proportional constants, s l_f is the standard distance between the follower and the leader, ag i From the perspective of agent i and the leader, ags i is the standard angle between agent i and the leader;
[0065] In the obstacle avoidance environment, the total reward function of the multi-agent system is as follows:
[0066]
[0067] In a formation environment, the total reward function of the multi-agent system is as follows:
[0068]
[0069] In a cluster environment, the total reward function of the multi-agent system is as follows:
[0070]
[0071] In the above solution, step S3 is specifically as follows:
[0072] Design the input state dimension unification and text semantic information differentiation task module, and design the evaluation network model and the strategy network model based on the dimension unification and text semantic information differentiation task module and the multi-layer perceptron; use the global state s and the joint action a in step S2 as the input of the evaluation network to complete the state-behavior evaluation, and use the observation state o in step S2 as the input of the evaluation network to complete the state-behavior evaluation. i and o text As the input of the policy network model to obtain the acceleration control amount of agent i.
[0073] In the above solution, step S3 specifically includes the following steps:
[0074] Design a module that adapts to changes in multi-task state dimensions:
[0075] In the obstacle avoidance environment, the local observation of agent i is
[0076] In the formation environment, the local observation of agent i is
[0077] In a cluster environment, the observed state of the leader agent can be defined as: The observed state of follower agent i can be defined as:
[0078] Taking the highest-dimensional local observation as the benchmark, the remaining local observations are zero-filled to obtain the local observation o of agent i i，padding ;
[0079] The task modules are distinguished according to the text semantic information, and the text corresponding to each task is extracted with the language representation model Bert to extract the feature h t ;
[0080] Policy network model: The policy network will locally observe o i，pa dding is sent to the state encoder State Encoder, which is composed of multiple fully connected layers FC. The output of each fully connected layer in the state encoder is i Weighted summation with the output of the text information processing module Then it is spliced with the output of the text information module and sent to the multi-layer perceptron MLP to obtain the output action a i ;
[0081] Evaluation network model: The evaluation network consists of a module that adapts to multi-task dimensional changes, a text semantic information differentiation task module, and a multi-layer perceptron. The global state s of step S2 is concatenated with the joint behavior a and then input into the evaluation network model. The evaluation model adopts the same design as the policy model. It is first processed by the state encoder, then weighted summed with the text information output, and finally concatenated with the output of the text information module and sent to the multi-layer perceptron to obtain the evaluation value Q.
[0082] In the above solution, step S4 includes:
[0083] Put each agent’s multiple task experiences into different experience replay pools, D i，j Represents the experience replay pool used by agent i in task j. Each training round is performed. When training task j, agent i starts from D i，j Extract experience to update subsequent algorithms.
[0084] In the above solution, step S5 includes:
[0085] Policy update method for deep reinforcement learning: In the collaborative control algorithm based on deep reinforcement learning, all intelligent agents are equivalent and share the same evaluation network and policy network. The policy network is denoted as π, and is evaluated using two independent evaluation networks.
[0086] Evaluation network parameter update: The agent's evaluation network takes the system state s and joint action a as input and optimizes the parameters by minimizing the temporal difference error. The loss function of the evaluation network is defined as follows:
[0087]
[0088] Among them, N is the number of samples drawn from the experience replay pool each time, Q μ and μ represent the shared evaluation network and the current policy network respectively, θ Q The parameter of the Q function is θ, and the target fitting value y j The definition is as follows:
[0089]
[0090] Among them, γ is the discount factor, s and o represent the system state and observation state at the next moment respectively, s J represents the system state at the next moment in the jth sample extracted, represents the local observation of the i-th agent in the j-th sample extracted at the next moment, is the target strategy network of the i-th agent according to The action generated, a represents the joint action at the next moment, represents the target evaluation network;
[0091] Policy network parameter update: The policy network is optimized by gradient descent. The objective of gradient optimization for agent i is expressed as follows:
[0092]
[0093] Among them, θ μ are the parameters of the policy network, s j 、 They represent the system state in the jth sample, the local observation of the i-th agent, and the action of the n-th agent, respectively. represents the local observation of the i-th agent in the j-th sample extracted, is the strategy network of the i-th agent according to The action produced, A S is the output action distribution of the expert network in knowledge distillation, A Tis the output action distribution of the policy network;
[0094] The target network adopts a soft update method. and denote the target evaluation network and target policy network respectively, τ Q With τ μ Represent the update ratios of the target evaluation network and the target policy network, Represent the parameters of the target evaluation network and the target policy network respectively:
[0095]
[0096] Because the present invention adopts the above technical means, it has the following beneficial effects:
[0097] 1. Because the present invention adopts a deep reinforcement learning multi-agent collaborative control method based on knowledge distillation for multiple tasks in a continuous action space, it solves the problems existing in the multi-task multi-agent training process, such as severe interference between tasks during the strategy update process, weak knowledge sharing capabilities, and low training efficiency. Utilizing knowledge distillation technology, an expert strategy is introduced for each task to assist in training, effectively migrating the knowledge of the expert strategy to the shared strategy, ensuring the use of high-quality samples during strategy updates, and significantly improving the learning speed and sample utilization of the agent. This alleviates interference during multi-task training, enhances knowledge sharing capabilities, and significantly improves the training efficiency of multi-task strategies.
[0098] 2. Knowledge distillation effectively alleviates the inefficiency of the agent's exploration process and prevents the agent from falling into a local optimal state, enabling the agent to learn the optimal strategy faster.
[0099] 3. This invention effectively solves the problem of relying on specific reinforcement learning algorithms in solving multi-task problems by introducing knowledge distillation. It introduces an expert strategy to assist in training for each task without the need for a specific algorithm update process, which can achieve effective task expansion. BRIEF DESCRIPTION OF THE DRAWINGS
[0100] Figure 1 is a flow chart of the method of the present invention;
[0101] Figure 2 This is a control flow diagram of the multi-task multi-agent collaborative control method based on knowledge distillation;
[0102] Figure 3 Schematic diagram of the evaluation network and strategy network structure. DETAILED DESCRIPTION
[0103] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.
[0104] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.
[0105] like Figure 1 As shown in FIG, a multi-task multi-agent collaborative control method based on knowledge distillation includes the following steps:
[0106] Step S1: Establishing an intelligent particle motion model and an obstacle detection model. The constructed models provide an interactive environment for offline training of multi-task multi-agent collaborative control based on reinforcement learning.
[0107] Step S2: Design state, action control, and reward functions for deep reinforcement learning based on the multi-task and multi-agent collaborative control task requirements and environmental constraints. The method is as follows:
[0108] State design: Based on the agent particle model and obstacle detection model constructed in step S1, in the obstacle avoidance task, the local observation O of agent i is constructed using the velocity information of agent i, the relative position and velocity information of the communicable neighboring agents, the position information of the target point, and the position information of the detected obstacles. i In the formation task, the local observation O of agent i is constructed using the speed information of agent i, the relative position and speed information of the communicable neighboring agents, and the position information of the target point. i In the cluster task, the leader agent constructs local observations based on its own speed information, the relative position and speed information of its neighboring agents, and the position information of the target point. The follower agent i constructs its own local observations based on its speed information, the relative position and speed information of its neighboring agents. In each task, by combining the local observations of all agents and the text semantic information, text To construct the global state s=(o1,o2,…,o i ,…, o n , o text );
[0109] Action design: Define the acceleration of the agent as the action a of the agent i , construct a joint action a=(a1,a2,…,a i ,…,an );
[0110] Reward function design: The control error generated by the multi-task and multi-agent collaborative control task is used as the reward value of the agent control. The control error of the obstacle avoidance task includes the target point reward item, obstacle avoidance penalty item, acceleration change penalty item, and non-straight path penalty item; the control error of the formation task includes the target point reward item, acceleration penalty item, leader non-straight path penalty item, and follower and adjacent agent distance penalty item; the control error of the swarm task includes the leader target point penalty item, leader non-straight path penalty item, acceleration penalty item, and distance penalty item between agents;
[0111] Step S3, designing a task module for unifying the input state dimension and distinguishing the state based on text semantic information, and designing an evaluation network model and a strategy network model based on the dimension unification and text semantic information distinction task module and the multi-layer perceptron; using the global state s and the joint action a in step S2 as the input of the evaluation network to complete the evaluation of the state v behavior, and using the observation state o ... i and o text As the input of the policy network model to obtain the acceleration control value of agent i;
[0112] Step S4: By designing a multi-task and multi-agent hierarchical replay pool, each agent's experience for each task during offline training is stored separately, reducing the interference caused by the agents receiving experience from different tasks;
[0113] Step S5: Designing a deterministic strategy optimization method for multi-agent reinforcement learning based on knowledge distillation to complete the parameter update of the strategy network model and the evaluation network model in step S3;
[0114] Step S6: The control process of the multi-task multi-agent collaborative control method based on knowledge distillation includes the following steps:
[0115] Step S6.1: Process the task input using the input state dimension unification and semantic information differentiation task module of step S3, maintain dimension unification, and include task text semantic information;
[0116] Step S6.2: Reconstruct the agent's observation state using the state construction method of step S2 and the complete observation information constructed in step S6.1;
[0117] Step S6.3: Send the agent observation state reconstructed in step S6.2 into the strategy network model designed in step S5 to obtain the agent behavior;
[0118] Step S6.4: Send the intelligent agent behavior obtained in step S6.3 into the intelligent agent particle motion model in step S1 to complete the motion control of the intelligent agent.
[0119] In the above technical solution, step S1 includes:
[0120] Step S1.1: Establishing the intelligent body particle motion model:
[0121] Consider each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i:
[0122]
[0123] in, and Represent the position, velocity and acceleration vector of agent i respectively, and Respectively and The differential of , n represents the number of agents;
[0124] Step S1.2: Establish the following obstacle detection model:
[0125] Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i can be expressed as:
[0126]
[0127] Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r d The confidence radius of the detection sensor equipped by the intelligent agent, within which the detected obstacle motion information is credible.
[0128] In the above technical solution, step S2 includes:
[0129] Design state space: Agent i establishes communication with neighboring agents to obtain the position and velocity information of the neighbors. Based on its own motion information and the motion information of the neighboring agents obtained through communication, the internal observation state of agent i can be constructed, which is expressed as:
[0130]
[0131] in, represents the relative motion state of neighbor agent j relative to agent i, The set of neighboring agents is represented as follows:
[0132]
[0133] Among them, ||v max The modulus of the maximum speed of the agent, r c is the normalized parameter of the relative position to the neighboring agent;
[0134] The observation state of agent i to the target point can be expressed as
[0135]
[0136] The observation state of some agents at the target point due to incomplete received information can be expressed as
[0137]
[0138] The radar observation information of obstacles received by agent i can be expressed as
[0139] Based on the observation state information described above, in the obstacle avoidance environment, the observation state of agent i can be defined as: The system state of multi-agent cooperative control is defined as the joint state s = (o1, o2, ..., o i ,…,o n , o text ); In the formation environment, the observation state of agent i can be defined as: The system state of multi-agent cooperative control is defined as the joint state s = (o1, o2, ..., o i ,…,o n , o text ); In a cluster environment, the observed state of the leader agent can be defined as: , the observed state of follower agent i can be defined as: The system state of multi-agent cooperative control is defined as the joint state s = (o leader , o1,…, o i ,…,o n-1 , o text );
[0140] Design action space: According to the motion model of the agent in step S1, the motion control of the agent can be achieved by simply providing the acceleration control amount, so the action a of the agent i is i Defined as acceleration control:
[0141] Design reward function: The system control error of the multi-agent collaborative control system is used as the agent reward r(s, a), and the control error includes the control error e of each agent i, which is expressed as follows:
[0142]
[0143] The most important goal of the agent is to move to the target point. The reward function should include a reward for the agent reaching the target point. Considering the sparsity of the arrival reward, the change in the distance between the agent and the target point is used as the intermediate reward value before the agent reaches the target point. The target point reward term in the reward function is designed as follows:
[0144]
[0145] Where c1 is a positive proportional constant, ε represents the allowable target distance error; R goal Indicates the reward value for reaching the target point; The distance between the agent and the target point at the previous moment t-1 and the current moment t, respectively, p des，i is the target point coordinate of agent i, is the coordinate of agent i at the current time t;
[0146]
[0147] In order to make the agent reach the target point as quickly as possible, the acceleration penalty term r is introduced acc (s t ) and the straight path penalty term, expressed as follows:
[0148]
[0149] Where c2 and c3 are positive proportional constants, is the acceleration of agent i at the current time t, is the distance between agent i and the straight line path between the starting point and the target point at the current time t;
[0150] In order to enable the agent to learn autonomous obstacle avoidance behaviors between agents and between agents and obstacles in the strategy, the obstacle avoidance penalty term is defined as follows:
[0151]
[0152] Where c4, c5 are positive proportional constants, d min，i represents the minimum distance between agent i and other agents and obstacles, d o is the radius of the obstacle avoidance buffer zone;
[0153]
[0154] The penalty for the follower to keep the distance and angle unchanged when following the leader is:
[0155] r angle (s t )=-c6·||ag i -ags i ||
[0156]
[0157] Where c6, c7 are positive proportional constants, d l_f is the standard distance between the follower and the leader, ag i From the perspective of agent i and the leader, ags i is the standard angle between agent i and the leader;
[0158] In the obstacle avoidance environment, the total reward function of the multi-agent system is as follows:
[0159]
[0160] In a formation environment, the total reward function of the multi-agent system is as follows:
[0161]
[0162] In a cluster environment, the total reward function of the multi-agent system is as follows:
[0163]
[0164] In the above technical solution, step S3 includes:
[0165] Design a module that adapts to multi-task state dimension changes: In the obstacle avoidance environment, the local observation of agent i is In the formation environment, the local observation of agent i is In a cluster environment, the observed state of the leader agent can be defined as: The observed state of follower agent i can be defined as: In order to better maintain the consistency of input dimensions and the selectivity of the agent to the learned strategy, the local observation of the highest dimension is used as the benchmark, and the remaining local observations are zero-filled to obtain the local observation o of agent i i,padding ;
[0166] The task modules are distinguished according to the text semantic information, and the text corresponding to each task is extracted with the language representation model Bert to extract the feature h t ;
[0167] Policy network model: The policy network will locally observe o i,padding The output of each fully connected layer in the state encoder is sent to the state encoder. The state encoder is composed of multiple fully connected layers FC.i Weighted summation with the output of the text information processing module Then it is spliced with the output of the text information module and sent to the multi-layer perceptron MLP to obtain the output action a i .
[0168] Evaluation network model: The evaluation network consists of a module that adapts to multi-task dimension changes, a text semantic information differentiation task module, and a multi-layer perceptron. The global state s and the joint behavior a from step S2 are concatenated and then input into the evaluation network model. The evaluation model adopts the same design as the policy model. It is first processed by the state encoder, then weighted summed with the text information output, and finally concatenated with the output of the text information module and sent to the multi-layer perceptron to obtain the evaluation value Q.
[0169] In the above technical solution, step S4 includes:
[0170] Put each agent’s multiple task experiences into different experience replay pools, D i，j Represents the experience replay pool used by agent i in task j. Each training round is performed. When training task j, agent i starts from D i，j Extract experience to update subsequent algorithms.
[0171] In the above technical solution, step S5 includes:
[0172] Deep reinforcement learning strategy update method: In multi-task multi-agent collaborative control tasks, the state space and behavior space attributes of the agents are exactly the same after S3 processing. In the collaborative control algorithm based on deep reinforcement learning, all agents are equivalent and share the same evaluation network and policy network. The policy network is denoted as π and is evaluated using two independent evaluation networks.
[0173] Evaluation network parameter update: The agent's evaluation network takes the system state s and joint action a as input and optimizes the parameters by minimizing the temporal difference error. The loss function of the evaluation network is defined as follows:
[0174]
[0175] Among them, N is the number of samples drawn from the experience replay pool each time, Q μ and μ represent the shared evaluation network and the current policy network respectively, θ Q The parameter of the Q function is θ, and the target fitting value y j The definition is as follows:
[0176]
[0177] Where γ is the discount factor, s′ and o′ represent the system state and observation state at the next moment, respectively, and a represents the joint action at the next moment.
[0178] Policy network parameter update: The policy network is optimized by gradient descent. The objective of gradient optimization for agent i is expressed as follows:
[0179]
[0180] Among them, θ μ are the parameters of the policy network. The target network adopts the soft update method. and denote the target evaluation network and target policy network respectively, τ Q With τ μ Represent the update ratios of the target evaluation network and the target policy network respectively:
[0181]
[0182] In summary, the present invention has the following characteristics:
[0183] 1. Improve training efficiency and reduce interference between tasks
[0184] This paper proposes a deep reinforcement learning collaborative control method based on knowledge distillation for multi-task, multi-agent systems in continuous action spaces. This method effectively addresses the problems of severe inter-task interference, weak knowledge sharing, and low training efficiency during multi-task training. By introducing knowledge distillation technology, an expert strategy is designed for each task, and the knowledge of the expert strategy is transferred to the shared strategy, ensuring that high-quality samples are used during policy updates. This not only significantly improves the learning speed of the agents, but also increases sample utilization, reduces inter-task interference, and enhances the training efficiency of multi-task strategies.
[0185] 2. Enhance the agent’s ability to adapt to changes in task status
[0186] This paper designs a module for unifying input state dimensions and a module for distinguishing tasks using textual semantic information. These modules effectively handle changes in task state dimensions and distinguish task types using textual semantic information. This design improves the agent's adaptability to changes in task state dimensions, enabling it to quickly switch and adapt to different task requirements in a multi-task environment, thus enhancing the agent's generalization capabilities.
[0187] 3. Optimizing the agent’s exploration process and strategy learning
[0188] Through knowledge distillation, this paper effectively mitigates the inefficiencies of intelligent agents during exploration, preventing them from falling into local optima. Knowledge distillation enables agents to learn optimal strategies more quickly, improving the stability and efficiency of strategy learning. Furthermore, the introduction of knowledge distillation enables agents to share knowledge in multi-task environments, further enhancing the robustness and adaptability of strategies.
[0189] 4. Reduce experience interference between tasks
[0190] This paper designs a hierarchical replay pool for multi-task, multi-agent systems, storing experience data for different tasks separately and reducing cross-task experience interference. Through round-based training, agents draw samples only from the experience replay pool for a specific task, ensuring the independence and effectiveness of the training process and further improving the stability of multi-task training.
[0191] 5. Improve the collaborative control effect of intelligent bodies
[0192] This paper significantly improves the collaborative control of intelligent agents in multi-task environments by designing a policy optimization method based on knowledge distillation. The intelligent agents are able to efficiently complete tasks such as obstacle avoidance, formation, and clustering in complex multi-task environments, demonstrating strong autonomous obstacle avoidance, path planning, and collaborative control capabilities. By introducing knowledge distillation, the intelligent agents are able to better share knowledge between tasks, improving the overall performance of multi-task collaborative control.
[0193] 6. Enhance the agent’s ability to understand task semantics
[0194] This paper introduces a task module that distinguishes text semantics and uses language representation models (such as BERT) to extract semantic features of task text, enhancing the agent's ability to understand task semantics. This enables the agent to adjust its strategy based on the specific semantic information of the task, improving the accuracy and efficiency of task execution.
[0195] 7. Applicable to various mission scenarios
[0196] The design of this invention does not rely on a specific reinforcement learning algorithm and can be widely applied to various mission scenarios. By introducing expert strategies for each task, this invention can effectively expand the range of mission types and is suitable for a variety of complex mission scenarios such as obstacle avoidance, formation, and swarming, showing strong versatility and scalability.
[0197] 8. Improve the autonomous obstacle avoidance and path planning capabilities of intelligent entities
[0198] This paper significantly enhances the agent's autonomous obstacle avoidance and path planning capabilities by designing a polynomial reward function that includes a target point reward, obstacle avoidance penalty, acceleration penalty, and path penalty. The agent can efficiently avoid obstacles in complex environments and quickly reach its target point, demonstrating strong autonomous decision-making capabilities.
[0199] In summary, the present invention significantly improves the training efficiency of multi-task and multi-agent collaborative control, the knowledge sharing capability between tasks, and the collaborative control effect of agents by introducing knowledge distillation technology, designing an input state dimension unification module, a text semantic information differentiation task module, and a multi-task hierarchical replay pool, and has broad application prospects.
Claims
1. A multi-task multi-agent collaborative control method based on knowledge distillation, characterized in that: The following steps are involved: Step S1: Establish an intelligent body particle motion model and obstacle detection model to provide an interactive environment for reinforcement learning; Step S2: Design a state space, an action space, and a reward function. The state space includes the agent's local observations and global state. The action space is defined as the agent's acceleration. The reward function includes different reward and penalty terms depending on the task type. Step S3: Design the input state dimension unification module and the text semantic information differentiation task module, and build the policy network model and the evaluation network model; Step S4: Design a multi-task and multi-agent hierarchical replay pool to store experience data for different tasks; Step S5: Update the parameters of the policy network and the evaluation network based on the policy optimization method based on knowledge distillation; Step S6: Execute the control process, including processing input state, reconstructing observation state, generating agent behavior and controlling movement.
2. A multi-task multi-agent collaborative control method based on knowledge distillation according to claim 1, characterized in that: The step S1 comprises: Step S1.1: Establishing the intelligent body particle motion model: Consider each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i: in, and Represent the position, velocity and acceleration vector of agent i respectively, and Respectively and The differential of , n represents the number of agents; Step S1.2: Establish the following obstacle detection model: Based on the detection distance of the agent, the set of obstacles in the environment observed by agent i is expressed as: Among them, V o is the set of all obstacles in the environment, is the position of obstacle k, r d The confidence radius of the detection sensor equipped by the intelligent agent, within which the detected obstacle motion information is credible.
3. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 1 is characterized in that: The step S2 includes the following sub-steps: Step S2.1: State Design: Based on the intelligent body particle model and obstacle detection model constructed in step S1, In the obstacle avoidance task, the local observation o of agent i is constructed using the speed information of agent i, the relative position and speed information of the communicable neighboring agents, the position information of the target point, and the position information of the detected obstacles. i ; In the formation task, the local observation o of agent i is constructed using the speed information of agent i, the relative position and speed information of the communicable neighboring agents, and the position information of the target point. i ; In the cluster task, the leader agent constructs a local observation based on its own speed information, the relative position and speed information of the communicable neighbor agents, and the position information of the target point. i , follower agent i uses the speed information, the relative position and speed information of the communicable neighboring agents to construct the local observation o of agent i i ; In each task, by combining the local observations of all agents and the text semantic information o text To construct the global state s=(o1,o2,…,o i ,…,o n , o text ); Step S2.2: Action design: Define the agent's acceleration as the agent's action a i , construct a joint action a=(a1,a2,…,a i ,…,a n ); Step S2.3: Reward function design: For obstacle avoidance tasks, formation tasks, and cluster tasks, a polynomial reward function including target point reward, obstacle avoidance penalty, acceleration penalty, and path penalty is defined respectively, where the total reward value is the weighted sum of the control errors of all agents.
4. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 3 is characterized in that: The step S2.1 includes the following steps: Agent i establishes communication with neighboring agents to obtain neighbor position and speed information and builds internal observation state: Among them, the relative motion state of neighbor agent j is expressed as: in, represents the relative motion state of neighbor agent j relative to agent i, The set of neighboring agents, ||v max represents the modulus of the agent’s maximum velocity, represents the position of agent i, represents the position of agent j, represents the velocity vector of agent j, represents the velocity vector of agent i, r c is the normalized parameter of the relative position to the neighboring agent; The observation state of agent i to the target point can be expressed as where p g Indicates the location of the target point; The observation state of some agents at the target point due to incomplete received information can be expressed as The radar observation information of obstacles received by agent i is expressed as Build global state based on task type: Obstacle avoidance task: s = (o1, o2, ..., o i ,…,o n , o text ),in Formation task: s = (o1, o2, ..., o i ,…,o n , o text ),in Cluster task: s=(o leader , o1,…, o i ,…,o n-1 , o text ), where the leader observes Follower Observation 5. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 4 is characterized in that: In step S2.3, the reward function is designed: The system control error of the multi-agent cooperative control system is taken as the agent return r(s, a), and the control error includes the control error e of each agent. i , which is expressed as follows: Before the agent reaches the target point, the change in the distance between the agent and the target point is used as the intermediate reward value. The target point reward term in the reward function is designed as follows: Where c1 is a positive proportional constant, ε represents the allowable target distance error; R goal Indicates the reward value for reaching the target point; The distance between the agent and the target point at the previous moment t-1 and the current moment t, respectively, p des，i is the target point coordinate of agent i, is the coordinate of agent i at the current time t; In order to make the agent reach the target point as quickly as possible, the acceleration penalty term r is introduced acc (s t ) and the straight path penalty term, expressed as follows: Where c2 and c3 are positive proportional constants, is the acceleration of agent i at the current time t, is the distance between agent i and the straight line path between the starting point and the target point at the current time t; In order to enable the agent to learn autonomous obstacle avoidance behaviors between agents and between agents and obstacles in the strategy, the obstacle avoidance penalty term is defined as follows: Where c4, c5 are positive proportional constants, d min，i represents the minimum distance between agent i and other agents and obstacles, d o is the radius of the obstacle avoidance buffer zone; The penalty for the follower to keep the distance and angle unchanged when following the leader is: r angle (s t )=-c6·||ag i -ags i || Where c6, c7 are positive proportional constants, d l_f is the standard distance between the follower and the leader, ag i From the perspective of agent i and the leader, ags i is the standard angle between agent i and the leader; In the obstacle avoidance environment, the total reward function of the multi-agent system is as follows: In a formation environment, the total reward function of the multi-agent system is as follows: In a cluster environment, the total reward function of the multi-agent system is as follows:
6. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 5 is characterized in that: Step S3 is specifically as follows: Design the input state dimension unification and text semantic information differentiation task module, and design the evaluation network model and the strategy network model based on the dimension unification and text semantic information differentiation task module and the multi-layer perceptron; use the global state s and the joint action a in step S2 as the input of the evaluation network to complete the state-behavior evaluation, and use the observation state o in step S2 as the input of the evaluation network to complete the state-behavior evaluation. i and o text As the input of the policy network model to obtain the acceleration control amount of agent i.
7. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 6 is characterized in that: Step S3 specifically includes the following steps: Design a module that adapts to changes in multi-task state dimensions: In the obstacle avoidance environment, the local observation of agent i is In the formation environment, the local observation of agent i is In a cluster environment, the observed state of the leader agent can be defined as: The observed state of follower agent i can be defined as: Taking the highest-dimensional local observation as the benchmark, the remaining local observations are zero-filled to obtain the local observation o of agent i i,padding ; The task modules are distinguished according to the text semantic information, and the text corresponding to each task is extracted with the language representation model Bert to extract the feature h t ; Policy network model: The policy network will locally observe o i,padding The state encoder is composed of multiple fully connected layers FC, and the output of each fully connected layer in the state encoder is i Perform weighted summation with the output prompt of the text information processing module Then it is spliced with the output of the text information module and sent to the multi-layer perceptron MLP to obtain the output action a i ; Evaluation network model: The evaluation network consists of a module that adapts to multi-task dimensional changes, a text semantic information differentiation task module, and a multi-layer perceptron. The global state s of step S2 is concatenated with the joint behavior a and then input into the evaluation network model. The evaluation model adopts the same design as the policy model. It is first processed by the state encoder, then weighted summed with the text information output, and finally concatenated with the output of the text information module and sent to the multi-layer perceptron to obtain the evaluation value Q.
8. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 1, characterized in that: The step S4 comprises: Put each agent’s multiple task experiences into different experience replay pools, D i，j Represents the experience replay pool used by agent i in task j. Each training round is performed. When training task j, agent i starts from D i，j Extract experience to update subsequent algorithms.
9. The multi-task multi-agent collaborative control method based on knowledge distillation according to claim 7, characterized in that: The step S5 comprises: Policy update method for deep reinforcement learning: In the collaborative control algorithm based on deep reinforcement learning, all intelligent agents are equivalent and share the same evaluation network and policy network. The policy network is denoted as π, and is evaluated using two independent evaluation networks. Evaluation network parameter update: The agent's evaluation network takes the system state s and joint action a as input and optimizes the parameters by minimizing the temporal difference error. The loss function of the evaluation network is defined as follows: Among them, N is the number of samples drawn from the experience replay pool each time, Q μ and μ represent the shared evaluation network and the current policy network respectively, θ Q The parameter of the Q function is θ, and the target fitting value y j The definition is as follows: Where γ is the discount factor, s′ is o ′ respectively represent the system state and observation state at the next moment, s j′ represents the system state at the next moment in the jth sample extracted, represents the local observation of the i-th agent in the j-th sample extracted at the next moment, is the target strategy network of the i-th agent according to The action generated, a′ represents the joint action at the next moment, represents the target evaluation network; Policy network parameter update: The policy network is optimized by gradient descent. The objective of gradient optimization for agent i is expressed as follows: Among them, θ μ are the parameters of the policy network, s j 、 They represent the system state in the jth sample, the local observation of the i-th agent, and the action of the n-th agent, respectively. represents the local observation of the i-th agent in the j-th sample extracted, is the strategy network of the i-th agent according to The action produced, A S is the output action distribution of the expert network in knowledge distillation, A T is the output action distribution of the policy network; The target network adopts a soft update method. and denote the target evaluation network and target policy network respectively, τ Q With τ μ Represent the update ratios of the target evaluation network and the target policy network, Represent the parameters of the target evaluation network and the target policy network respectively: