Multi-agent autonomous control method and device based on general skill representation

Abstract

The present application relates to a multi-agent autonomous control method and device based on general skill representation, belonging to the technical field of multi-agent control. The method comprises: constructing a multi-agent reinforcement learning model comprising a general skill learning network, a skill allocation network and a lower layer environment interaction policy network; obtaining the current observation information and the current action information of each agent, inputting the general skill learning network to obtain a set of general skill representations that can be transferred across tasks; inputting the current observation information into the skill allocation network to generate first local trajectory information, and combining the general skill representation to allocate a target skill representation for the target agent; inputting the observation information into the lower layer environment interaction policy network to generate second local trajectory information, and combining the target skill representation to generate a control action; executing the action and collecting feedback data to update the network parameters. The present application realizes zero-sample transfer and small-sample rapid adaptation of the policy when the task type changes, significantly improving the transfer performance and sample efficiency of the multi-agent system.

Description

Technical Field

[0001] This invention relates to the field of multi-agent control technology, and in particular to a multi-agent autonomous control method and apparatus based on general skill representation. Background Technology

[0002] Multi-agent reinforcement learning has shown broad application prospects in the field of cooperative control of unmanned systems. For example, in scenarios such as drone logistics delivery, large-area environmental monitoring, and intelligent warehouse robot scheduling, multi-agent collaborative decision-making can significantly improve task execution efficiency. However, the task requirements in practical applications are often not static. Taking drone logistics delivery as an example, changes in population density and order volume in different areas will lead to dynamic increases or decreases in the number of drones deployed, thus changing the complexity of the task. Similarly, in environmental monitoring tasks, changes in the area of ​​the monitoring region or the increase or decrease in monitoring indicators will also lead to changes in the number of agents participating in the monitoring. These dynamic changes in task types pose a severe challenge to pre-trained cooperative strategies. Traditional multi-agent reinforcement learning methods are usually trained and tested in fixed scenarios. Once the number of agents or the task type in the deployment environment changes, the trained policy model often fails due to input-output dimension mismatch or is difficult to reuse due to changes in task characteristics. New policies must be trained from scratch, resulting in low sample efficiency and insufficient policy robustness.

[0003] The main reasons for the above problems are twofold. First, the neural network models used in most deep reinforcement learning methods are quite sensitive to changes in the dimensionality of input and output. When the number of agents changes, the dimensions of the observation state and action space also change, and the original parameterized policy model cannot be directly transferred to tasks of a new scale. Second, even network structures capable of handling variable-length inputs struggle to extract generalizable knowledge from the policies learned from the source task when the task type changes substantially, leading to a decline in policy performance on unseen tasks. Summary of the Invention

[0004] Therefore, it is necessary to provide a multi-agent autonomous control method and device based on general skill representation that can achieve efficient policy transfer and robust control when the task type changes dynamically, in order to address the above-mentioned technical problems.

[0005] A multi-agent autonomous control method based on general skill representation, the method comprising: Construct a multi-agent reinforcement learning model that includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network; Acquire the current observation and action information of each agent in a multi-agent system; The current observation information and current action information are input into the general skill learning network. By performing feature encoding on the current observation information and current action information, predicting subsequent observation and reward information, and constraining the differences in skill representations, a set of general skill representations that can be transferred across tasks is obtained. The current observation information of each agent is input into the skill allocation network to generate first local trajectory information; based on the first local trajectory information and the general skill representation set, a corresponding target skill representation is generated for each agent. The current observation information of each agent is input into the lower-level environmental interaction policy network to generate second local trajectory information; based on the second local trajectory information and the target skill representation, the control actions of each agent are generated. The system executes control actions for each agent while simultaneously collecting environmental feedback data and updating the parameters of the general skill learning network, the skill allocation network, and the lower-level environmental interaction strategy network.

[0006] On the other hand, a multi-agent autonomous control device based on general skill representation is also provided, comprising: The model building module is used to build a multi-agent reinforcement learning model that includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network. The information acquisition module is used to acquire the current observation information and current action information of each agent in the multi-agent system; A general skill learning module is used to input the current observation information and the current action information into the general skill learning network. By performing feature encoding on the current observation information and the current action information, predicting subsequent observation and reward information, and constraining the differences in skill representations, a set of general skill representations that can be transferred across tasks is obtained. The skill allocation module is used to input the current observation information of each agent into the skill allocation network to generate first local trajectory information; and to generate corresponding target skill representations for each agent based on the first local trajectory information and the general skill representation set. The strategy generation module is used to input the current observation information of each agent into the lower-level environment interaction policy network to generate second local trajectory information; and to generate control actions for each agent based on the second local trajectory information and the target skill representation. The execution and update module is used to execute the control actions of each intelligent agent, while collecting environmental feedback data and updating the parameters of the general skill learning network, the skill allocation network and the lower-level environmental interaction strategy network.

[0007] Compared with existing technologies, the multi-agent autonomous control method and device based on general skill representation provided by this invention have the following beneficial effects: 1. By learning skill representations that can be transferred across tasks through a general skill learning network, policies trained on the source task can be directly applied to unseen target tasks without retraining, achieving efficient transfer of zero-shot policies and improving policy reusability.

[0008] 2. When zero-sample transfer performance degrades, only a small number of samples are needed for fine-tuning, which allows the strategy to quickly adapt to new tasks. This significantly saves training time and computational resources compared to training from scratch, and improves sample efficiency.

[0009] 3. The skill allocation network and the lower-level environment interaction policy network can handle observation inputs and action outputs of varying lengths, solving the problem of traditional network models failing due to dimensional mismatch caused by changes in the number of agents.

[0010] 4. By predicting subsequent observation and reward information and constraining the differences in skill representation, we can ensure that the learned skills have clear semantic distinguishability. This enables the upper-level skill allocation network to dynamically allocate appropriate skills to the lower-level agents according to the task difficulty, thereby maintaining stable cooperative control performance and improving robustness when the task type changes.

[0011] 5. A hierarchical architecture is adopted, from skill allocation to policy execution. The upper-layer skill allocation network can generate target skill representations in a distributed manner without communication, while the lower-layer environment interaction policy network controls actions based on skill generation. This ensures both the efficient collaboration of centralized training and the flexible deployment of distributed execution. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the multi-agent autonomous control method based on general skill representation provided in Example 1; Figure 2 A flowchart of the multi-agent reinforcement learning model provided in Example 1; Figure 3 A flowchart of the permutation invariance module provided in Example 1; Figure 4 A flowchart of the permutation isomorphism module provided in Example 1; Figure 5 A flowchart illustrating the upper-layer Bayesian network product strategy in the skill allocation network provided in Example 1; Figure 6The flowchart of the lower-layer Bayesian network product strategy in the lower-layer environment interaction strategy network provided in Example 1 is shown. Figure 7 This is a structural block diagram of the multi-agent autonomous control device based on general skill representation in Example 2.

[0014] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0016] It should be noted that in this invention, the use of terms such as "first," "second," etc., is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0017] It is understood that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] Example 1 like Figure 1 As shown, this embodiment provides a multi-agent autonomous control method based on general skill representation, including the following steps: Step 201: Construct a multi-agent reinforcement learning model that includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network.

[0020] It is understood that the hierarchical reinforcement learning model constructed in this step is the basic architecture for achieving cross-task transfer and dynamic adaptation in this invention. The general skill learning network is responsible for extracting reusable skill representations from environmental interaction data; the skill allocation network, as the upper-level decision-maker, allocates appropriate skills to each agent based on the agent's local trajectory information; and the lower-level environmental interaction policy network, as the lower-level executor, generates specific control actions based on the allocated skills. The three networks work together to form a complete closed loop of "skill learning, skill allocation, and action generation".

[0021] Step 202: Obtain the current observation information and current action information of each agent in the multi-agent system.

[0022] It is understandable that multi-agent systems exist in partially observable environments, where each agent can only obtain local observational information, including its own state, the states of neighboring agents, and environmental features. Meanwhile, the actions performed in the previous moment are also of significant reference value for current decision-making. This step provides raw data for subsequent feature encoding and skill learning.

[0023] Step 203: Input the current observation information and the current action information into the general skill learning network. By performing feature encoding on the current observation information and the current action information, predicting subsequent observation and reward information, and constraining the differences in skill representations, a set of general skill representations that can be transferred across tasks is obtained.

[0024] It is understandable that this step utilizes a self-supervised prediction task to implicitly bind skill representations with state transition characteristics and reward function characteristics, so that the learned skills have clear semantic distinguishability and can be reused in different task scenarios.

[0025] Step 204: Input the current observation information of each agent into the skill allocation network to generate first local trajectory information; generate corresponding target skill representations for each agent based on the first local trajectory information and the general skill representation set.

[0026] This step understandably achieves a unification of centralized training and distributed execution by modeling the decision dependencies between agents using a directed acyclic graph (DAG) and employing knowledge distillation techniques. During training, parent node information is utilized to improve decision accuracy, while during execution, only local historical information is needed to output the target skill representation.

[0027] Step 205: Input the current observation information of each agent into the lower-level environment interaction policy network to generate second local trajectory information; generate control actions for each agent based on the second local trajectory information and the target skill representation.

[0028] This step understandably translates the skill representations assigned at the upper level into specific lower-level control instructions. It employs a structure symmetrical to the skill allocation network, also based on DAG and knowledge distillation, to ensure that each agent can independently generate coordinated actions even without communication.

[0029] Step 206: Execute the control actions of each agent, while collecting environmental feedback data and updating the parameters of the general skill learning network, skill allocation network and lower-level environmental interaction strategy network.

[0030] This step effectively achieves closed-loop optimization of the model. Through continuous interaction with the environment, it collects data on state transitions and rewards, and uses the PPO algorithm, GAE advantage estimation, and knowledge distillation loss to iteratively update the parameters of the three networks. Ultimately, this allows the model to adapt to changes in task type, achieving zero-shot transfer and rapid fine-tuning with small samples.

[0031] In the aforementioned multi-agent autonomous control method based on general skill representations, a general skill learning network learns skill representations that can be transferred across tasks. This allows policies trained on the source task to be directly applied to unseen tasks without retraining, achieving efficient transfer of zero-shot policies and improving policy reusability. Secondly, when zero-shot transfer performance degrades, only a small number of samples are needed for fine-tuning, enabling the policy to quickly adapt to new tasks. This significantly saves training time and computational resources compared to training from scratch, improving sample efficiency. Simultaneously, the skill allocation network and the lower-level environment interaction policy network can handle variable-length observation inputs and action outputs, solving the problem of traditional network models failing due to dimensional mismatch caused by changes in the number of agents. Furthermore, by predicting subsequent observations and reward information and constraining the differences in skill representations, the learned skills are ensured to have clear semantic discriminative power. This allows the upper-level network to dynamically allocate appropriate skills to the lower-level agents based on task difficulty, thus maintaining stable cooperative control performance when task types change. Finally, a hierarchical architecture of skill allocation and strategy execution is adopted. The upper-layer skill allocation network can generate target skill representations in a distributed manner under no communication conditions, while the lower-layer environment interaction strategy network controls actions based on skill generation. This ensures both the efficient collaboration of centralized training and the flexible deployment of distributed execution.

[0032] In the specific implementation of step 201, a multi-agent reinforcement learning model (PSSR) is first constructed, which includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network. Its overall topology is as follows: Figure 2 As shown. The upper-level skill allocation network is configured to contain... Each intelligent agent contains a lower-level environment interaction policy network. Assuming that each lower-level environmental interaction agent can possess different skills, then... Define the real state set of the environment as... intelligent agent The local observation space is , by observation function Determine the observation information received by the agent in the current state ,in, , The state transition equation is: ,in This is a coordinated action by lower levels.

[0033] This embodiment further models the aforementioned multi-agent reinforcement learning model (PSSR) as a Decentralized Partially Observable Markov Decision Process with Skill Representations (Dec-POMDP-SR), using tuples. It indicates. Among them, A collection of intelligent agents for the upper-level skill allocation system. A collection of intelligent agents for the lower-level environmental interaction system; These are the reward functions for the upper and lower layers, respectively. As a discount factor, This is the cutoff coefficient.

[0034] The sequential decision-making process is as follows: First, each skill-assigning agent in the upper layer determines its first local trajectory information. and strategy Select skill action Combined skill actions Subsequently, each lower-level environmental interaction agent... i Based on current observation information and assigned skill actions Output environment interaction actions joint action Environmental reception joint action Then, according to the state transition equation Update the status and provide corresponding rewards to the multi-agent system, where the rewards received by the skill allocation system are recorded as follows: The reward obtained by the multi-agent system interacting with the lower-level environment is denoted as .

[0035] The goal of the upper-level system is to find a joint strategy. To maximize the state value function or joint state action value function Lower-level alliance strategy Aiming to maximize the state value function or joint state action value function .

[0036] Furthermore, to overcome the dimensionality sensitivity problem caused by the dynamic change in the number of agents, the general skill learning network, skill allocation network, and lower-level environment interaction policy network all employ permutation invariance (PI) and permutation equivariance (PE) modules, which are independent of the input-output scale. The permutation invariance module is used to extract global features independent of the order of the agents, such as... Figure 3 As shown, the permutation invariance model is for the input feature set Through a fully connected hypernetwork with shared parameters Generate a unique weight matrix for each input vector. Then calculate the weighted result for each input vector, and apply it to all... The weighted results are summed and pooled to obtain the final output features. The calculation process is expressed as follows: ; In the formula, Indicates the permutation-invariant characteristic; Indicating targeting The generated weight matrix; Indicates to Perform fully connected layer operations.

[0037] In the permutation invariance module, since the hypernetwork parameters are shared across all inputs and the final operation is summation, the dimension of its output features is fixed and independent of the input order and the number of agents. This is irrelevant, thus achieving permutation invariance and adaptability to the number of dynamic entities.

[0038] like Figure 4 As shown, the permutation equivariance module is used to generate individual features for each agent that correspond one-to-one with the input sequence. This module, based on the PI module, further establishes the binding relationship between input and output elements. For the input feature set... First, the permutation-invariant features are calculated using the PI module. Meanwhile, a fully connected hypernetwork with shared parameters is still used. For each input Generate exclusive weights Then, for the intelligent agent... to its original features The generated weights Permutation invariant characteristics Matrix multiplication is performed to obtain the output features. The calculation process is expressed as follows: ; In the formula, Indicates output features; Indicating targeting The generated weight matrix.

[0039] This embodiment uses the PI module and the PE module together to form a feature representation scheme that is "globally invariant and individually equally variable", providing a scale-robust feature foundation for subsequent skill learning, allocation and strategy generation.

[0040] In the specific implementation of step 202, each agent in the multi-agent system makes a decision at the time of decision-making. Collect local observation information This is recorded as the current observation information, which includes the target's relative position, its own motion state, the states of neighboring agents, and environmental obstacle features. The local observation information of all agents is integrated to form a joint observation. Simultaneously, data is collected from the agents. Moment information The current action information is recorded as a fixed-dimensional binary code to overcome the problem of action dimension variation caused by changes in the number of agents. This integrates the current action information of all agents to form a joint action. The joint observations and joint actions are stored in the experience replay pool as input data for subsequent network training and inference, ensuring a consistent input data format that adapts to various network architecture requirements.

[0041] In the specific implementation of step 203, the general skill learning network is constructed based on the architecture of encoder and decoder, including an observation encoder with permutation invariance, a skill encoder, an action encoder, an observation skill decoder and a reward skill decoder constructed based on fully connected layers.

[0042] First, predefined A skill is an abstract representation of a set of basic behavioral patterns and decision-making intentions that an intelligent agent uses to perform a task, possessing general semantics, reusability, and cross-task transferability. Each skill is represented by a... Dimensional heat vector For intelligent agents The one-hot vector, current action information, and current observation information are input into the skill encoder, action encoder, and observation encoder, respectively, to obtain the corresponding embedded features: The calculation expression is: ; In the formula, Indicates the first Each skill embedding feature; Represents intelligent agents exist Action encoding features at any given moment; Represents intelligent agents exist The characteristics of the observation coding at any given time; Indicates a skill encoder; Indicates an action encoder; Indicates the observation encoder; This represents the network parameters of the skill encoder; This represents the network parameters of the motion encoder; This represents the network parameters of the observation encoder; express Observational information at any given time; express Moment information; Indicates the first The one-hot vector corresponding to each skill.

[0043] The observation encoder uses the aforementioned PI module to output fixed-dimensional features independent of the agent's order. All skill embedding features are concatenated into an initial set of general skill representations. Then, the observation embedding, action embedding, and initial general skill representation set are used. Input the observation skill decoder and the reward skill decoder respectively, and predict the predicted observation features for the next time step and the predicted reward value for the current time step. The calculation process is expressed as follows: = ; = ; In the formula, express The predicted observation characteristics at time 1 are denoted as the predicted observation characteristics at the next time 2. express The predicted reward value at time point is denoted as the predicted reward value at the current time point; This represents the network parameters of the observation skill decoder; This represents the network parameters of the reward skill decoder.

[0044] From the experience replay pool Mid-sampled quadruplets Calculate the reward prediction loss between the predicted reward value and the actual reward value; calculate the cosine similarity loss between different skill embedding vectors as a constraint on skill representation differences, the calculation expression is: ; Then, the observation prediction loss, reward prediction loss, and skill representation difference constraint are weighted and summed to obtain the total loss, which is expressed as: ; In the formula, Indicates the total loss; Indicates the observed and predicted loss; Indicates the predicted loss from the reward; Represents constraints on the differences in skill representation; , This represents the preset weight coefficients of the loss function; Indicates from the experience replay pool Mid-sampled quadruplets The mathematical expectation; This represents the joint observation of all intelligent agents; Represents the actions of the intelligent agent; This represents the actual reward value for all agents; This represents the actual observation characteristics of all agents at the next moment; This represents the observation skill decoder used by the agent. according to Action coding features at any time, The output is the observation coding features at time and the initial general skill representation set. Predictive observation characteristics at time ; This represents the network parameters of the observation skill decoder; This represents a reward skill decoder used by the agent. according to Action coding features at any time, The output is the observation coding features at time and the initial general skill representation set. Predicted reward value at time ; This represents the network parameters of the reward skill decoder; Represents intelligent agents The actual observation characteristics at the next moment; and Cosine similarity between them; Represents a set of skill indexes; Indicates the number of skills.

[0045] By minimizing the total loss to update the network parameters of each encoder and decoder, the final output is a set of general skill representations that can be transferred across tasks. .

[0046] In the specific implementation of step 204, such as Figure 5As shown, the skill allocation network adopts a feature representation network that is independent of the input and output scale and has permutation invariance and permutation equivariance. It includes a PI module, a fully connected layer (FC), a nonlinear activation layer (ReLU), a temporal memory network (GRU), and a PE module connected in sequence.

[0047] For intelligent agents Current observation information The data passes sequentially through a PI module, a fully connected layer (FC), ReLU activation, a gated recurrent unit (GRU), and a PE module to obtain first local trajectory information with fixed dimensions that is independent of the number of agents. The first local trajectory information preserves historical observations and action sequences, and maintains the same variability in agent number permutations.

[0048] In the skill allocation network, a directed acyclic graph is predefined among the agents. ,in, Let be the vertex, representing the set of indices of agents in the skill allocation network; The set of edges connecting vertices; the agent Treating the agent as the one currently making a decision, obtain the set of parent nodes according to the directed acyclic graph. and the corresponding set of parent actions .

[0049] Building an upper-level teacher network and upper-level student networks During the training phase, the upper-level teacher network uses the multi-head attention (MHA) module to model parent node dependencies and optimizes by maximizing the truncated policy gradient objective to update the parameters of the upper-level teacher network. The expression is: ; ; In the formula, Represents agents in a skill allocation network exist The advantage function at time; Indicates the discount factor; Indicates the parameters estimated by the generalized dominance function; Represents agents in a skill allocation network exist The instant reward value gained at any given moment; This indicates the upper-level skill allocation network. The value function of the state; This indicates the upper-level skill allocation network. The value function of the state; This represents the optimization loss function of the upper-level teacher network; This represents the number of agents in the skill allocation network; This indicates that the state-action pair is run using the historical strategy in the upper-level skill allocation network. To solve for the expectation; This represents the upper-level teacher network; This indicates the historical strategy for allocating upper-level skills; This represents the truncation function; This represents the cutoff coefficient.

[0050] The upper-level student network is optimized by minimizing the KL divergence with the teacher network's output distribution to update the parameters of the lower-level student network. The expression is: ; In the formula, This indicates the loss of higher-level knowledge during distillation. Indicates the distribution of students from the upper-level student network. Mid-sampling skill allocation sequence The mathematical expectation; This refers to the upper-level student network.

[0051] After training is complete, deployment will only use the upper-level student network. The first local trajectory information With the set of general skill representations Perform vector dot product to output the target skill representation for each agent. This is then passed to the lower-level environment interaction strategy network.

[0052] In the specific implementation of step 205, such as Figure 6 As shown, the lower-level environment interaction strategy network structure is symmetrical with the skill allocation network. It adopts a feature representation network that is independent of the input and output scale and has permutation invariance and permutation equivariance. It includes a PI module, a fully connected layer (FC), a nonlinear activation layer (ReLU), a temporal memory network (GRU), and a PE module connected in sequence.

[0053] For intelligent agents Current observation information The data passes sequentially through a PI module, a fully connected layer (FC), ReLU activation, a gated recurrent unit (GRU), and a PE module to obtain second local trajectory information with fixed dimensions that is independent of the number of agents. The second local trajectory information retains the agent's historical decision-making information and maintains the permutation-like property of the agent's ID number.

[0054] In the lower-level environment interaction policy network, a directed acyclic graph is defined among the agents in the lower-level environment interaction policy network. ,in, , where is a vertex, representing the number of agents in the lower-level environment interaction policy network; The set of edges connecting vertices; the agent Treating the agent as the one currently making a decision, obtain the set of parent nodes according to the directed acyclic graph. and the corresponding set of parent actions and obtain the corresponding target skill representation. .

[0055] Building a network of lower-level teachers and lower-level student networks During the training phase, the lower-level teacher network also utilizes the multi-head attention (MHA) module to model parent node dependencies and optimizes by maximizing the truncated policy gradient objective to update the parameters of the lower-level teacher network. The expression is: ; ; In the formula, This represents the agent in the lower-level environment interaction policy network. exist The advantage function at time; Indicates the discount factor; Indicates the parameters estimated by the generalized dominance function; This represents the agent in the lower-level environment interaction policy network. exist The instant reward value gained at any given moment; This indicates that the underlying environment interaction multi-agent system is in State value function; This indicates that the underlying environment interaction multi-agent system is in State value function; This represents the optimization loss function of the lower-level teacher network; This indicates the number of agents in the lower-level environment interaction policy network; This indicates that the state-action pair is generated using the historical policy in the lower-level policy network. To solve for the expectation; This indicates the current strategy of the lower-level teacher network; This represents the historical strategy of lower-level teacher networks; This represents the truncation function; Indicates the cutoff coefficient; The target skill is represented by the agent. Assigned to the Each skill is embedded in a feature.

[0056] The lower-level student network is optimized by minimizing the KL divergence with the teacher network's output distribution to update the parameters of the lower-level student network. The expression is: ; In the formula, This indicates the loss of knowledge during distillation. Indicates the distribution of the student network from the lower level. Mid-sampling joint action The mathematical expectation; This refers to the lower-level student network.

[0057] After the training is completed, the lower-level student network The second local trajectory information With target skill representation Perform vector concatenation to generate an intelligent agent. Control actions .

[0058] Furthermore, during the training phase, both the skill allocation network and the lower-level environment interaction policy network establish a joint action distribution through a Bayesian network product policy. The joint distribution expression of the Bayesian network product policy is as follows: ; In the formula, This represents a joint strategy for multiple intelligent systems; Indicates a joint action; Indicates the state of the environment; Represents an intelligent agent based on a Bayesian network. Strategies; Represents intelligent agents The action.

[0059] In the specific implementation of step 206, all agents perform joint actions. The environment is determined by state transition probabilities. Enter the next state and provide instant rewards. and The immediate reward for each agent is determined according to the task design. The corresponding parameters are stored in the experience pools of the general skill learning network, the skill allocation network, and the lower-level environment interaction policy network. It should be noted that the upper-level skill allocation network is not executed at every lower-level decision step, but rather at preset intervals between lower-level decision steps, and the upper-level reward value... With lower-level reward value The following cumulative relationship is satisfied:

[0060] In the formula, Indicates the preset interval. When When =1, the rewards are shared between the upper and lower levels. = At this point, the optimization objective of the entire hierarchical system is to maximize the state-value function of the joint policy. .

[0061] In each training cycle, if the current round number is less than the preset number of rounds for skill representation learning, the parameters in the multi-agent reinforcement learning model are updated accordingly.

[0062] Specifically, mini-batch data is randomly sampled from the experience pool of the general skill learning network, and gradient descent is performed based on the total loss in step 203 to update the parameters in the observation encoder, skill encoder, action encoder, observation skill decoder, and reward skill decoder.

[0063] Randomly sample small batches of data from the experience pool of the skill allocation network and perform joint optimization by gradient descent according to the optimization loss functions in step 204 to update the parameters in the upper-level teacher network and the upper-level student network.

[0064] Randomly sample small batches of data from the experience pool of the lower-level environment interaction policy network, and perform joint optimization by gradient descent according to the optimization loss functions in step 205 to update the parameters in the lower-level teacher network and the lower-level student network.

[0065] After sufficient training, the multi-agent reinforcement learning model can obtain high-performance policies on the source task and can quickly adapt to unseen tasks with zero-sample transfer or only a small number of samples for fine-tuning.

[0066] It should be understood that, although this embodiment Figure 1 The steps are shown sequentially as indicated by the arrows, but they are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0067] Example 2 Based on the multi-agent autonomous control method based on general skill representation in Embodiment 1, this embodiment discloses a multi-agent autonomous control device based on general skill representation, such as... Figure 7As shown, the multi-agent autonomous control device based on general skill representation includes: a model building module 401, an information acquisition module 402, a general skill learning module 403, a skill allocation module 404, a policy generation module 405, and an execution and update module 406, wherein: The model building module 401 is used to build a multi-agent reinforcement learning model that includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network.

[0068] The information acquisition module 402 is used to acquire the current observation information and current action information of each agent in the multi-agent system.

[0069] The general skill learning module 403 is used to input the current observation information and the current action information into the general skill learning network. By performing feature encoding on the current observation information and the current action information, predicting subsequent observation and reward information, and constraining the differences in skill representations, a set of general skill representations that can be transferred across tasks is obtained.

[0070] The skill allocation module 404 is used to input the current observation information of each agent into the skill allocation network to generate first local trajectory information; and to generate corresponding target skill representations for each agent based on the first local trajectory information and the general skill representation set.

[0071] The strategy generation module 405 is used to input the current observation information of each agent into the lower-level environment interaction policy network to generate second local trajectory information; and to generate control actions for each agent based on the second local trajectory information and the target skill representation.

[0072] The execution and update module 406 is used to execute the control actions of each agent, while collecting environmental feedback data and updating the parameters of the general skill learning network, the skill allocation network and the lower-level environmental interaction strategy network.

[0073] In this embodiment, the specific working processes and principles of the model building module 401, information acquisition module 402, general skill learning module 403, skill allocation module 404, strategy generation module 405, and execution and update module 406 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in a computer device in hardware form, or it can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0074] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multi-agent autonomous control method based on general skill representation, characterized in that, The method includes: Construct a multi-agent reinforcement learning model that includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network; Acquire the current observation and action information of each agent in a multi-agent system; The current observation information and current action information are input into the general skill learning network. By performing feature encoding on the current observation information and current action information, predicting subsequent observation and reward information, and constraining the differences in skill representations, a set of general skill representations that can be transferred across tasks is obtained. The current observation information of each agent is input into the skill allocation network to generate first local trajectory information; based on the first local trajectory information and the general skill representation set, a corresponding target skill representation is generated for each agent. The current observation information of each agent is input into the lower-level environmental interaction policy network to generate second local trajectory information; based on the second local trajectory information and the target skill representation, the control actions of each agent are generated. The system executes control actions for each agent while simultaneously collecting environmental feedback data and updating the parameters of the general skill learning network, the skill allocation network, and the lower-level environmental interaction strategy network.

2. The multi-agent autonomous control method based on general skill representation according to claim 1, characterized in that, The general skill learning network is constructed based on an encoder and decoder architecture, including an observation encoder with permutation invariance, a skill encoder, an action encoder, an observation skill decoder, and a reward skill decoder constructed based on fully connected layers; The current observation information, current action information, and preset one-hot vector of each agent are input into the corresponding encoder for processing, and the skill embedding features, action coding features and observation coding features are output. The skill embedding features are concatenated into an initial set of general skill representations; The action coding features, the observation coding features, and the initial general skill representation set are respectively input into each decoder for processing, and the predicted observation features for the next time step and the predicted reward value for the current time step are output. Calculate the observation prediction loss between the predicted observation features and the actual observation features at the next time step; calculate the reward prediction loss between the predicted reward value and the actual reward value. The cosine similarity loss between different skill embedding vectors is calculated as a constraint on the difference in skill representation; The observation prediction loss, the reward prediction loss, and the skill representation difference constraint are weighted and summed to obtain the total loss. The network parameters of each encoder and decoder are updated by minimizing the total loss, and finally a set of general skill representations that can be transferred across tasks is output.

3. The multi-agent autonomous control method based on general skill representation according to claim 2, characterized in that, The current observation information, current action information, and preset one-hot vectors of each agent are input into the corresponding encoders for processing, and the outputs skill embedding features, action coding features, and observation coding features are calculated as follows: ； In the formula, Indicates the first Each skill embedding feature; Represents intelligent agents exist Action encoding features at any given moment; Represents intelligent agents exist The characteristics of the observation coding at any given time; Indicates a skill encoder; Indicates an action encoder; Indicates the observation encoder; This represents the network parameters of the skill encoder; This represents the network parameters of the motion encoder; This represents the network parameters of the observation encoder; express Observational information at any given time; express Moment information; Indicates the first The one-hot vector corresponding to each skill.

4. The multi-agent autonomous control method based on general skill representation according to any one of claims 1 to 3, characterized in that, The function expression for the total loss is: ； in: ； In the formula, Indicates the total loss; Indicates the observed and predicted loss; Indicates the predicted loss from the reward; Represents constraints on the differences in skill representation; , This represents the preset weight coefficients of the loss function; Indicates from the experience replay pool Mid-sampled quadruplets The mathematical expectation; This represents the joint observation of all intelligent agents; This represents the joint action of all intelligent agents; This represents the actual reward value for all agents; This represents the actual observation characteristics of all agents at the next moment; This represents the observation skill decoder used by the agent. according to Action coding features at any time, The output is the observation coding features at time and the initial general skill representation set. Predictive observation characteristics at time ; This represents the network parameters of the observation skill decoder; This represents a reward skill decoder used by the agent. according to Action coding features at any time, The output is the observation coding features at time and the initial general skill representation set. Predicted reward value at time ; This represents the network parameters of the reward skill decoder; Represents the initial set of general skill representations; Represents intelligent agents The actual observation characteristics at the next moment; Indicates skill embedding features With skills embedded features Cosine similarity between them; Represents a set of skill indexes; Indicates the number of skills.

5. The multi-agent autonomous control method based on general skill representation according to any one of claims 1 to 3, characterized in that, The skill allocation network adopts a feature representation network that is independent of input and output scale and has permutation invariance and permutation equivariance. The current observation information of each agent is input into the skill allocation network to obtain first local trajectory information with fixed dimensions that is independent of the number of agents. Define a directed acyclic graph (DAG) between agents in a skill allocation network; [then define the agents...] The agent is considered as the one currently making a decision, and the set of parent nodes and the corresponding set of parent actions are obtained according to the directed acyclic graph. Construct upper-level teacher networks and upper-level student networks; During the training phase, the upper-level teacher network is optimized by maximizing the truncated policy gradient objective; the upper-level student network is optimized by minimizing the KL divergence with the output distribution of the teacher network, and a joint action distribution is established through a Bayesian network product strategy. After training, the upper-level student network performs a vector dot product between the first local trajectory information and the general skill representation set, and outputs the target skill representation corresponding to each agent.

6. The multi-agent autonomous control method based on general skill representation according to claim 5, characterized in that, The upper-level teacher network is optimized by maximizing the objective of the truncated policy gradient, expressed as: ； The upper-level student network is optimized by minimizing the KL divergence with the teacher network's output distribution, expressed as: ； In the formula, Represents agents in a skill allocation network exist The advantage function at time; Indicates the discount factor; This represents the optimization loss function of the upper-level teacher network; This represents the number of agents in the skill allocation network; This indicates that the state-action pair is run using the historical strategy in the upper-level skill allocation network. To solve for the expectation; This represents the upper-level teacher network; Represents the old teacher strategy network; This represents the truncation function; Indicates the cutoff coefficient; This indicates the loss of higher-level knowledge during distillation. Indicates the distribution of students from the upper-level student network. Mid-sampling skill allocation sequence The mathematical expectation; This refers to the upper-level student network.

7. The multi-agent autonomous control method based on general skill representation according to any one of claims 1 to 3, characterized in that, The lower-level environment interaction strategy network adopts a feature representation network that is independent of input and output scale and has permutation invariance and permutation equivariance. The current observation information of each agent is input into the lower-level environment interaction policy network to obtain second local trajectory information with fixed dimensions that is independent of the number of agents. Define a directed acyclic graph (DAG) between agents in the lower-level environment interaction policy network; and connect the agents... The target skill representation is obtained by treating the agent as the agent to be decided at present, obtaining the set of parent nodes and the corresponding set of parent actions according to the directed acyclic graph; Build lower-level teacher networks and lower-level student networks; During the training phase, the lower-level teacher network is optimized by maximizing the gradient objective of the truncated policy; the lower-level student network is optimized by minimizing the KL divergence with the output distribution of the teacher network, and a joint action distribution is established through a Bayesian network product policy. After training, the lower-level student network concatenates the second local trajectory information with the target skill representation to generate the control actions of each agent.

8. The multi-agent autonomous control method based on general skill representation according to claim 7, characterized in that, The lower-level teacher network is optimized by maximizing the truncated policy gradient objective, expressed as: ； The lower-level student network is optimized by minimizing the KL divergence with the teacher network's output distribution, expressed as: ； In the formula, This represents the agent in the lower-level environment interaction policy network. exist The advantage function at time; This represents the optimization loss function of the lower-level teacher network; This indicates the number of agents in the lower-level environment interaction policy network; This indicates that the state-action pair is generated using the historical policy in the lower-level policy network. To solve for the expectation; This indicates the current strategy of the lower-level teacher network; This represents the historical strategy of lower-level teacher networks; This represents the truncation function; Indicates the cutoff coefficient; This indicates the loss of knowledge during distillation. Demonstrates the distribution of student networks from lower levels. Mid-sampling joint action The mathematical expectation; The target skill is represented by the agent. Assigned to the Each skill is embedded in a feature.

9. The multi-agent autonomous control method based on general skill representation according to any one of claims 1 to 6, characterized in that, The skill allocation network performs skill allocation once at preset intervals for lower-level decision steps, and the upper-level reward value... With lower-level reward value The following cumulative relationship is satisfied: In the formula, Indicates the preset interval.

10. A multi-agent autonomous control device based on general skill representation, characterized in that, The device includes: The model building module is used to build a multi-agent reinforcement learning model that includes a general skill learning network, a skill allocation network, and a lower-level environment interaction policy network. The information acquisition module is used to acquire the current observation information and current action information of each agent in the multi-agent system; A general skill learning module is used to input the current observation information and the current action information into the general skill learning network. By performing feature encoding on the current observation information and the current action information, predicting subsequent observation and reward information, and constraining the differences in skill representations, a set of general skill representations that can be transferred across tasks is obtained. The skill allocation module is used to input the current observation information of each agent into the skill allocation network to generate first local trajectory information; and to generate corresponding target skill representations for each agent based on the first local trajectory information and the general skill representation set. The strategy generation module is used to input the current observation information of each agent into the lower-level environment interaction policy network to generate second local trajectory information; and to generate control actions for each agent based on the second local trajectory information and the target skill representation. The execution and update module is used to execute the control actions of each intelligent agent, while collecting environmental feedback data and updating the parameters of the general skill learning network, the skill allocation network and the lower-level environmental interaction strategy network.

Images