Heterogeneous equipment control method based on heterogeneous multi-agent reinforcement learning algorithm

Abstract

The invention discloses a heterogeneous equipment control method based on a heterogeneous multi-agent reinforcement learning algorithm, and the method aims at the equipment heterogeneity problem in a multi-agent system, models a heterogeneous equipment cooperative control problem into a heterogeneous multi-agent reinforcement learning problem, employs a centralized training distributed execution framework, and achieves the heterogeneous equipment control. The optimal-quality multi-agent strategy network is obtained, and efficient control over heterogeneous equipment is achieved. According to the method, cooperative control of different types of equipment can be processed, the adaptability and convergence of the algorithm are improved, and the method has good robustness and expansibility and is suitable for multi-agent systems with different scales and complexity.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a heterogeneous device control method based on a heterogeneous multi-agent reinforcement learning algorithm. Background Art

[0002] Multi-agent systems are an important research direction in the field of artificial intelligence and control, and are widely used in many fields such as robot group control, drone formation, smart grid, etc. In a multi-agent system, each agent completes a complex task by working together. However, in practical applications, different agents usually have different physical properties, observation spaces, action spaces, and decision logics, that is, there is significant heterogeneity between agents.

[0003] Most traditional multi-agent reinforcement learning algorithms are based on the homogeneity assumption, that is, all agents use the same policy network structure and parameters. This assumption greatly limits the applicability of the algorithm in actual scenarios with heterogeneous agents. In addition, traditional algorithms face problems such as unstable training and difficulty in convergence when dealing with heterogeneity and complex interactions.

[0004] In the existing technology, there are relatively limited methods to solve the heterogeneity problem in multi-agent systems. Some methods try to deal with heterogeneity through individual optimization and adjustment, but these methods often have difficulty in achieving effective collaborative control in large-scale multi-agent systems. At the same time, the dynamic characteristics of the environment of the multi-agent system are complex and changeable. Traditional methods find it difficult to flexibly adjust the strategy update order according to environmental changes, resulting in low strategy learning efficiency and slow convergence.

[0005] In order to solve the above problems, it is urgent to develop a new type of multi-agent reinforcement learning algorithm, which should be able to effectively handle heterogeneous agent environments, improve the applicability and convergence of the algorithm, and have good robustness and scalability to adapt to multi-agent systems of different scales and complexities. Summary of the invention

[0006] Aiming at the heterogeneity problem in multi-agent systems, designing a novel reinforcement learning algorithm has important theoretical significance and practical value, and plays a key role in promoting the application and development of multi-agent systems. The purpose of the present invention is to propose a heterogeneous device control method based on a heterogeneous multi-agent reinforcement learning algorithm, which uses a centralized training distributed execution framework to dynamically adjust the strategy update sequence of the device using the dynamic characteristics of the environment, achieves monotonic strategy improvement, and significantly improves the convergence speed and applicability of the algorithm. The method of the present invention performs well in the collaborative control of heterogeneous devices, and provides important technical support for the practical application of multi-agent systems.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A heterogeneous equipment control method based on a heterogeneous multi-agent reinforcement learning algorithm comprises the following steps:

[0009] The cooperative control problem of heterogeneous devices is modeled as a heterogeneous multi-agent reinforcement learning problem, and the state space, action space, transition probability function, reward function and objective function of the heterogeneous multi-agent reinforcement learning are defined; the heterogeneous multi-agent includes different types of agents that perform different actions;

[0010] A centralized training distributed execution framework is adopted. First, the relevant parameters are initialized. During the centralized training process, the sampled trajectory data is stored in the experience replay pool. The update order of the heterogeneous multi-agent is calculated based on the dynamic characteristics of the environment, and the parameterized proxy function is calculated. The agent strategy is sequentially updated in the direction of maximizing the parameterized proxy function, achieving monotonic improvement of the policy network parameters of the heterogeneous multi-agent.

[0011] During distributed execution, the updated heterogeneous multi-agent policy network is distributedly executed on heterogeneous devices to coordinate and control different types of heterogeneous devices;

[0012] Repeat the sequential updating steps of the above heterogeneous multi-agent strategy until the policy network parameters converge, and finally obtain the converged optimal multi-agent policy network, which is executed to achieve optimal control of heterogeneous devices.

[0013] Preferably, the characteristics of the heterogeneous equipment include:

[0014] Each device has different physical characteristics, functions, and control objectives;

[0015] Each device has different observation space and action space;

[0016] Each device can perform different types of control tasks, such as temperature control, pressure regulation, energy consumption management, etc.

[0017] Preferably, modeling the cooperative control problem of heterogeneous devices as a heterogeneous multi-agent reinforcement learning problem includes:

[0018] The state space, action space, transition probability function, reward function and objective function of heterogeneous multi-agent reinforcement learning are defined according to the characteristics of heterogeneous devices; the state space includes the local observations of the heterogeneous devices; the action space is the joint action space of the heterogeneous devices; the transition probability function describes the probability of the agent transferring from one state to another; the reward function is designed based on global information and is used to evaluate the contribution of the joint action to the collaborative task of heterogeneous devices; the objective function represents the expected cumulative reward of the agent under the strategy.

[0019] Preferably, the centralized training distributed execution framework includes:

[0020] Initialize the policy network parameters and global value function network parameters of each heterogeneous multi-agent, configure the experience replay pool, and set the initial environment state and the set of each heterogeneous multi-agent. In the centralized training phase, collect the state, action, result and reward data of each heterogeneous multi-agent during the execution process and store them in the experience replay pool. Calculate the update order of the heterogeneous multi-agent based on the dynamic characteristics of the environment, and then calculate the parameterized proxy function. Update the policy network parameters of each heterogeneous multi-agent according to the parameterized proxy function sequence. In the distributed execution phase, distribute the updated policy network parameters to all heterogeneous devices, so that each heterogeneous device can independently execute tasks according to the updated strategy and collect new execution data.

[0021] Preferably, determining the update order of the heterogeneous multi-agents based on the dynamic characteristics of the environment comprises:

[0022] Based on the application scenario, obtain indicators that reflect the dynamic characteristics of the environment;

[0023] According to the application scenario, the importance of heterogeneous multi-agents is ranked according to indicators to generate an update order for the heterogeneous multi-agents.

[0024] Preferably, the calculating parameterized proxy function comprises:

[0025] Calculate the parameterized proxy function of the i-th agent based on the sampled information

[0026]

[0027] in, Represents the current strategy of the i-th agent Last updated policy The probability ratio, a i represents the action of the ith agent, and s represents the current state; is the advantage function, which is calculated by the difference between the action value function and the state value function; clip(g t (θ),1±ε) is a clipping method to ensure that g t The value of (θ) does not exceed the range of [1-ε,1+ε] set by the parameter ε.

[0028] Preferably, the sequential updating of the heterogeneous multi-agent strategy in the direction of maximizing the parameterized proxy function comprises: updating the strategy network parameters of each heterogeneous multi-agent in the direction of maximizing the parameterized proxy function according to the calculated update order of the heterogeneous multi-agent: where θ k and θ k+1Represent the current and updated policy network parameters, π θ represents the policy with policy network parameters θ.

[0029] The present invention provides a heterogeneous equipment control method based on a heterogeneous multi-agent reinforcement learning algorithm, which has the following beneficial effects:

[0030] This paper specifically solves the problem that traditional multi-agent reinforcement learning algorithms cannot effectively handle the heterogeneity of agents. By establishing a heterogeneous multi-agent reinforcement learning problem model and designing an algorithm framework based on centralized training and distributed execution, it can effectively handle the collaborative learning of different types of agents.

[0031] An agent update sequence strategy based on the dynamic characteristics of the environment is proposed, which can dynamically adjust the agent update priority according to different environmental dynamic characteristics and improve the adaptability and convergence of the algorithm.

[0032] An innovative monotonic strategy improvement mechanism based on parameterized proxy functions was designed, which achieved stable strategy convergence by sequentially updating the policy network parameters of heterogeneous multi-agents.

[0033] The algorithm has good scalability and can be applied to heterogeneous multi-agent systems of different scales and complexities, providing wide applicability for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flow chart of the heterogeneous equipment control method based on heterogeneous multi-agent reinforcement learning algorithm.

[0035] Figure 2 Diagram of the distributed execution framework for centralized training. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1

[0038] like Figure 1 As shown, the present invention provides a heterogeneous device control method based on a heterogeneous multi-agent reinforcement learning algorithm, comprising the following steps:

[0039] Step S1: Modeling the cooperative control problem of heterogeneous devices as a heterogeneous multi-agent reinforcement learning problem, wherein the heterogeneous multi-agent includes different types of agents that perform different actions, specifically including:

[0040] 1. State space S: The state space is the global state shared by heterogeneous multi-agents, including the local observation information of each agent.

[0041] 2. Action space A: The action space is the Cartesian product of the action spaces of each agent, that is, A = A 1 ×A 2 ×...×A n , where A i represents the action space of the ith agent. Different types of agents can perform different actions, which are represented by express.

[0042] 3. Transition probability function P: The transition probability function P: S×A×S→[0,1] describes the probability of transitioning from state s to state s′ when taking joint action a.

[0043] 4. Reward function r: The reward function r: S×A→R is designed based on global information and is used to evaluate the contribution of joint actions to multi-agent collaborative tasks.

[0044] 5. Objective function J(π): Objective function represents the expected cumulative reward under the joint strategy π, where S~ρ π Indicates that state S is the state distribution ρ under strategy π π Sampling, a~π means that action a is sampled from state S according to strategy π, γ is the discount factor, r t is the reward at time step t. The optimization goal is to find the optimal joint strategy π * , so that the objective function J(π) reaches its maximum value.

[0045] Step S2: Based on the centralized training distributed execution (CTDE) framework, monotonic policy improvement is achieved by sequentially updating the policy network parameters of the heterogeneous multi-agent. Figure 2 As shown, specifically including:

[0046] 1. Initialization parameters: Initialize the policy network parameters of each heterogeneous multi-agent The global value function network parameter ξ. Among them, Denote the initial policy network parameters of the ith agent. Set the experience replay pool D to store heterogeneous multi-agent interaction data. Determine the initial environment state s, the heterogeneous multi-agent set N, the number of iterations T, and the sampling length H.

[0047] 2. Centralized training:

[0048] a. Sampled trajectory data: record the state, action, next state and reward of each agent at each time step, and integrate them into a data set Deposit into experience replay pool D. Among them, is the state of the ith agent at time step t, is the action of the ith agent at time step t, is the next state, r t is the reward at time step t.

[0049] b. Calculate the update order of heterogeneous multi-agents based on the dynamic characteristics of the environment: obtain indicators that reflect the dynamic characteristics of the environment, such as the sensitivity index of each agent relative to other agents; rank the importance of heterogeneous multi-agents based on the indicator to generate the update order of heterogeneous multi-agents.

[0050] c. Calculate the parameterized proxy function: Use the parameterized proxy function to replace the objective function, and use the data stored in the experience replay pool to calculate the parameterized proxy function:

[0051]

[0052] in, Represents the current strategy of the i-th agent The previous strategy The probability ratio, a i represents the action of the ith agent, and s represents the current state; is the advantage function, which is calculated by the difference between the action value function and the state value function. t (θ),1±ε) is a clipping method to ensure that g t The value of (θ) does not exceed the range of [1-ε,1+ε] set by the parameter ε.

[0053] d. Strategy update: Update the policy network parameters of each heterogeneous agent in the direction of maximizing the parameterized proxy function according to the update order:

[0054]

[0055] Among them, θ k represents the policy network parameters after the last update, θ k+1 Indicates the policy network parameters after the update.

[0056] 3. Distributed execution:

[0057] a. Strategy deployment: Update the policy network parameters θ k+1 Distributed to all heterogeneous devices, each heterogeneous device independently executes tasks according to the updated strategy.

[0058] b. Independent Execution and Data Collection: Heterogeneous devices independently execute tasks according to the updated policies in their operating environments and continue to collect new state, action, and reward data.

[0059] Step S3: Repeat the centralized training and distributed execution steps in the above step S2 until the policy network parameters converge and the optimal heterogeneous multi-agent policy network is obtained. The optimal heterogeneous multi-agent policy network is distributed to heterogeneous devices to achieve optimal control of heterogeneous devices.

Claims

1. A heterogeneous equipment control method based on heterogeneous multi-agent reinforcement learning algorithm, characterized in that: The steps include: The cooperative control problem of heterogeneous devices is modeled as a heterogeneous multi-agent reinforcement learning problem, and the state space, action space, transition probability function, reward function and objective function of the heterogeneous multi-agent reinforcement learning are defined; the heterogeneous multi-agent includes different types of agents that perform different actions; A centralized training distributed execution framework is adopted. First, the relevant parameters are initialized. During the centralized training process, the sampled trajectory data is stored in the experience replay pool. The update order of the heterogeneous multi-agent is calculated based on the dynamic characteristics of the environment, and the parameterized proxy function is calculated. The heterogeneous multi-agent strategy is sequentially updated in the direction of maximizing the parameterized proxy function, achieving monotonic improvement of the strategy network parameters of the heterogeneous multi-agent. During distributed execution, the updated heterogeneous multi-agent policy network is distributedly executed on heterogeneous devices to coordinate and control different types of heterogeneous devices; Repeat the sequential updating steps of the above heterogeneous multi-agent strategy until the policy network parameters converge, and finally obtain the converged optimal multi-agent policy network, which is executed to achieve optimal control of heterogeneous devices.

2. The heterogeneous device control method based on heterogeneous multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: Features of the heterogeneous device include: Each device has different physical characteristics, functions, and control objectives; Each device has different observation space and action space; Individual devices are capable of performing different types of control tasks, including temperature control, pressure regulation, and energy management.

3. The heterogeneous device control method based on heterogeneous multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The method of modeling the cooperative control problem of heterogeneous devices as a heterogeneous multi-agent reinforcement learning problem includes: The state space, action space, transition probability function, reward function and objective function of heterogeneous multi-agent reinforcement learning are defined according to the characteristics of heterogeneous devices; the state space includes the local observations of the heterogeneous devices; the action space is the joint action space of the heterogeneous devices; the transition probability function describes the probability of the agent transferring from one state to another; the reward function is designed based on global information and is used to evaluate the contribution of the joint action to the collaborative task of heterogeneous devices; the objective function represents the expected cumulative reward of the agent under the strategy.

4. The heterogeneous device control method based on heterogeneous multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The centralized training distributed execution framework includes: Initialize the policy network parameters and global value function network parameters of each heterogeneous multi-agent, configure the experience replay pool, and set the initial environment state and the set of each heterogeneous multi-agent. In the centralized training phase, collect the state, action, result and reward data of each heterogeneous multi-agent during the execution process and store them in the experience replay pool. Calculate the update order of the heterogeneous multi-agent based on the dynamic characteristics of the environment, and then calculate the parameterized proxy function. Update the policy network parameters of each heterogeneous multi-agent according to the parameterized proxy function sequence. In the distributed execution phase, distribute the updated policy network parameters to all heterogeneous devices, so that each heterogeneous device can independently execute tasks according to the updated strategy and collect new execution data.

5. The heterogeneous device control method based on heterogeneous multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The updating sequence of calculating heterogeneous multi-agents based on the dynamic characteristics of the environment includes: Based on the application scenario, obtain indicators that reflect the dynamic characteristics of the environment; According to the application scenario, the importance of heterogeneous multi-agents is ranked according to indicators to generate an update order for the heterogeneous multi-agents.

6. The heterogeneous device control method based on heterogeneous multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The calculating of the parameterized proxy function includes: calculating the parameterized proxy function of the i-th agent according to the information obtained by sampling in, Represents the current strategy of the i-th agent Last updated policy The probability ratio, a i represents the action of the ith agent, and s represents the current state; is the advantage function, which is calculated by the difference between the action value function and the state value function; clip(g t (θ),1±ε) is a clipping method to ensure that g t The value of (θ) does not exceed the range of [1-ε,1+ε] set by the parameter ε.

7. The heterogeneous device control method based on heterogeneous multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The sequential updating of the heterogeneous multi-agent strategy in the direction of maximizing the parameterized proxy function includes: updating the strategy network parameters of each heterogeneous multi-agent in the direction of maximizing the parameterized proxy function according to the calculated update sequence of the heterogeneous multi-agent: where θ k and θ k+1 Respectively represent the policy network parameters after the last update and the current update, π θ represents the policy with policy network parameters θ.

Images