Computer system based on multi-agent reinforcement learning network and network optimization method

Abstract

The invention provides a computer system based on a multi-agent reinforcement learning network and a network optimization method, and relates to the technical field of deep reinforcement learning. The computer system based on the multi-agent reinforcement learning network comprises a strategy network module used for receiving local state information and capability vectors corresponding to target agents, processing the local state information and the capability vectors and outputting probability values of action information corresponding to the target agents, the local state information comprises observation information of at least two other agents except the target agent, and the observation information is sequentially input to the strategy network module in a parallel stacking mode; and the value network module is used for receiving the global state information or the local state information corresponding to the target agent, and obtaining the long-term value information corresponding to the target agent according to the global state information or the local state information, so as to optimize the strategy network module and the value network module. According to the invention, the universality of a network algorithm can be improved.

Description

Technical Field

[0001] The present application relates to the field of deep reinforcement learning technology, and in particular to a computer system and network optimization method based on a multi-agent reinforcement learning network. Background Art

[0002] With the continuous development of deep learning technology, multi-agent reinforcement learning technology has made significant progress. In scenarios with multiple agents and types of agents, it is usually based on deep neural networks, through a large amount of trial and error learning and strategy iteration, so that agents can learn collaborative or competitive strategies in complex environments; in general, existing reinforcement learning technologies for multi-agents assume that the number of agents is fixed and the environment is static; in this case, the algorithm can optimize the agent's strategy through repeated training.

[0003] However, the inventors of the present application have discovered that, in actual applications, the number of agents is often uncertain and may change due to changes in task requirements, resource constraints, or environmental conditions; in addition, the types of agents may also be different; in this case, the use of existing multi-agent reinforcement learning technology will result in the network algorithm being unable to make reasonable predictions and decisions when faced with unknown or complex environments, thereby reducing the versatility of the network algorithm. Summary of the invention

[0004] In order to improve the versatility of the algorithm, the present application provides a computer system and a network optimization method based on a multi-agent reinforcement learning network.

[0005] The present application provides a computer system based on a multi-agent reinforcement learning network, which adopts the following technical solutions:

[0006] A computer system based on a multi-agent reinforcement learning network, comprising:

[0007] A policy network module is used to receive local state information and capability vector corresponding to the target intelligent agent, process the local state information and capability vector, and output the probability value of the action information corresponding to the target intelligent agent, wherein the local state information includes observation information of at least two other intelligent agents except the target intelligent agent, and is input into the policy network module in the form of parallel stacking;

[0008] The value network module is used to receive the global state information or local state information corresponding to the target intelligent agent, and obtain the long-term value information corresponding to the target intelligent agent based on the global state information or the local state information, so as to optimize the strategy network module and the value network module.

[0009] According to some embodiments, the above-mentioned strategy network module includes: an input-level network sub-module, which is used to receive local state information and capability vectors, and perform feature extraction and dimensionality reduction processing on the local state information and capability vectors to obtain feature representation; a feature processing network sub-module, which is connected to the input-level network sub-module, and is used to obtain all initial action information corresponding to the target intelligent agent based on the feature representation; an output-level network sub-module, which is connected to the feature processing network sub-module, and is used to perform action fitting and normalization processing on all initial action information, and output the probability values ​​corresponding to all the initial action information respectively.

[0010] According to some embodiments, the above-mentioned input-level network sub-module includes: an information combination network unit, which is used to receive the local state information and capability vector corresponding to the target intelligent agent, and combine the local state information and the capability vector to obtain a network input vector; a feature extraction network unit, which is connected to the information combination network unit and is used to extract vector features from the network input vector; a dimensionality reduction network unit, which is connected to the feature extraction network unit and is used to perform dimensionality reduction processing on the vector features to obtain a feature representation.

[0011] According to some embodiments, the above-mentioned output-level network sub-module includes: an action fitting network unit, which is used to fit all initial action information; a normalization network unit, which is connected to the action fitting network unit, and is used to normalize all the action information after fitting to obtain the probability value of all action information corresponding to the target intelligent agent.

[0012] According to some embodiments, the above-mentioned computer system based on a multi-agent reinforcement learning network also includes: an action interaction network module, which is connected to the output-level network sub-module, and is used to sample target action information from all initial action information based on the probability values ​​corresponding to all initial action information, and interact the target action information with the environment.

[0013] According to some embodiments, the above-mentioned feature processing network sub-module includes: a multi-head attention network unit, which is composed of multiple encoder layers stacked, connected to the input-level network sub-module, and is used to process the feature representation to obtain multiple first feature representations; a feedforward neural network unit, connected to the output-level network sub-module, and is used to receive multiple first feature representations, and perform nonlinear expression capability increase processing on the multiple first feature representations, and output all action features corresponding to the target intelligent agent.

[0014] According to some embodiments, the above-mentioned feature processing network submodule also includes: a residual and layer normalization network unit, which is connected to the multi-head attention network unit and the feedforward neural network unit, and is used to perform scale transformation and normalization processing on the first feature representation and all action features to obtain the second feature representation corresponding to the first feature representation and the initial action information corresponding to all action features respectively.

[0015] This application provides an optimization method for a multi-agent reinforcement learning network, which adopts the following technical solutions:

[0016] A multi-agent reinforcement learning network optimization method, used to optimize the multi-agent reinforcement learning network in the above-mentioned computer system;

[0017] Optimization methods for multi-agent reinforcement learning networks include:

[0018] Collect multiple sets of sample data, wherein each set of sample data in the multiple sets of sample data contains trajectory information corresponding to the intelligent agent; the multiple sets of sample data are sample data collected in different environments;

[0019] Based on the trajectory information, optimize the objective functions of the strategy network module and the value network module;

[0020] The objective function of the optimized strategy network module is maximized to obtain the first optimization parameter of the strategy network module, and the objective function of the optimized value network module is minimized to obtain the second optimization parameter of the value network module.

[0021] According to some embodiments, the above-mentioned collection of multiple groups of sample data includes: inputting the local state information corresponding to at least two target agents in the same environment into the strategy network module in sequence to obtain action information corresponding to at least two target agents, and interacting the action information with multiple environments to obtain multiple reward information associated with the multiple environments in parallel; collecting local state information, action information and reward information in parallel to obtain multiple groups of sample data corresponding to the multiple environments.

[0022] According to some embodiments, the optimization objective function of the above-mentioned policy network is as follows:

[0023]

[0024] in The parameters ∈ and ∈1 represent hyperparameters between 0 and 1. The hyperparameter β is used to control the size of the latter objective function. clip(x, l, u) represents a truncation function, defined as clip(x, l, u) = min(max(x; l); u), where max(x; l) is the maximum value between x and l, min(x; u) is the minimum value between x and u, N represents the amount of data of the own agent in the same environment, a t Indicates action; π i (a t |o t ) indicates that the current strategy is in state o t The distribution of the following actions, Indicates the state o under the historical strategyt The distribution of the moving is the distribution ratio of the actions of the ith agent at time t, E represents the mathematical expectation, A i It represents the advantage function of the i-th agent under state and action, and the calculation formula is

[0025] A μ (s t , a t )=δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ,

[0026] Where: t =r t +γV(s t+1 )-V(s t ), γ is a discount factor, λ is a given parameter, V(s t ) is in state s t The value function under s t Indicates the current status.

[0027] According to some embodiments, the optimization objective function of the above value network is as follows:

[0028]

[0029] in, R(s i , a i ) means in state s i Execute action a i The reward obtained, γ is a discount factor ranging from 0 to 1, N represents the amount of data of the own agent among at least two agents, V Φ (s i ) means in state s i The value function under , E represents the mathematical expectation.

[0030] This application provides an optimization system for a multi-agent reinforcement learning network, which adopts the following technical solutions:

[0031] A multi-agent reinforcement learning network optimization system, used for training and optimizing the multi-agent reinforcement learning network in the above-mentioned computer system;

[0032] The optimization system of the multi-agent reinforcement learning network includes: a sample data acquisition unit, an objective function optimization unit, an optimization parameter determination unit and a network optimization unit, wherein:

[0033] A sample data collection unit is used to collect multiple sets of sample data, wherein each set of sample data in the multiple sets of sample data contains trajectory information corresponding to the intelligent agent; the multiple sets of sample data are sample data collected in different environments;

[0034] An objective function optimization unit is used to optimize the objective functions of the strategy network and the value network based on trajectory information;

[0035] The optimization parameter determination unit is used to maximize the objective function of the optimized strategy network module to obtain the first optimization parameter of the strategy network module, and minimize the objective function of the optimized value network module to obtain the second optimization parameter of the value network module.

[0036] According to some embodiments, the above-mentioned sample data collection unit is specifically used to: input the local state information corresponding to at least two target intelligent agents in the same environment into the strategy network module in sequence to obtain the action information corresponding to the at least two target intelligent agents, and interact the action information with multiple environments to obtain multiple reward information associated with one or more environments in parallel; collect local state information, action information and reward information in parallel to obtain multiple groups of sample data corresponding to multiple environments.

[0037] According to some embodiments, the optimization objective function of the above-mentioned policy network module is as follows:

[0038]

[0039] in The parameters ∈ and ∈1 represent hyperparameters between 0 and 1. The hyperparameter β is used to control the size of the latter objective function. clip(x, l, u) represents a truncation function, defined as clip(x, l, u) = min(max(x; l); u), where max(x; l) is the maximum value between x and l, min(x; u) is the minimum value between x and u, N represents the amount of data of one's own agent among at least two agents, and a t Indicates action; π i (a t |o t ) indicates that the current strategy is in state o t The distribution of the following actions, Indicates the state o under the historical strategy t The distribution of the moving is the distribution ratio of the actions of the ith agent at time t, E represents the mathematical expectation, A i It represents the advantage function of the i-th agent under state and action, and the calculation formula is

[0040] A μ (s t, a t )=δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ,

[0041] Where: t =r t +γV(s t+1 )-V(s t ), γ is a discount factor, λ is a given parameter, V(s t ) is in state s t The value function under s t Indicates the current status.

[0042] According to some embodiments, the optimization objective function of the above-mentioned value network module is as follows:

[0043]

[0044] in, R(s i , a i ) means in state s i Execute action a i The reward obtained, γ is a discount factor ranging from 0 to 1, N represents the amount of data of the own agent among at least two agents, V Φ (s i ) means in state s i The value function under , E represents the mathematical expectation.

[0045] The present application provides an electronic device, which adopts the following technical solution:

[0046] An electronic device, comprising:

[0047] processor;

[0048] A memory stores a computer program, and when the computer program is executed by a processor, the processor executes the optimization method of the multi-agent reinforcement learning network.

[0049] The present application provides a computer-readable storage medium, which adopts the following technical solution:

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the above-mentioned optimization method for the multi-agent reinforcement learning network.

[0051] According to the above-mentioned embodiments provided in the present application, a multi-agent reinforcement learning network including a strategy network module and a value network module is constructed, and at its input end, local state information of a target agent including state information of at least two other agents except the target agent can be received in parallel, and action information corresponding to the target agent and the probability value of the action information can be obtained. The target action is sampled based on the probability value and interacts with the environment to realize the transfer and utilization of strategy knowledge in different environments, thereby improving the reusability of the network algorithm. In addition, a certain amount of sample data is generated by the interaction between the multi-agent reinforcement learning network and the environment, and the strategy network module and the value network module are trained using these sample data, so that the agent can make decisions based on the relationship and ability between the agents, and can interact with multiple different environments in parallel at the same time, thereby improving the versatility of the network algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the network architecture of the policy network module of the embodiment of the present application:

[0053] Figure 2 : is a schematic diagram of a block diagram of an optimization method for a multi-agent reinforcement learning network according to an embodiment of the present application:

[0054] Figure 3 : is a block diagram of an optimization system for a multi-agent reinforcement learning network according to an embodiment of the present application:

[0055] Figure 4 It is a schematic diagram of an electronic device according to an embodiment of the present application.

[0056] Description of reference numerals:

[0057] 30: Optimization system of multi-agent reinforcement learning network; 301: Sample data acquisition unit; 302: Objective function optimization unit; 303: Optimization parameter determination unit; 40: Electronic device; 401: Processor; 402: Bus; 403: Memory; 404: Transceiver. DETAILED DESCRIPTION

[0058] The following is combined with Figure 1-Figure 4 This application is described in further detail.

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the drawings in the embodiments of the present application are collected below to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The embodiment of the present application provides a computer system based on a multi-agent reinforcement learning network;

[0061] A computer system based on a multi-intelligence reinforcement learning network model includes: a strategy network module and a value network module, wherein the strategy network module is used to receive local state information and capability vectors corresponding to two target agents, process the local state information and capability vectors, and output the probability values ​​of the action information corresponding to the target agents. The value network module is used to receive global state information or local state information corresponding to the target agents, and obtain the long-term value information corresponding to the target agents according to the global state information or the local state information; so as to optimize the strategy network module and the value network module.

[0062] In some embodiments, the local state information corresponding to the target agent includes observation information of at least two other agents except the target agent, which is input into the policy network module in the form of parallel stacking. The long-term value system information is used to guide the optimization of the multi-agent reinforcement learning network. The agent can be an adversarial operation control subject, such as an adversarial robot, etc.; the adversarial robot is used as an example for explanation below; the movement, defensive action, and machine attack action of the adversarial robot reflect the position state and control state of the adversarial robot as the state information of the adversarial robot.

[0063] In some embodiments, it is assumed that there are N agents on one side and M agents on the other side; taking any agent on the one side as the benchmark, i.e., the target agent, the observation information of all agents seen by any agent is constructed into a local state information, and N+M-1 local state information is obtained; then, the N+M-1 local state information is input into the policy network module in the form of parallel stacking. At the same time, different ability vectors are constructed for the different abilities of each agent, i.e., N+M ability vectors are obtained, and then, the N+M ability vectors are input into the policy network module in the form of parallel stacking. After being processed by the policy network module, the action information corresponding to each agent and the probability value corresponding to each action information are obtained.

[0064] In some embodiments, N+M-1 local state information or the global state information of the agent is input into the value network module for processing, and the long-term value information of each agent is output, wherein the long-term value information is used to evaluate the goodness (reward) of the state of the agent; the long-term value information is the weighted average of the rewards.

[0065] In some embodiments, the strategy network module and the value network module can be a convolutional neural network or a multilayer perceptron neural network, wherein the number of neurons in each layer, the number of layers and the activation function of each layer are set according to the specific problem, and the embodiments of the present application do not make specific limitations.

[0066] Reference Figure 1 The above-mentioned strategy network module includes: an input-level network submodule, which is used to receive local state information and capability vectors, and perform feature extraction and dimensionality reduction processing on the local state information and capability vectors to obtain feature representation; a feature processing network submodule, which is connected to the input-level network submodule, and is used to obtain all initial action information corresponding to the target intelligent agent based on the feature representation; an output-level network submodule, which is connected to the feature processing network submodule, and is used to receive action fitting and normalization processing of all initial action information, and output the probability values ​​corresponding to all the initial action information.

[0067] The above-mentioned input-level network submodule includes: an information combination network unit, which is used to receive the local state information and capability vector corresponding to the target intelligent agent, and combine the local state information and capability vector to obtain a network input vector; a feature extraction network unit, which is connected to the information combination network unit and is used to extract vector features from the network input vector; a dimensionality reduction network unit, which is connected to the feature extraction network unit and is used to perform dimensionality reduction processing on the vector features to obtain a feature representation.

[0068] In some embodiments, Figure 1 V in the representation represents the capability vector set, Represent the local state information of the intelligent agent; N+M-1 local state information and N+M capability vector capabilities are received by the information combination network unit included in the input-level network submodule, and the N+M-1 local state information and N+M capability vector capabilities are combined by the information combination network unit, and the network input vector is output; then, the network input vector is input to the feature extraction network unit in the input-level network submodule, and the vector features (information such as the state and action of the intelligent agent) are extracted from the network input vector by the feature extraction network unit; then, the vector features are input to the dimensionality reduction network unit in the input-level network submodule, and the vector features are mapped to a low-dimensional space by dimensionality reduction technology, and the vector features are converted into multiple feature representations, such as Figure 1 In

[0069] Reference Figure 1 The above-mentioned feature processing network submodule includes: a multi-head attention network unit, which is composed of multiple encoder layers stacked, connected to the input-level network submodule, and is used to process the feature representation to obtain multiple first feature representations; a feedforward neural network unit, which is connected to the output-level network submodule, and is used to receive multiple first feature representations and perform nonlinear expression ability increase processing on the multiple first feature representations to obtain all action features corresponding to the target intelligent agent respectively.

[0070] The above-mentioned feature processing network submodule also includes: a residual and layer normalization network unit, which is connected to the multi-head attention network unit and the feedforward neural network unit, and is used to scale and normalize the first feature representation and all action features to obtain the second feature representation corresponding to the first feature representation and the initial action information corresponding to all action features.

[0071] In some embodiments, the multi-head attention network unit is composed of multiple identical encoder layers stacked together, allowing the model to process each input position while paying attention to other positions in the sequence, thereby better capturing contextual information.

[0072] In some embodiments, the multiple feature representations obtained by conversion are input into a multi-head attention network unit in a feature processing network submodule, and multiple first feature representations are output through processing by the multi-head attention network unit. Subsequently, the multiple first feature representations are input into a residual and layer normalization network unit connected to the multi-head attention network unit, and the multiple first feature representations are processed to output second feature representations; thereby reducing the occurrence of gradient vanishing and accelerating training, and improving the stability of the policy network module during the training process.

[0073] Subsequently, the second feature representation is input into the feedforward neural network unit in the feature processing network submodule, the input features of each position are processed by the feedforward neural network unit, and the expression ability of the model is enhanced by nonlinear transformation, so as to output all actions corresponding to the target intelligent agent, that is, multiple action features; then, all action features are input into the residual and layer normalization network unit connected to the output end of the feedforward neural unit, and multiple action features are processed to reduce the occurrence of gradient vanishing and accelerate training, so as to output all initial action information corresponding to each intelligent agent.

[0074] In some embodiments, the residual and layer normalization network units and the feedforward neural network units may be convolutional neural networks or fully connected neural networks, which are not specifically limited in the embodiments of the present application.

[0075] Reference Figure 1 The above-mentioned output-level neural network submodule includes: an action fitting network unit, which is used to fit all initial action information; a normalization network unit, which is connected to the action fitting network unit, and is used to normalize all the initial action information after fitting to obtain the probability value of all action information corresponding to the target intelligent agent.

[0076] In some embodiments, the action fitting network unit is connected to the residual and layer normalization network unit connected to the output end of the feedforward neural network unit, and is used to receive multiple initial action information output by the residual and layer normalization network unit; then, the multiple initial action information is fitted by the action fitting network unit, and the action information corresponding to each intelligent agent is output, wherein the action of the intelligent agent itself is fitted with its own movement action using the action fitting network, such as Figure 1 In, by The agent's own actions are fitted action,...., Action; Use the action fitting network to fit each of your own agent's support actions, such as Figure 1 In, by Motion fitting Action, by Fitting Use the action fitting network to fit the opponent's attack action, such as Figure 1 In, by Motion fitting Action, by Fitting Then, the obtained multiple fitting actions are input into the normalization network connected with the action fitting network, such as Figure 1 In the softmax network unit in , the normalized exponential function is used to output the probability value corresponding to each action information.

[0077] In some embodiments, the computer system based on the multi-agent reinforcement learning network also includes: an action interaction network module, which is connected to the output level network sub-module, and is used to sample target action information from all initial action information based on the probability values ​​corresponding to all initial action information, and interact the target action information with the environment.

[0078] In some embodiments, the probability values ​​corresponding to multiple initial action information output by the normalized network unit in the output-level network sub-module are input into the action interaction network module, and by analyzing the probability value of each action information, the target action information is sampled from the multiple initial action information, and the target action information is interacted with the environment.

[0079] The embodiment of the present application provides an optimization method for a multi-agent reinforcement learning network, which can be executed by an electronic device, wherein the electronic device can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed device composed of multiple physical servers, or a cloud server providing cloud computing services; the server can be an independent device, or it can be installed in a SCADA system (Supervisory Control And Data Acquisition, industrial automation and real-time monitoring system), or it can be installed in an intelligent body. The terminal device can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present invention does not limit this.

[0080] A multi-agent reinforcement learning network optimization method is used to optimize the multi-agent reinforcement learning network in the above-mentioned computer system.

[0081] Reference Figure 2 , the optimization method of the multi-agent reinforcement learning network includes: step S201, step S202 and step S203, wherein,

[0082] S201, collecting multiple groups of sample data.

[0083] In some embodiments, each set of sample data in the multiple sets of sample data contains trajectory information corresponding to the target intelligent agent; the multiple sets of sample data are sample data collected in different environments.

[0084] In some embodiments, the environment is initialized; the policy network module parameters and the value network module parameters are initialized; the local state information of the target agents corresponding to the multiple environments are input into the policy network module, and the local state information is trained through the strategy in the policy network module to obtain the action information of the target agents corresponding to the multiple environments; then, the action information interacts with the environment and is transferred to the next moment, and feedback results on whether the action meets the requirements, i.e., reward information, are obtained; the electronic device will collect the information of the target agents in the same environment in parallel, and collect the local state information, action information, reward information and other information representing the trajectory information corresponding to each agent according to the criteria for collecting the information of the target agents in the same environment, and obtain multiple groups of sample data, wherein there is a one-to-one correspondence between the multiple environments and the multiple groups of sample data.

[0085] At the same time, the electronic device inputs the local state information and the trajectory information corresponding to the target intelligent agent in a single set of sample data into the value network module, and outputs the long-term value information corresponding to the target intelligent agent, wherein the long-term value information is the weighted average of the rewards; the long-term value information is used to evaluate the quality of the local state information of the intelligent agent and guide the training of the strategy network module.

[0086] S202, based on the trajectory information, optimizing the objective functions of the strategy network module and the value network module.

[0087] In some embodiments, when training the policy network module and the value network module included in the multi-agent reinforcement learning network, the reinforcement learning problem can be represented by a quintuple (S, O, A, P, R), where S represents the state space of the agent in the environment, O represents the local state space of the agent, A represents the action space of the agent performing actions, P represents the transition probability function of the agent interacting with the environment in state s and transferring to the next state s' after performing an action a, and R represents the total reward function obtained after the multi-agent interacts with the environment and performs action a.

[0088] In some embodiments, in each iteration, the optimization method of the multi-agent reinforcement learning network is divided into sample data collection and network update; in the network update process, it is proposed to optimize the objectives of the policy network module and the objectives of the value network module. Among them, in the optimization of the objectives of the policy network module, the optimization strategy in the objective function will use subsequent information to directly optimize the strategy in the current state. The quality of the subsequent trajectory information is directly reflected in the current state, which promotes the agent to reflect on the current behavior and realize efficient use of the trajectory information.

[0089] In some embodiments, the following describes the optimization process of implementing the policy network module, and proposes the following optimization objective function, as shown in the following formula:

[0090]

[0091] in The parameters ∈ and ∈1 represent hyperparameters between 0 and 1. The hyperparameter β is used to control the size of the latter objective function. clip(x, l, u) represents a truncation function, defined as clip(x, l, u) = min(max(x; l); u), where max(x; l) is the maximum value between x and l, min(x; u) is the minimum value between x and u, N represents the amount of data of the own agent in the same environment, a t Indicates action; π i (a t |o t ) indicates that the current strategy is in state o t The distribution of the following actions, Indicates the state o under the historical strategy t The distribution of the moving is the distribution ratio of the actions of the ith agent at time t, E represents the mathematical expectation, A i It represents the advantage function of the i-th agent under state and action, and the calculation formula is

[0092] A μ (s t , a t )=δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ,

[0093] Where: t =r t +γV(s t+1 )-V(s t ), γ is a discount factor, λ is a given parameter, V(s t ) is in state s t The value function under s t Indicates the current status.

[0094] In some embodiments, the following receiving and implementing the optimization process of the value network module proposes an optimization objective function, as shown in the following formula:

[0095]

[0096] in, R(s i , a i ) means in state s i Execute action a i The reward obtained, γ is a discount factor ranging from 0 to 1, N represents the amount of data of the own agent among at least two agents, V Φ (s i ) means in state s i The value function under , E represents the mathematical expectation.

[0097] S203, maximizing the objective function of the optimized strategy network module to obtain a first optimization parameter of the strategy network module, and minimizing the objective function of the optimized value network module to obtain a second optimization parameter of the value network module.

[0098] In some embodiments, the electronic device uses the stochastic gradient ascent method to maximize the objective function of the optimized policy network module, thereby obtaining the first optimization parameter of the updated policy network module; and simultaneously uses the random gradient descent method to minimize the optimized value network module, thereby obtaining the second optimization parameter of the updated value network module. Subsequently, the policy of the policy network module is updated based on the first optimization parameter, and the policy of the value network module is updated based on the second optimization parameter.

[0099] The following is a programming representation of the optimization method for a multi-agent reinforcement learning network:

[0100]

[0101] In step S102, multiple groups of sample data are collected, including: local state information corresponding to at least two target agents in the same environment are input into the strategy network module in sequence to obtain action information corresponding to at least two target agents, and the action information is interacted with multiple environments to obtain multiple reward information associated with the multiple environments in parallel; local state information, action information and reward information are collected in parallel to obtain multiple groups of sample data corresponding to one or more environments.

[0102] In some embodiments, single sets of sample data corresponding to multiple environments are collected in parallel. The electronic device can collect at least two trajectory information in a single environment, that is, input the local state information corresponding to at least two target agents in the same environment into the policy network module, output the action information corresponding to each agent through the policy network module, interact the action information with the environment, jump to the next moment, and obtain the reward information corresponding to the action information; the electronic device collects the information representing the trajectory information such as the local state information, action information and reward information corresponding to at least two target agents respectively, and obtains a single set of sample information; the electronic device can also collect the trajectory information of at least two target agents corresponding to different environments in parallel to obtain multiple sets of sample data.

[0103] In some embodiments, in order to better optimize the policy network module and the value network module, the same batch of sample data can be used for optimization multiple times to achieve more efficient optimization of network parameters; and it can interact with multiple different environments in parallel to learn a strategy that adapts to these environments, thereby improving the versatility of the algorithm, and the learned strategy can perform knowledge transfer in similar environments, thereby improving the utilization efficiency of the collected data.

[0104] This application provides an optimization system for a multi-agent reinforcement learning network, which adopts the following technical solutions:

[0105] Reference Figure 3, a multi-agent reinforcement learning network optimization system 30, used for training and optimizing the multi-agent reinforcement learning network in the above-mentioned computer system;

[0106] The optimization system 30 of the multi-agent reinforcement learning network includes: a sample data collection unit 301, an objective function optimization unit 302, an optimization parameter determination unit 303 and a network optimization unit 304, wherein:

[0107] The sample data collection unit 301 is used to collect multiple sets of sample data, wherein each set of sample data in the multiple sets of sample data contains trajectory information corresponding to the intelligent agent; the multiple sets of sample data are sample data collected in different environments;

[0108] An objective function optimization unit 302, used to optimize the objective functions of the strategy network module and the value network module based on the trajectory information;

[0109] The optimization parameter determination unit 303 is used to maximize the objective function of the optimized strategy network module to obtain the first optimization parameter of the strategy network module, and to minimize the objective function of the optimized value network module to obtain the second optimization parameter of the value network module.

[0110] According to some embodiments, the above-mentioned sample data collection unit 301 is specifically used to: input the local state information corresponding to at least two target agents in the same environment into the strategy network module in sequence to obtain the action information corresponding to the at least two target agents, and interact the action information with multiple environments to obtain multiple reward information associated with the multiple environments in parallel; collect local state information, action information and reward information in parallel to obtain multiple groups of sample data corresponding to one or more environments.

[0111] According to some embodiments, the optimization objective function of the above-mentioned policy network module is as follows:

[0112]

[0113] in The parameters ∈ and ∈1 represent hyperparameters between 0 and 1. The hyperparameter β is used to control the size of the latter objective function. clip(x, l, u) represents a truncation function, defined as clip(x, l, u) = min(max(x; l); u), where max(x; l) is the maximum value between x and l, min(x; u) is the minimum value between x and u, N represents the amount of data of one's own agent among at least two agents, and a t Indicates action; π i (a t |o t ) indicates that the current strategy is in state ot The distribution of the following actions, Indicates the state o under the historical strategy t The distribution of the moving is the distribution ratio of the actions of the ith agent at time t, E represents the mathematical expectation, A i It represents the advantage function of the ith agent under state and action, and the calculation formula is

[0114] A μ (s t , a t )=δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ,

[0115] Where: t =r t +γV(s t+1 )-V(s t ), γ is a discount factor, λ is a given parameter, V(s t ) is in state s t The value function under s t Indicates the current status.

[0116] According to some embodiments, the optimization objective function of the above-mentioned value network module is as follows:

[0117]

[0118] in, R(s i , a i ) means in state s i Execute action a i The reward obtained, γ is a discount factor ranging from 0 to 1, N represents the amount of data of the own agent among at least two agents, V Φ (s i ) means in state s i The value function under , E represents the mathematical expectation.

[0119] In some embodiments, the sample data acquisition unit 301 may include a logic circuit, or may be implemented by a central processing unit, a digital signal processor, or a field programmable gate array contained in an electronic device; the objective function optimization unit 302 may include a logic circuit, or may be implemented by a central processing unit, a digital signal processor, or a field programmable gate array contained in an electronic device; the optimization parameter determination unit 303 may include a logic circuit, or may be implemented by a central processing unit, a digital signal processor, or a field programmable gate array contained in an electronic device.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] An embodiment of the present application discloses an electronic device, comprising: a processor; and a memory storing a computer program. When the computer program is executed by the processor, the processor executes the above-mentioned optimization method of the multi-agent reinforcement learning network.

[0122] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor executes an optimization method for a multi-agent reinforcement learning network.

[0123] An embodiment of the present application discloses an electronic device, including: a processor; a memory storing a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned data encryption transmission method based on the two-dimensional code.

[0124] For example, refer to Figure 4 , Figure 4 The electronic device 40 shown includes: a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the electronic device 40 may also include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the electronic device 40 does not constitute a limitation on the embodiments of the present invention.

[0125] Processor 401 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in the disclosure of the present invention. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0126] The bus 402 may include a path to transmit information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0127] The memory 403 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0128] The memory 403 is used to store application code for executing the solution of the present invention, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the above method embodiment.

[0129] Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0130] An embodiment of the present application discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the processor executes a data encryption transmission method based on a two-dimensional code.

[0131] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0132] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A computer system based on a multi-agent reinforcement learning network, characterized in that: include: A policy network module, configured to receive local state information and a capability vector corresponding to a target agent, process the local state information and the capability vector, and output a probability value of action information corresponding to the target agent, wherein the local state information includes observation information of at least two other agents except the target agent, and is input to the policy network module in a parallel stacked form; The value network module is used to receive the global state information or the local state information corresponding to the target intelligent agent, and obtain the long-term value information corresponding to the target intelligent agent based on the global state information or the local state information, so as to optimize the strategy network module and the value network module.

2. The system according to claim 1, characterized in that The policy network module includes: An input-level network submodule, used to receive the local state information and the capability vector, and perform feature extraction and dimensionality reduction processing on the local state information and the capability vector to obtain a feature representation; A feature processing network submodule, connected to the input-level network submodule, for obtaining all initial action information corresponding to the target intelligent agent based on the feature representation; The output-level network submodule is connected to the feature processing network submodule, and is used to perform action fitting and normalization processing on all the initial action information, and output probability values ​​corresponding to all the initial action information.

3. The system according to claim 2, characterized in that The input-level network submodule comprises: An information combination network unit, configured to receive the local state information and the capability vector corresponding to the target agent, and combine the local state information and the capability vector to obtain a network input vector; A feature extraction network unit, connected to the information combination network unit, for extracting vector features from the network input vector; The dimension reduction network unit is connected to the feature extraction network unit and is used to perform dimension reduction processing on the vector feature to obtain the feature representation.

4. The system according to claim 2, characterized in that The output stage network submodule comprises: An action fitting network unit, used for fitting all the initial action information; The normalization network unit is connected to the action fitting network unit and is used to normalize all the initial action information after fitting to obtain the probability value of all the action information corresponding to the target intelligent agent.

5. The system according to claim 2, characterized in that Also includes: The action interaction network module is connected to the output-level network submodule and is used to sample target action information from all the initial action information based on the probability values ​​respectively corresponding to all the initial action information, and interact the target action information with the environment.

6. The system according to claim 2, characterized in that The feature processing network submodule includes: A multi-head attention network unit, formed by stacking multiple encoder layers, connected to the input-level network submodule, and used to process the feature representation to obtain multiple first feature representations; A feedforward neural network unit is connected to the output-level network submodule, and is used to receive the multiple first feature representations and perform nonlinear expression capability increase processing on the multiple first feature representations to obtain all action features corresponding to the target intelligent agents respectively.

7. The system according to claim 6, characterized in that The feature processing network submodule also includes: The residual and layer normalization network unit is connected to the multi-head attention network unit and the feedforward neural network unit, and is used to scale and normalize the first feature representation and all the action features to obtain the second feature representation corresponding to the first feature representation and the initial action information corresponding to all the action features.

8. A method for optimizing a multi-agent reinforcement learning network, characterized in that: Used to optimize a multi-agent reinforcement learning network in a computer system according to any one of claims 1 to 7; The method comprises: Collecting multiple sets of sample data, wherein each set of sample data in the multiple sets of sample data contains trajectory information corresponding to the intelligent agent; the multiple sets of sample data are sample data collected in different environments; Based on the trajectory information, optimizing the objective functions of the strategy network module and the value network module respectively; The objective function of the optimized strategy network module is maximized to obtain the first optimization parameter of the strategy network module, and the objective function of the optimized value network module is minimized to obtain the second optimization parameter of the value network module.

9. The method according to claim 8, characterized in that The collecting of multiple groups of sample data includes: Inputting local state information corresponding to at least two target agents in the same environment into the policy network module in sequence to obtain action information corresponding to the at least two target agents, and interacting the action information with multiple environments to obtain multiple reward information associated with one or more environments in parallel; The local state information, the action information, and the reward information are collected in parallel to obtain multiple groups of sample data corresponding to the multiple environments respectively.

10. The method according to claim 8, characterized in that The objective function of the policy network module is as follows: in The parameters ∈ and ∈1 represent hyperparameters between 0 and 1. The hyperparameter β is used to control the size of the latter objective function. clip(x, l, u) represents a truncation function, defined as clip(x, l, u) = min(max(x; l); u), where max(x; l) is the maximum value between x and l, min(x; u) is the minimum value between x and u, N represents the amount of data of the own agent in the same environment, a t Indicates action; π i (a t |o t ) indicates that the current strategy is in state o t The distribution of the following actions, Indicates the state o under the historical strategy t The distribution of the moving is the distribution ratio of the actions of the ith agent at time t, E represents the mathematical expectation, It represents the advantage function of the ith agent under state and action, and the calculation formula is A μ (s t ,a t )=d t +(gl)d t+1 +…+(cl) T-t+1 d T-1 , Where: t =r t +γV(s t+1 )-V(s t ), γ is a discount factor, λ is a given parameter, V(s t ) is in state s t The value function under s t Indicates the current status.

11. The method according to claim 9, characterized in that The objective function of the value network module is as follows: in, R(s i , a i ) means in state s i Execute action a i The reward obtained, γ is a discount factor ranging from 0 to 1, N represents the amount of data of the own agent among at least two agents, V Φ (s i ) means in state s i The value function under , E represents the mathematical expectation.

12. An optimization system for a multi-agent reinforcement learning network, characterized in that: Used to optimize a multi-agent reinforcement learning network in a computer system according to any one of claims 1 to 7; The system comprises: A sample data collection unit, used to collect multiple sets of sample data, wherein each set of sample data in the multiple sets of sample data contains trajectory information corresponding to the intelligent agent; the multiple sets of sample data are sample data collected in different environments; An objective function optimization unit, used for optimizing the objective functions of the strategy network module and the value network module based on the trajectory information; The optimization parameter determination unit is used to maximize the objective function of the optimized strategy network module to obtain the first optimization parameter of the strategy network module, and minimize the objective function of the optimized value network module to obtain the second optimization parameter of the value network module.

13. An electronic device, characterized in that: include: processor; A memory storing a computer program, which, when executed by the processor, enables the processor to execute the method according to any one of claims 8 to 11.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 8 to 11.

Images