A knowledge-driven biological intelligent swarm collaborative decision-making method

Abstract

The application belongs to the technical field of reinforcement learning, and discloses a knowledge-driven biological intelligent group collaborative decision-making method. In order to overcome the problems that the existing multi-agent reinforcement learning method cannot utilize high-order logical relationship, ignores the importance of different agents, and cannot capture the time dependence and dynamic graph structure of the multi-agent system, the application proposes a knowledge-driven biological intelligent group collaborative decision-making method, which combines high-order model representation, attention mechanism, long short-term memory network and other technologies, finally realizes sufficient learning and representation of the multi-agent dynamic system, and is used in the subsequent reinforcement learning process.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning technology and relates to a knowledge-driven collaborative decision-making method for biological intelligent groups. Background Technology

[0002] From bacteria and insects to social animals and humans, collaborative behavior among intelligent biological groups is ubiquitous in nature. For example, human societies face numerous collaborative problems such as traffic control, resource management, and autonomous driving. These problems can be formulated as multi-agent cooperative systems aiming to maximize agent incentives. One research direction for this problem is multi-agent reinforcement learning. However, multi-agent reinforcement learning methods struggle to learn complex policies in high-dimensional state and action spaces. Furthermore, in multi-agent environments, interactions between agents are highly complex, and each agent struggles to effectively process the massive amounts of information from other agents, severely impacting collaborative behavior. Moreover, the relationships between agents change over time, and existing methods cannot capture spatial structure and temporal dependencies to learn stable policies.

[0003] To address these issues, numerous multi-agent reinforcement learning algorithms have been proposed, but these methods all have limitations. Lowe et al.'s 2017 work at NeurIPS, "Multi-Agent Actor-Critic for MixedCooperative-Competitive Environments," employed a multi-agent deep deterministic policy gradient method, using a framework of centralized training and distributed execution to enhance the cooperative behavior of agents in mixed cooperative-competitive environments. Based on the same objective, Foerster et al.'s 2018 work at AAAI, "Counterfactual multi agent policy gradients," proposed a counterfactual agent policy gradient method, employing a centralized criticism approach and designing counterfactual benchmarks to marginalize the influence of each agent. While these methods can improve the cooperative ability of agents, they are only applicable to environments with a fixed number of agents. That is, each agent's network is trained in an environment with a fixed total number of agents, and retraining is still required when the number of agents changes. Furthermore, since these methods require the observations of all agents to construct the criticism network, the training difficulty increases significantly when the number of agents is large. In their 2018 paper "Mean Field Multi-Agent Reinforcement Learning" published at ICML, Yang et al. proposed the mean field method, which captures the interactions between agents by averaging the effects of neighboring agents, thus making it applicable to environments with a large number of agents. However, this method treats the observations of all agents equally, neglecting the relative importance of different agents. Furthermore, current multi-agent reinforcement learning algorithms still lack the ability to represent higher-order information relationships between agents, failing to leverage this higher-order information to improve the performance of the learning process. Summary of the Invention

[0004] To overcome the problems of existing multi-agent reinforcement learning methods, such as the inability to utilize higher-order logical relationships, the neglect of the importance of different agents, and the inability to capture the temporal dependence and dynamic graph structure of multi-agent systems, this invention proposes a knowledge-driven biological intelligent group collaborative decision-making method. This method combines higher-order motif representation, attention mechanism, long short-term memory network and other technologies to achieve full learning and representation of multi-agent dynamic systems, which can then be used in subsequent reinforcement learning processes.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A knowledge-driven collaborative decision-making method for biological intelligent groups first models a multi-agent system, constructing a graph of the spatial relationships between agents and their respective observations. Second, it extracts high-order logical relationships between agents through higher-order motif representations, using these as higher-order attributes of the agents. Next, it calculates the attention weight coefficients of other neighboring agents using an attention mechanism, and stabilizes the learning process through a multi-head attention mechanism as an aggregator, obtaining embeddings of two types of attributes. Then, to further represent entities, it merges the higher-order attribute embeddings and observation embeddings. Next, it uses a long short-term memory network to capture the system's temporal dependencies and complex dynamic structure to obtain more effective state observations. Finally, it inputs these observed states into a policy optimization network to predict the state and the probability distribution of all possible actions.

[0007] The specific steps are as follows:

[0008] Step 1: Model the multi-agent system, defining each agent as a node in a dynamic graph; specifically, define the observation vector of agent i at time t. As its node attribute And define the N×N adjacency matrix A at time t. t In this case, the ij elements of the adjacency matrix are a ij When agent i communicates with agent j, a ij =1, otherwise a ij =0, where N is the number of nodes;

[0009] Step 2: For each agent, define its entity motif degree at time t. In the formula E p This indicates that agent i is simultaneously included in multiple different p-th motifs; further, the dynamic graph entity motif degree matrix is ​​defined as follows: In this method, the dynamic graph entity modality matrix is ​​used as a higher-order attribute matrix for training;

[0010] Step 3: Calculate the attention weight coefficient e of node j with respect to node i. ij =a(Wh i ,Wh j and the standardized form of attention weight coefficients Where 'a' represents a single-layer feedforward neural network, W is a linear weight matrix, and LeakyReLU is a non-linear activation function with a negative input slope. The set of neighboring nodes of node i; the aggregate representation of node i at time t. Here, σ is a nonlinear function; then, a multi-head attention mechanism is used as an aggregator to stabilize the attention learning process, extracting different state representations of the agent from different state word spaces; at this time, the representation of node i at time t is updated as follows: Where m represents the number of multi-head attention; similarly, each row of the high-order attribute matrix M generated in step two... The representation at time t is as follows

[0011] Step 4: Represent the nodes obtained in Step 3 And row representation The original observation embedding matrix H and the higher-order attribute embedding matrix H are respectively formed. s Aggregation is performed to better represent the relationships between entities and the overall system, resulting in the feature aggregation matrix H. agg =H+H s ;

[0012] Step 5: Use LSTM to characterize the temporal dependencies and dynamic topology of the multi-agent system; this part consists of three gates: the forget gate f t Input gate i t and output gate o t The detailed calculation process is as follows:

[0013]

[0014]

[0015]

[0016] Where σ(·) represents the sigmoid function, and for the input value x, the sigmoid function outputs... W f W i W o These represent the weight matrices for the forget gate, input gate, and output gate, respectively. f b i b o These represent the bias vectors for the forget gate, input gate, and output gate, respectively. h represents the input at time t. t-1 This represents the output of the LSTM hidden layer at time t-1;

[0017] The current hidden layer is defined as W C Represents the weight matrix, b C Represents the bias vector. The output value is within the range [-1, 1]; therefore, the current memory cell consists of two parts: first, the information to be retained from the previous memory cell is filtered through the forget gate; second, the current hidden layer information is updated through the input gate. The current memory cell is obtained by adding the information retained from the previous memory cell to the current hidden layer information, i.e.:

[0018]

[0019] Where ⊙ represents element-wise multiplication;

[0020] The current LSTM hidden layer state is obtained by updating the current memory cell through the output gate:

[0021] h t =o t ⊙tanh(C t )

[0022] Step Six: Agent Policy Learning; First, the agent initializes the action observations as a stochastic process Ran and obtains an initial state value z; then, for each agent i, according to the parameter θ... i policy network μ θi The output selects its own action a i =μ θi (z i After adding Ran, execute the actions of all agents: a = (a1, a2, ..., a...). N The system retrieves the reward r and the system's new state value z', and then stores the quadruple (z, a, r, z') in the experience replay buffer. Then, for each agent, the changed state value z' is assigned to z; next, from the experience replay buffer... A small batch of S quadruplet samples (z) are randomly selected from the sample. j ,a j ,r j ,z ’j Let j be the sample number, 1 ≤ j ≤ S, and define the reward corresponding to the j-th sample for agent i. Among them, all agent actions a *j Agent i faces state z j New moves Replace the original action a of the sample j The corresponding positions are combined to form, that is γ is a discount factor between 0 and 1. Let Q be the action value function of agent i, which is defined as follows: That is, the total expected return R given state z and action a. iThe expectation, E, is the expectation operator considering all possible cases, and the total expected return of agent i. T represents the time range, γ t The discount factor γ raised to the power of t is used as the return weight, γ t The value decreases as t increases, meaning the further away from the current time, thus reducing the reward for agent i at time t, which is further from the current time. The impact; finally, by minimizing the loss function. To update the action value Q function of agent i That is, given state z j and action a j The expected total return under the given conditions, where S is the expected value from the experience replay buffer. The number of small-batch quadruplets randomly selected from S, where j is the sample index, 1≤j≤S, R j For the reward corresponding to the j-th sample for agent i, the policy gradient of the sample is used. Update the policy network parameters for each agent, where S is the value from the experience replay buffer. The number of mini-batch quadruplets randomly selected from the sample, where j is the sample index, 1≤j≤S, μ i (z j Let θ be the policy network of agent i, with parameters θ. i Accepting state z j Input and output actions Together with the actions of other agents, they form all agent actions a *j , Let Q be the action value function of agent i, i.e., given state z. j and action a *j The expected total return under the given conditions and The policy network and the action-value Q function are respectively related to the parameter θ. i and input action a *j The gradient, the new parameters θ' of the policy network i For the old parameter θ i With policy gradient Weighted average: The weight τ represents the learning rate;

[0023] Step 7: Iterate through the above steps until the policy optimization network model of this method converges to the optimal value.

[0024] The beneficial effects of this invention are as follows: By introducing heterogeneous motifs, this invention can extract rich semantic information in complex multi-agent systems. This information can improve the cooperative communication effect between agents, thereby improving the collaborative decision-making among agents. At the same time, the attention mechanism and long short-term memory mechanism introduced in this invention can distinguish agents of different importance, increase the density of important messages, and simultaneously model time and space, supporting the retrospectiveness and timeliness of collaborative decision-making among agents. Attached Figure Description

[0025] Figure 1 This is the basic framework of the present invention.

[0026] Figure 2(a) is an example of the entity modality matrix;

[0027] Figure 2(b) is a schematic diagram of the calculation method of the solid modulus matrix. Detailed Implementation

[0028] The specific embodiments of the present invention will be further described in detail below with reference to the technical solutions and accompanying drawings.

[0029] Example

[0030] A knowledge-driven collaborative decision-making method for biological intelligent groups, comprising the following steps:

[0031] Step 1: Model the multi-agent system and obtain the observation vector of agent i at time t. As its node attribute And calculate the N×N adjacency matrix A at time t. t ,like Figure 1 As shown in the figure. This embodiment sets up 6 intelligent agents, including 4 predators and 2 prey; the observation status specifically includes its own speed, its own position, the relative position of landmarks, the relative positions of other intelligent agents, and other communication information.

[0032] Step 2: For each agent, as shown in Figures 2(a) and 2(b), calculate its entity motif degree using third-order and fourth-order motifs. And the dynamic graph entity modality matrix is In this embodiment, p=5, and high-order topological relationships are extracted using a third-order motif and the first three fourth-order motifs.

[0033] Step 3: Calculate the attention weight coefficient e of node j with respect to node i. ij =a(Wh i ,Wh j and standardize. Therefore, the aggregate representation of node i at time t Then, the representation of node i at time t is calculated using a multi-head attention mechanism. Higher-order attributes are also embedded and learned following the same process. Similarly, each entity motif degree in the dynamic graph entity motif degree matrix M generated in step two is... The representation at time t is as follows

[0034] Step 4: Feature aggregation uses a summation aggregator to add the elements at corresponding positions in the matrix, and then represents the nodes obtained in Step 3. Representation of solid model The original observation embedding matrix H and the higher-order attribute embedding matrix H are composed of s Aggregation is performed to obtain the characteristic aggregation matrix H. agg =H+H s .

[0035] Step 5: Input the feature aggregation matrix into the LSTM for processing.

[0036]

[0037]

[0038]

[0039] Calculate the current hidden layer The information retained in the previous memory unit is added to the current information to obtain the current memory cell. Here, ⊙ represents element-wise multiplication; then, the output gate updates the LSTM hidden layer state h at the current time step based on the current memory cell. t =o t ⊙tanh(C t In this embodiment, the time window is set to its maximum, meaning that the state at time t depends on all previous states.

[0040] Step Six: Agent Policy Learning. First, the agent initializes action observations as a stochastic process Ran and obtains an initial state value z; then, for each agent i, based on the parameter θ... i policy network μ θi The output selects its own action a i =μ θi (z i After adding Ran, execute the actions of all agents: a = (a1, a2, ..., a...). N The system retrieves the reward r and the system's new state value z', and then stores the quadruple (z, a, r, z') in the experience replay buffer. Then, for each agent, the changed state value z' is assigned to z; next, from the experience replay buffer... A small batch of S quadruplet samples (z) are randomly selected from the sample. j ,a j ,r j ,z ’j Let j be the sample number, 1 ≤ j ≤ S, and define the reward corresponding to the j-th sample for agent i. Among them, all agent actions a *j Agent i faces state z j New move a' i =μ θi (z j Replace the original action of the sample a j The corresponding positions are combined to form, that is γ is a discount factor between 0 and 1. Let Q be the action value function of agent i, which is defined as follows: That is, the total expected return R given state z and action a. i The expectation, E, is the expectation operator considering all possible cases, and the total expected return of agent i. T represents the time range, γ t The discount factor γ raised to the power of t is used as the return weight, γ t The value decreases as t increases, meaning the further away from the current time, thus reducing the reward r of agent i at time t, which is further away from the current time. i t The impact; finally, by minimizing the loss function. To update the action value Q function of agent i That is, given state z j and action a j The expected total return under the given conditions, where S is the expected value from the experience replay buffer. The number of small-batch quadruplets randomly selected from S, where j is the sample index, 1≤j≤S, R j For the reward corresponding to the j-th sample for agent i, the policy gradient of the sample is used. Update the policy network parameters for each agent, where S is the value from the experience replay buffer. The number of mini-batch quadruplets randomly selected from the sample, where j is the sample index, 1≤j≤S, μ i (z j Let θ be the policy network of agent i, with parameters θ. i Accepting state z j Input, output action a' i Together with the actions of other intelligent agents, they constitute all intelligent agent actions a *j , Let Q be the action value function of agent i, i.e., given state z. j and action a *jThe expected total return under the given conditions and The policy network and the action-value Q function are respectively related to the parameter θ. i and input action a *j The gradient, the new parameters θ' of the policy network i For the old parameter θ i With policy gradient Weighted average: The weight τ represents the learning rate; in this embodiment, the batch size is set to 1024 and τ is set to 0.95.

[0041] Step 7: Iterate through the above steps until the policy optimization network model of this method converges to the optimal value.

[0042] Verification results:

[0043] This invention designs three multi-agent cooperative tasks—formation control, group control, and predator-prey game—to verify the effectiveness of a knowledge-driven biological intelligent swarm collaborative decision-making method. In all tasks, agents have only a limited scope of observation; the only way to increase this scope is through communication to obtain more information from other agents, which encourages cooperation among agents. In the verification, the knowledge-driven biological intelligent swarm collaborative decision-making method is compared with MADDPG, R-MADDPG, and TRANSFER. MADDPG uses a centralized training and distributed execution framework to enhance the cooperative behavior of agents in a mixed cooperative-competitive environment, requiring the states of all agents during training to construct its critique network. R-MADDPG is a recurrent version of MADDPG in a partially observable environment. TRANSFER embeds the multi-agent system as a graph and uses an attention mechanism for communication, but does not consider the time-varying relationships between agents. Each method performs 100 independent simulations for each task, and the average performance of success rate, number of steps, and reward is calculated as evaluation metrics, as shown in Tables 1, 2, and 3.

[0044] Task 1: Formation Control

[0045] Formation control is divided into four cases: 3, 6, 10, and 15 agents. In these cases, the goal of all agents is to be uniformly distributed around the center and avoid collisions. Table 1 shows the average performance of each method in 100 independent simulations of the formation control task, including success rate, number of steps, and reward.

[0046] Table 1 Average performance of each method for formation control tasks

[0047]

[0048] For the case of three agents, all methods exhibit similar performance. This is because the relationships between agents are relatively simple, and satisfactory policies can be learned regardless of whether these methods employ attention mechanisms. Nevertheless, it can still be observed that, upon completion of training, the knowledge-driven biological intelligence swarm collaborative decision-making method can obtain more rewards than all other methods. As the number of agents increases, other methods perform poorly, while the performance of the knowledge-driven biological intelligence swarm collaborative decision-making method remains essentially unchanged, obtaining more rewards than other methods. Therefore, the results of formation control demonstrate that the knowledge-driven biological intelligence swarm collaborative decision-making method can handle complex interactions between a large number of agents, can utilize LSTM to capture dynamic graph structures induced by moving agents, and has relatively better handling capabilities for complex interactions and dynamic graph structures, making it more adept at handling scenarios with a large number of agents.

[0049] Task 2: Group Control

[0050] Group control is divided into two scenarios: 6 agents against 2 landmarks and 10 agents against 2 landmarks. In these scenarios, all agents need to be divided into two groups, evenly distributed around their respective landmarks, and avoid collisions. Table 2 shows the average performance of each method in 100 independent simulations of the group control task, including success rate, number of steps, and reward. Similarly, the knowledge-driven biological intelligence swarm collaborative decision-making method performs better and achieves greater rewards than other methods, while the other methods perform relatively poorly. Notably, MADDPG failed to complete any of the tasks.

[0051] Table 2 Average performance of each method for grouped control tasks

[0052]

[0053] Task 3: Predator-Prey Game

[0054] The predator-prey game comprises four game scenarios: 3v1, 5v2, 7v4, and 3v3. Prey must avoid being captured. Predators need to capture all prey, but move more slowly and have more limited observation and communication range. This necessitates cooperation among predators to achieve the objective. Table 3 shows the average performance of each method in 100 independent simulations of the predator-prey game task, including success rate, number of steps, and reward. Similarly, in the 3v1 scenario, all methods can complete the task with similar performance. This is because the relationships between agents are relatively simple, allowing these methods to learn satisfactory strategies regardless of whether they have attention mechanisms. Nevertheless, it can be seen that, upon completion of training, the knowledge-driven biological intelligence swarm collaborative decision-making method can achieve more rewards with fewer steps than all other methods. As the number of agents increases, the optimal strategy for agents becomes less explicit because the interactions between groups of agents become more complex and diverse. For example, a predator can cooperate with other predators to capture prey. If prey is nearby, it can also capture prey alone. As shown in Table 3, the knowledge-driven bio-intelligence group collaborative decision-making method outperforms other methods in terms of success rate, step consumption, and reward.

[0055] Table 3. Average performance of each method in the predator-prey game task.

[0056]

[0057] In summary, the knowledge-driven bio-intelligent swarm collaborative decision-making method can learn the complex interactions between a large number of agents, capture the dynamic graph structure caused by the mobile agent, has a stronger ability to handle complex interactions and dynamic graph structures, and performs well in multi-agent cooperative tasks.

[0058] The embodiments described above are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A knowledge-driven bio-smart swarm collaborative decision making method, characterized in that, Firstly, the spatial relationship between multi-agent and the observation value of each agent are constructed into a graph. Secondly, the high-order logic relationship between agents is extracted through high-order body representation and taken as the high-order attribute of the agent. Thirdly, the attention weight coefficient of other adjacent agents is calculated through the attention mechanism, and the learning process is stabilized through the multi-head attention mechanism as an aggregator to obtain the embedding of two types of attributes. Then, in order to further represent the entity, the high-order attribute embedding and the observation value embedding of the entity are combined. Then, the long short-term memory network is used to capture the time dependence and complex dynamic structure of the system to obtain more effective state observations. Finally, these observation states are input into the policy optimization network to predict the state and the probability distribution of all possible actions. The specific steps are as follows: Step one: Multi-agent system modeling, define each agent as a node in the dynamic graph; Specifically, define the observation vector of agent i at time t ot i as its node attribute ht i, and define the N*N adjacency matrix A at time t t , where the ij element of the adjacency matrix is a ij , when agent i communicates with agent j a ij = 1, otherwise a ij = 0, N is the number of nodes; Step 2: For each agent, define its entity modality degree. =[E1,E2,…,E p ]; where E p This indicates that agent i is simultaneously included in multiple different p-th motifs; further, the dynamic graph entity motif degree matrix is ​​defined as M=[ , …, ] T In this method, the dynamic graph entity modality matrix is ​​used as a higher-order attribute matrix for training. Step three: compute the attention weight coefficient e of node j for node i ij = a(Wh i , Wh j ) and the normalized form of attention weight coefficient a ij = ; where a represents a single-layer feedforward neural network, W is a linear weight matrix, LeakyReLU is a nonlinear activation function with a negative input slope, represents the neighbor node set of node i; the aggregated representation ht’ i of node i at time t is = σ( ), where σ is a nonlinear function; then, the multi-head attention mechanism is used as an aggregator to stabilize the attention learning process and extract different state representations of the agent from different state word spaces; at this time, the representation of node i at time t is updated to ht’ i = ||Mm=1σ( ), where m represents the number of multi-head attention; similarly, the representation of each row of the high-order attribute matrix M generated in step two at time t is Et’ i = ||Mm=1σ( ). ​ Step four: the node representation ht'i obtained in step three and the row representation Et'i are combined to form the original observation embedding matrix H and the high-order attribute embedding matrix H s Aggregation is performed to better represent the relationship between entities and the overall system, and a feature aggregation matrix H is obtained agg =H+H s ; Step five: Use LSTM to capture the time dependence and dynamic topology of the multi-agent system; LSTM consists of three gates: forget gate f t , input gate i t and output gate o t , the detailed calculation process is as follows: f t = σ(W f · [h t-1 , hagg] + b f ); i t =σ(W i ·[h t-1 ,ht agg]+b i ) ; o t =σ(W o ·[h t-1 ,ht agg]+b o ) ; wherein σ(·) represents a sigmoid function, and the sigmoid function outputs σ(x) = 1 / (1 + e , W f , W i , W o respectively represent weight matrices of the forget gate, the input gate, and the output gate, b f , b i , b o respectively represent bias vectors of the forget gate, the input gate, and the output gate, h t-1 t-1 represents an LSTM hidden layer state output at the t-1 time point. The current hidden layer is defined as t = tanh(W C · [h t-1 , hagg] + b C ), W C represents a weight matrix, b C represents a bias vector, and tanh(x) = = , and the output value is in the interval [-1, 1]; thus, the current memory cell is composed of two parts, one is the information retained by the previous unit memory cell through the forgetting gate, and the other is the current hidden layer information updated through the input gate; the current memory cell is obtained by adding the information retained by the previous unit memory cell and the current hidden layer information, that is: C t = C t-1 ⊙f t + t ⊙i t ; Wherein, is the element-wise multiplication; The current time LSTM hidden layer state output is obtained through the output gate according to the current memory cell update: h t =o t ⊙tanh(C t ) ; Step Six: Agent Policy Learning; First, the agent initializes action observations as a stochastic process. And obtain the initial state value z; then, for each agent i, according to the parameter θ i policy network μ θi The output selects its own action a i =μ θi (z i )+ Then, execute the actions of all agents: a = (a1, a2, ..., a...). N The system retrieves the reward r and the system's new state value z', and then stores the quadruple (z, a, r, z') in the experience replay buffer. Then, for each agent, the changed state value z' is assigned to z; next, from the experience replay buffer... A small batch of S quadruplet samples (z) are randomly selected from the sample. j ,a j ,r j ,z ’j ), where j is the sample number, S, where S is the reward for the j-th sample to agent i. All agent actions State is faced by agent i New moves Replace the original action a of the sample j The corresponding positions are combined to form, that is γ is a discount factor between 0 and 1. Let Q be the action value function of agent i, which is defined as follows: That is, given state and actions Total expected return under the condition The expectation, E, is the expectation operator considering all possible cases, and the total expected return of agent i. T represents the time range. The discount factor γ raised to the power of t is used as the return weight. The larger t is, the smaller it becomes as the distance from the current time increases, thus reducing the reward for agent i at time t, which is further away from the current time. The impact; finally, by minimizing the loss function. To update the action value Q function of agent i That is, given state and actions The expected total return under the given conditions, where S is the expected value from the experience replay buffer. The number of small batches of quadruplets randomly selected from the sample. For the reward corresponding to the j-th sample for agent i, the policy gradient of the sample is used. Update the policy network parameters for each agent, where, Let θ be the policy network of agent i. i Accepting status Output action Together with the actions of other agents, they constitute all agent actions. , Let Q be the action value function of agent i, i.e., given the state. and actions The expected total return under the given conditions and The policy network and the action-value Q function are respectively related to the parameter θ. i and input actions The gradient, new parameters of the policy network For the old parameter θ i With policy gradient Weighted average: Weight Represents the learning rate; Step seven: through the above steps, iterative training is continuously carried out until the policy optimization network model of the method converges to the optimal.

Images