A method for solving intelligent agent control tasks in continuous action space

Abstract

The present invention discloses a method for solving an intelligent agent control task facing a continuous action space, the method comprising: using VAE to embed observations from an external environment and performing dimensionality reduction processing on input data; the intelligent agent continuously interacts with the environment to generate sample data for training strategies; sampling virtual targets, recalculating rewards and adding them to a replay buffer to complete the expansion of sample data; performing kernel density estimation on the state that the intelligent agent has achieved, and calculating the distance between the current state and the desired target state as a coefficient to generate an intrinsic reward at each moment; using the intrinsic reward to set a dynamic ratio and select the appropriate initial state for the next episode; through the training method, the intelligent agent facing the continuous action space will automatically learn the intelligent agent control method and realize the artificially arranged tasks. The present invention is used for sparse reward tasks in deep reinforcement learning, and can improve the performance of the algorithm in completing tasks.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning, and in particular to a method for solving intelligent agent control tasks in a continuous action space. Background Art

[0002] In actual application scenarios, the control tasks of intelligent agents are mostly oriented to continuous action spaces, and the implementation is relatively complex. At present, the relevant technologies of deep reinforcement learning are mainly used to solve this problem, train them, and learn to complete the corresponding tasks. However, in this scenario, the sparse reward problem of deep reinforcement learning algorithms will become more and more serious, seriously affecting the performance of the algorithm. As we all know, reward signals are an important part of reinforcement learning, and strategy optimization depends entirely on reward guidance. Especially for commonly used offline strategy algorithms, the sample utilization rate itself is low. The lack of guidance from reward signals will slow down the rate of the intelligent agent, and complex tasks cannot be learned. Therefore, the sparse reward problem is a major problem faced by the solution method of intelligent agent control tasks facing continuous action spaces. At present, there are two main mainstream methods to solve this problem. One is to improve the exploration efficiency of intelligent agents facing continuous action spaces through some methods; the other is the classic ex post experience replay method. Although the two methods can solve the problem of sparse rewards to a certain extent, they still have their own defects and the actual use effect is not good.

[0003] The first type of method is to prioritize the replay buffer samples generated during the learning process of the agent facing the continuous action space according to different criteria, and try to find new states to improve the exploration efficiency of policy training. For example, by using the size of TD-error to extract and reuse samples in the experience pool, and using state prediction errors to find novel states, the agent facing the continuous action space is encouraged to explore quickly. However, this type of method usually has the following problems: First, the model is more complex after introducing priorities and prediction networks, and the amount of calculation is large. Second, the blind pursuit of novel states will destroy the balance between exploration and utilization of the agent facing the continuous action space, and it is easy to fall into the local optimum.

[0004] The second type of experience replay algorithm uses the idea of ​​"hindsight" to modify the initial goal set by the agent facing the continuous action space to the goal actually achieved during the exploration process, so as to transform the failure experience of the agent facing the continuous action space into successful experience. In this way, the agent facing the continuous action space can obtain more valuable sample data to encourage learning. However, this method still has the problem of low sample efficiency, mainly because its exploration strategy still uses random exploration. With the increase of task complexity and state space dimension, it is difficult to access the entire environment by taking random action sequences in complex tasks. Therefore, this method is limited to relatively simple tasks, and the actual effect of use in complex tasks is average. Summary of the invention

[0005] In view of the related defects of the above-mentioned prior art, the present invention provides a deep reinforcement learning method based on kernel density estimation, which aims to solve the problem that the existing post-experience replay method lacks effective exploration in complex tasks and the balance between the related efficient exploration methods and utilization in the intelligent agent control task facing the continuous action space. Finally, the solution of the complex intelligent agent control task facing the continuous action space is realized.

[0006] To achieve the above object, the present invention proposes a method for solving intelligent agent control tasks in a continuous action space, comprising the following steps:

[0007] Step 1: First, use a machine learning existing data operation tool, the variational autoencoder (VAE), to embed observations from the external environment. Then, the agent is modeled for reinforcement learning, and the external environment observation data and high-dimensional images or state feature values ​​are used as input through the encoder network q in the variational autoencoder. φ (where φ is the encoder network q φ The variational autoencoder outputs the parameters μ and σ of the multivariate Gaussian distribution;

[0008] The variational autoencoder then samples the latent vector, which is then decoded by the decoder network p ψ (where ψ is the decoder network p ψ The present invention jointly optimizes the parameters φ and ψ of the encoder and decoder to maximize the loss function of the variational autoencoder;

[0009] In the present invention, the variational autoencoder supports taking high-dimensional images or state feature values ​​as input of the variational autoencoder;

[0010] Step 2: Sample data collection. This invention follows the basic framework of traditional offline reinforcement learning methods and goal-guided reinforcement learning algorithms to train the action selection strategy model of the agent. The agent interacts with the environment multiple times. Before each interaction episode, the target state g and the initial state s0 are determined. Then the agent interacts with the environment and a sample trajectory τ is sampled based on the data obtained from the interaction. The trajectory is in the form of [s0, a0, r0, s1, a1, r1, ..., s T , a T , r T ], where T is the time step. After calculating the reward based on the target state, the sample trajectory data obtained from the sampling is combined with the target state g and the state ag that has been reached at the current moment to construct a single-step transfer tuple and convert each single-step transfer tuple (s t , a t , r t ,g,s t+1 , ag) is stored in the replay experience buffer as a sample data;

[0011] Among them, s t Represents the current state, a t represents the action taken at the current moment, r t Represents the delayed reward obtained after taking action at the current time t. The value is calculated by the reward function. t+1 represents the state at the next moment t+1, ag represents the state reached at the current moment, t=1, 2, ..., T, the reward function is designed as follows:

[0012]

[0013] The reward function follows the standard sparse reward binary reward function design, where is the characteristic representation of the current state at time t, and ||*|| represents the operation of taking the absolute value, that is, taking the absolute value of the difference between the two states.

[0014] δ is a preset threshold, which means that when the gap between the state reached by the agent and the expected target state is within the threshold, the task can be regarded as successful. At this time, the reward function r g (s t , a t , g)=0.

[0015] At the same time, the size of the replay experience buffer is fixed to N, and different values ​​are set according to different tasks of the agent. The idea of ​​queue is used in the design. When the replay experience buffer is full, the newly collected single-step transfer tuple will replace the single-step transfer tuple that enters the buffer earliest, following the first-in-first-out rule.

[0016] Step 3: Expand the replay experience buffer using the idea of ​​​​post-experience replay:

[0017] By reselecting several artificially set “virtual targets” g′ different from the original targets on the current trajectory τ (here k are selected in the experiment, k is a hyperparameter in the algorithm, and the effect is best when k=4), re-examining this trajectory on this basis, using these “virtual targets” and the above reward function to re-give the reward for each single-step transfer tuple in this trajectory, and converting these new tuples (s t , a t , r t ,g′,s t+1 , ag) is also stored in the replay experience buffer, and the storage rules are as described in step 2.

[0018] At this time, the replay experience buffer will contain a portion of the delayed reward r t Single-step transfer tuples with positive values ​​will play a beneficial role in the learning of the agent.

[0019] Step 4: Use the kernel density estimation method to estimate the density value of the agent's current realized state and calculate the corresponding intrinsic reward:

[0020] The present invention introduces intrinsic rewards To expand the reward mechanism, the delayed reward obtained by the agent consists of two parts: in is the intrinsic reward at the current time t, is the external reward at the current time t, that is, the reward given by the environment. The goal of the agent is to maximize the value of the total delayed reward accumulated to time T.

[0021] At the current time t, store all realized states ag in the replay experience buffer into the set G ag In , the traditional kernel density estimation is used to measure the novelty of the state. The density of a realized state is defined as the quotient of the number of realizations of the realized state and the number of realizations of all realized states in the set. ag The data in is the sample data. Then the kernel density estimation algorithm of the standard Gaussian kernel function is used to estimate the density of each state in the set. This kernel density estimation model is recorded as F ag , for the set G ag Each achieved goal in g i , density ρ i The calculation is as follows:

[0022] ρ i =F ag (g i )

[0023] Therefore, the intrinsic reward design should be inversely proportional to the density of the current state. The state with a larger density means that it has been visited many times and is no longer desirable, so it can be ignored; the state with a smaller density is more novel to the agent, and it should be explored in the future. At the same time, considering the existence of the original target state that is expected to be reached at the beginning of training, the state that is closer to the expected target state and has a smaller density is the state that the present invention focuses on.

[0024] Therefore, the kernel density estimation algorithm is used to estimate the current state s t In the collection G ag The density value in, and the gap d between the current state and the desired target state g, the intrinsic reward The design is as follows:

[0025]

[0026] Among them, f h (*) is a kernel density estimation function with a Gaussian kernel, which can produce a smooth density estimate of the state *, Indicates f h (*) Take the average value. j represents the jth realized state, η is a predefined hyperparameter, 0, n) represents taking The Clip function is used to ensure the difference between the success state and the failure state. At the same time, the smaller the gap d is, the greater the value of the intrinsic reward is. As the training of the action selection strategy model of the training agent continues, the intrinsic reward will gradually converge to 0.

[0027] Step 5: Selection of dynamic initial state:

[0028] Different from the traditional method that explores the environment from a fixed or random initial state in each interaction episode, in order to further improve the efficiency of exploring the environment, the present invention allows the agent to start exploring from novel states that it has experienced before, because the agent has acquired some capabilities.

[0029] For each single-step transition tuple in the replay experience buffer, first use the intrinsic reward generated in step 4 Those intrinsic rewards First, select the single-step transfer tuples with positive values; secondly, select the successful single-step transfer tuples that reach the target state; finally, select the one with the largest intrinsic reward among these selected tuples, and use the ag in the tuple with the largest intrinsic reward as the initial state of the next interaction round (episode).

[0030] Considering the balance between exploration and exploitation, the dynamic ratio α will be used to determine whether to select a novel state that is less frequently reached as the initial state of the next interaction round, or to select the initial state at the beginning of each interaction round fixed before model training starts as the initial state of the next interaction round. The specific selection is as follows: before each interaction round starts, a number between [0, 1] will be randomly selected. When the number is less than α, the novel state will be selected as the initial state of the interaction round. Otherwise, the fixed initial state at the beginning of each interaction round will be selected as the initial state of the interaction round. The ratio α is designed as follows:

[0031] α=γe -2×succsss_rate

[0032] Among them, × represents multiplication, γ is a hyperparameter, γ∈(0,1), and success_rate is the task success rate at the current moment. This can not only ensure further diversification of the initial state, but also prevent performance degradation in the later learning process.

[0033] According to the above steps, the intelligent agent facing the continuous action space obtains sample data for training by interacting with the environment. The intelligent agent will automatically learn how to control and complete various tasks assigned by humans to achieve the solution of the intelligent agent control task.

[0034] This invention proposes an efficient solution to the sparse reward problem in current deep reinforcement learning algorithms. The kernel density estimation algorithm is used to generate corresponding intrinsic rewards, combined with post-experience replay, to encourage the agent to explore efficiently, making up for the defects of random exploration. At the same time, the exploration and utilization are balanced by setting the dynamic initial state to avoid falling into the local optimal problem. In addition, the initial variational autoencoder can process relatively complex tasks, which truly improves the efficiency of solving complex sparse reward tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the principle diagram of the variational autoencoder of the present invention;

[0036] Figure 2 Generate a model for the intrinsic rewards of the present invention;

[0037] Figure 3 This is the overall structure diagram of the sparse reward task solving model based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation modes and the accompanying drawings.

[0039] The present invention proposes a method for solving intelligent agent control tasks in a continuous action space, which comprises the following steps:

[0040] Step 1: First, use a machine learning existing data operation tool, the variational autoencoder VAE, to embed observations from the external environment, that is, external environment observation data, to model the agent for reinforcement learning, and use the external environment observation data and high-dimensional images or state feature values ​​as input through the encoder network q in the variational autoencoder φ After processing, the variational self-encoder outputs the parameters μ and σ of the multivariate Gaussian distribution, where φ is the encoder network q φ Parameters;

[0041] The variational autoencoder then samples the latent vector, which is then decoded by the decoder network p ψ Decode the latent vector into the original state space, where ψ is the decoder network p ψ , jointly optimize the parameters φ and ψ of the encoder and decoder to maximize the loss function of the variational autoencoder;

[0042] Step 2: Sample data collection

[0043] Following the basic framework of traditional offline reinforcement learning methods and goal-guided reinforcement learning algorithms, the action selection strategy model of the agent is trained. This model represents how the agent facing the continuous action space selects appropriate actions under different environmental conditions. The model will be introduced in detail in P1 later. The agent interacts with the environment multiple times. Before the start of each interaction episode, the target state g and the initial state s0 are determined. Then the agent interacts with the environment and a sample trajectory τ is sampled based on the data obtained from the interaction. The trajectory is in the form of [s0, a0, r0, s1, a1, r1, ..., s r , a T , r T ], where T is the time step. After calculating the reward based on the target state, the sample trajectory data obtained from the sampling is combined with the target state g and the state ag reached at the current moment to construct a single-step transfer tuple and convert each single-step transfer tuple (s t , a t , r t ,g,s t+1 , ag) is stored in the replay experience buffer as a sample data; where s t Represents the current state, a t represents the action taken at the current moment, r t Represents the delayed reward obtained after taking action at the current time t. The value is calculated by the reward function. t+1represents the state at the next moment t+1, ag represents the state reached at the current moment, t=1, 2, ..., T, the reward function is designed as follows:

[0044]

[0045] The reward function follows the standard sparse reward binary reward function design, where is the characteristic representation of the current state at time t, ||*|| represents the operation of taking the absolute value, that is, taking the absolute value of the difference between the two states, δ is the preset threshold, which means that when the difference between the state reached by the agent and the expected target state is within the threshold, it can be regarded as a task success. At this time, the reward function r g (s t , a t , g)=0;

[0046] At the same time, the size of the replay experience buffer is fixed to N, and different values ​​are set according to different tasks of the agent. The idea of ​​queue is used in the design. When the replay experience buffer is full, the newly collected single-step transfer tuple will replace the single-step transfer tuple that enters the buffer earliest, following the first-in-first-out rule.

[0047] Step 3: Expand the replay experience buffer using the idea of ​​​​post-experience replay:

[0048] By reselecting several artificially set “virtual targets” g′ different from the original target state g on the current trajectory τ, we re-examine the trajectory τ on this basis, and use these “virtual targets” and the above reward function to re-give the reward of each single-step transfer tuple in the trajectory τ, and convert these new tuples (s t , a t , r t ,g′,s t+1 , ag) is also stored in the replay experience buffer. The storage rules are the same as step 2. At this time, the replay experience buffer will contain a part of the delayed reward r t Single-step transfer tuples with positive values ​​will play a beneficial role in the learning of the agent;

[0049] Step 4: Use the kernel density estimation method to estimate the density value of the agent's current realized state and calculate the corresponding intrinsic reward:

[0050] By introducing intrinsic rewards To expand the reward mechanism, the delayed reward obtained by the agent consists of two parts: in is the intrinsic reward at the current time t, is the external reward at the current moment t, that is, the reward given by the environment. The goal of the agent is to maximize the value of the total delayed reward accumulated to the moment T;

[0051] At the current time t, store all realized states ag in the replay experience buffer into the set G ag In the example, the traditional kernel density estimation is used to measure the novelty of the state. From the single-step transfer tuple, it can be seen that when the task is successfully completed, the current state in the last single-step transfer tuple is the state of task completion. At this time, the state ag reached by the agent is the same as g. Therefore, in the set G ag The state ag that has been achieved is the same as the original target state g. The density of a certain achieved state is defined as the number of times the achieved state is achieved and the set G ag The quotient of the number of realizations of all realized states in the set G ag The data in is the sample data, and then the kernel density estimation algorithm of the standard Gaussian kernel function is used to estimate the density of each state in the set. This kernel density estimation model is recorded as F ag , for the set G ag Each achieved goal in g i , density ρ i The calculation is as follows:

[0052] ρ i =F ag (g i )

[0053] Therefore, the intrinsic reward design should be inversely proportional to the density of the current state. A state with a larger density means that it has been visited many times and is no longer desirable, so it should be ignored. A state with a smaller density is more novel to the agent and should be explored in the future. At the same time, considering the existence of the original target state that is expected to be reached at the beginning of training, the state that is closer to the expected target state and has a smaller density should be the focus of attention.

[0054] Therefore, the kernel density estimation algorithm is used to estimate the current state s t In the collection G ag The density value in, and the gap d between the current state and the desired target state g, the intrinsic reward The design is as follows:

[0055]

[0056] Among them, f h (*) is a kernel density estimation function with a Gaussian kernel, which produces a smoothed density estimate of the state *, Indicates f h (*) Take the average value; ag j represents the jth realized state, η is a predefined hyperparameter, 0, n) represents taking The Clip function is used to ensure the difference between the success state and the failure state; at the same time, the smaller the gap d, the greater the value of the intrinsic reward;

[0057] As the agent's action selection strategy model training continues, the intrinsic reward Will gradually converge to 0;

[0058] Step 5: Selection of dynamic initial state:

[0059] In order to further improve the efficiency of exploring the environment, since the agent has acquired some capabilities, let the agent start exploring from the novel states it has experienced before;

[0060] For each single-step transition tuple in the replay experience buffer, first use the intrinsic reward generated in step 4 Those intrinsic rewards First, select the single-step transition tuples with positive values; second, select the successful single-step transition tuples that reach the target state; finally, select the one with the largest intrinsic reward from these selected tuples, and use the ag in the tuple with the largest intrinsic reward as the initial state of the next interaction episode;

[0061] Considering the balance between exploration and exploitation, before each interaction round begins, a number between [0, 1] will be randomly selected. When the number is less than α, the novel state will be selected as the initial state of the interaction round. Otherwise, the fixed initial state at the beginning of each interaction round will be selected as the initial state of the interaction round. The ratio α is designed as follows:

[0062] α=γe -2×succsss_rate

[0063] Among them, × represents multiplication, γ is a hyperparameter, γ∈(0,1), and success_rate is the task success rate at the current moment;

[0064] According to the above steps, the intelligent agent facing the continuous action space obtains sample data for training by interacting with the environment. The intelligent agent will automatically learn how to control and complete various tasks assigned by humans to achieve the solution of the intelligent agent control task.

[0065] See also Figure 1 In order to improve the performance of sparse reward tasks in complex high-dimensional states, the present invention uses a variational autoencoder to compress the external environment observation data.

[0066] First, the encoder in step S1 is a neural network. Its input is the observation value of the intelligent agent facing the continuous action space, which can be an image or a state feature value. The output of step S1 is the latent vector z, which is the dimension reduction of the input state feature value. It also needs to obey the Gaussian distribution. The process of the encoder outputting z can actually be decomposed into two steps:

[0067] 1) First, the encoder outputs the parameters of the Gaussian distribution (mean μ, variance σ), which are different for each data point of the external environment observation data;

[0068] 2) Fuse the noise with the Gaussian distribution and obtain z by sampling from the Gaussian distribution fused with the noise.

[0069] The decoder in step S2 is also a neural network. Its input is the latent vector z output by step S1, and its output is the probability distribution of the data obtained by decoding the latent vector z into the original state space.

[0070] The variational autoencoder consists of an encoder and a decoder. The input of the variational autoencoder is encoded by the encoder, and the encoded result can be decoded by the decoder. The loss function of the variational autoencoder is a negative log-likelihood function with a regularization term. Because there is no shared latent vector between all data points of the variational autoencoder's input data, the loss of each data point is l m are independent, the total loss is the sum of the losses of each data point, and M represents the total number of data points. i The loss i It can be expressed as:

[0071]

[0072] Among them, q φ is the encoder φ is the parameter of the encoder network, p ψ Decoder ψ is the parameter of the decoder network, s (i) is the data point x i The state of the agent at time , β is a hyperparameter. The first term D KL (q φ (z|s (i) )||p(z)) is the regularizer, i.e. the encoder distribution q φ (z|s (i)) and the Kullck-Leibler divergence between p(z). This parameter measures the discreteness, that is, how much information is lost when the original data is encoded by the encoder and then decoded by the decoder compared to the original data. At the same time, p(z) represents a priori specified standard normal distribution, that is: p(z) = N(0, 1) (N(0, 1) represents the standard normal distribution, that is, a normal distribution with an expected value of 0 and a standard deviation of 1). The second term is the expected negative log-likelihood - reconstruction loss. p ψ (s (i) |z)] is the expected negative log-likelihood-reconstruction loss, where p ψ (s (i) |z) represents the decoder distribution, Indicates that the encoder distribution is q φ (z|s (i) ) is used to find the expectation under the condition that the expectation is the distribution of the result obtained after the data is input into the encoder network for encoding;

[0073] The present invention performs dimensionality reduction processing on the external environment observation input data through steps S1 and S2, and then follows the basic process of the traditional offline strategy reinforcement learning algorithm. The core part is the intrinsic reward generation model. For details, see Figure 2 :

[0074] Those skilled in the art will understand that the combination of intrinsic rewards and extrinsic rewards can encourage the agent to explore efficiently. For some novel states that are rarely reached, they may contain more information, which can help the agent to explore the less-reached states more fully to obtain more knowledge about the environment. Therefore, a higher intrinsic reward can be given to guide it. See the reward generation model. Figure 2 , which is divided into the following 5 steps:

[0075] P1: Using the idea of ​​offline strategy algorithm DDPG, the agent interacts with the environment, generates sample data, and collects the sample data. The replay experience buffer contains several experience tuples (s t , a t , r t ,g,s t+1 , ag), the achieved target set G ag , the desired target set G dg , where the action selection strategy is as follows and H is the added noise.

[0076] a t =π(s t , g)+H

[0077] Among them, π(s t, g) means that in the action selection strategy π, when the parameters are state s t The action probability distribution at target g.

[0078] P2: Virtual target sampling, expansion of the replay experience buffer. This step mainly expands the experience tuple sampled in step P1, replaces the initial target g with the virtual target g′, and recalculates the reward under the virtual target g′ to obtain a new experience tuple, and then puts the new experience tuple into the replay experience buffer. For a trajectory τ randomly sampled in the replay experience buffer, k states on this trajectory and after the current state are randomly selected as new targets relative to the original trajectory. That is, if the current sample is (s t , a t ,s t+1 ), then it will be in s t+1 …s T The target corresponding to k states is selected as the new target. In this case, the number of samples stored in the replay experience buffer is k times the number of samples actually sampled.

[0079] P3: Estimation of the kernel density of the goals that the agent has achieved. This step mainly estimates the set G in the replay experience buffer after expansion in step P2. ag In order to reduce the computational complexity, the present invention uses a simple kernel density estimation algorithm to estimate the data density, thereby obtaining the target set G that the agent has achieved. ag The kernel density estimation model F ag , where the kernel function uses a smoother Gaussian kernel function, as shown below:

[0080]

[0081] Where K(u) represents the kernel function, and u is a parameter set in advance.

[0082] Therefore, for the set G ag Each achieved goal in g i , its density value can be calculated as:

[0083] ρ i =f h (g i )

[0084] where f h (*) is a kernel density estimation function with a Gaussian kernel, and the Gaussian kernel is K(u) mentioned above.

[0085] Next, the density values ​​are normalized as follows:

[0086]

[0087] in, Indicates the goal achieved i The normalized density value of N′ represents the number of achieved goals stored in the current replay experience buffer, that is, the set G ag The size of ρ min It represents the minimum density in the current replay experience buffer. After normalization, the density values ​​of all achieved goals are between [0, 1]. The present invention measures the novelty of the state by estimating the density of the state. The area with lower density means that the agent has rarely explored it, which will have a positive impact on the learning of the agent. There is no need to repeat the learning of the area with higher density. The intrinsic reward can be constructed according to the density value to distinguish the state value, so that the area with lower density rarely explored by the agent has higher value, and the area with lower density often explored by the agent has lower value, which encourages the agent to explore.

[0088] P4: Gap calculation. Since the present invention focuses on the multi-goal guided reinforcement learning algorithm, the initial target state will be set for the agent at the beginning of each interaction round. Therefore, the low-density areas close to the target state are the areas we focus on, which can guide the agent to explore more quickly. Therefore, in each round of interaction, for the set G ag Each state ag i (From the single-step transfer tuple, we can see that when the task is successfully completed, the current state in the last single-step transfer tuple is the state of task completion. At this time, the state ag and g reached by the agent are the same, so in the set G ag The state ag that has been reached in the part is the same as the original target state g), and the gap d between it and the initial expected target state g can be calculated. The gap d is defined as the Euclidean distance:

[0089]

[0090] The smaller the gap, the closer the current state is to the target state, and the state should have a greater value.

[0091] P5: Design of intrinsic rewards. The principle of intrinsic reward design in this invention is that states with low density and close distance to the target state should have larger reward values, and vice versa, states with high density or far distance should have lower reward values. At the same time, it should also be ensured that the intrinsic reward gradually decreases as the action selection strategy model of the training agent proceeds. The intrinsic reward design is as follows:

[0092]

[0093] Here, η is a predefined hyperparameter.

[0094] For the current state t , the intrinsic reward obtained by the agent is designed to be the difference between the inverse of its density value and the minimum density in the current achieved target set. At the same time, a coefficient is introduced, which is the inverse of the difference between the current state and the initial expected target state. This ensures that the states in low-density areas with smaller gaps will receive larger intrinsic rewards. The states in these low-density areas are considered to be more novel than the states in higher-density areas and are worthy of continued learning by the agent.

[0095] Figure 3 For the overall process of the present invention, after the introduction of intrinsic rewards, the agent receives the sum of the intrinsic rewards and the extrinsic rewards at time t. It can encourage intelligent agents to explore the environment efficiently, discover more valuable new states, and expand to the entire state space faster.

[0096] At the same time, the present invention also designs a dynamic initial state selection module. Considering the sparse reward task in reinforcement learning, if the agent always starts from a fixed state within a limited step range, it is sometimes difficult for the agent to access the entire state space. In order to further improve the exploration efficiency, the present invention enables the agent to start from those novel states that have been experienced before and are rarely reached, rather than always starting from a fixed initial state. Since the agent has acquired the ability to reach certain states, it does not have to always start from scratch.

[0097] The initial state of the next interaction round should have the following properties:

[0098] 1) These states are novel compared to the already realized states currently stored in the replay buffer;

[0099] 2) The agent should be able to reach the state under the current action selection strategy.

[0100] Therefore, the present invention selects the initial state according to the above intrinsic reward. First, for each tuple in the replay buffer, the intrinsic reward module estimates the density value of the current state of the agent using the kernel density estimation method and generates the corresponding intrinsic reward. Then, the tuples that successfully reach the target state in the replay experience buffer can be extracted from these tuples. Among the extracted tuples, the state with the largest intrinsic reward is selected as the qualified state. For each state s in these tuples that successfully reach the target state, i′ For example, i′ The probability of being selected is as follows:

[0101]

[0102] Indicates that in state si′ The average delay reward under the condition, N″ is the state s in the tuple that successfully reaches the target state i′ The total quantity.

[0103] In order to avoid overfitting and collapse of the action selection strategy, it is necessary to maintain a balance between selecting a new initial state and keeping the initial state of each interaction round fixed before the training of the action selection strategy model of the training agent. The present invention solves this problem by introducing dynamic ratio replacement. Specifically, a dynamic hyperparameter ratio α is used to determine the state selection. This probability is designed to be related to the real-time success rate of the task, as follows:

[0104] α=γe -2×succsss_rate

[0105] At the beginning of the training of the action selection strategy model of the training agent, the success rate of the task is low. At this time, the dynamic ratio value will be set larger to encourage the agent to replace the initial state, which can speed up the learning of the agent. At the same time, these selected initial states implicitly constitute a series of courses from easy to difficult, guiding the agent to quickly explore the final goal. In the later stage of training, as the action selection strategy of the agent is continuously optimized and the success rate of the task increases, this ratio is set to a smaller value, and it will return to basically using only the initial state of each interaction round fixed before the start of model training as the initial state of each interaction round, thus achieving a balance between exploration and utilization.

[0106] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A method for solving intelligent agent control tasks in a continuous action space, characterized in that: The method comprises the following steps: Step 1: First, use a machine learning existing data operation tool, the variational autoencoder VAE, to embed observations from the external environment, that is, external environment observation data, to model the agent for reinforcement learning, and use the external environment observation data and high-dimensional images or state feature values ​​as input through the encoder network q in the variational autoencoder φ After processing, the variational self-encoder outputs the parameters μ and σ of the multivariate Gaussian distribution, where φ is the encoder network q φ Parameters; The variational autoencoder then samples the latent vector, which is then decoded by the decoder network p ψ Decode the latent vector into the original state space, where ψ is the decoder network p ψ , jointly optimize the parameters φ and ψ of the encoder and decoder to maximize the loss function of the variational autoencoder; Step 2: Sample data collection Following the basic framework of traditional offline reinforcement learning methods and goal-guided reinforcement learning algorithms, the action selection strategy model of the agent is trained. This model represents how the agent facing the continuous action space selects appropriate actions in different environmental states. The agent interacts with the environment multiple times. Before the start of each interaction episode, the target state g and the initial state s0 are determined. Then the agent interacts with the environment and a sample trajectory τ is sampled based on the data obtained from the interaction. The trajectory is in the form of [s0, a0, r0, s1, a1, r1, ..., s T ,a T ,r T ], where T is the time step. After calculating the reward based on the target state, the sample trajectory data obtained from the sampling is combined with the target state g and the state ag reached at the current moment to construct a single-step transfer tuple and convert each single-step transfer tuple (s t ,a t ,r t ,g,s t+1 ,ag) is stored in the replay experience buffer as a sample data; where s t Represents the current state, a t represents the action taken at the current moment, r t Represents the delayed reward obtained after taking action at the current time t. The value is calculated by the reward function. t+1 represents the state at the next moment t+1, ag represents the state reached at the current moment, t=1,2,…,T, the reward function is designed as follows: The reward function follows the standard sparse reward binary reward function design, where is the characteristic representation of the current state at time t, ||*|| represents the operation of taking the absolute value, that is, taking the absolute value of the difference between the two states, δ is the preset threshold, which means that when the difference between the state reached by the agent and the expected target state is within the threshold, the task is considered successful. At this time, the reward function r g (s t ,a t ,g)=0; At the same time, the size of the replay experience buffer is fixed to N, and different values ​​are set according to different tasks of the agent. The idea of ​​queue is used in the design. When the replay experience buffer is full, the newly collected single-step transfer tuple will replace the single-step transfer tuple that enters the buffer earliest, following the first-in-first-out rule. Step 3: Expand the replay experience buffer using the idea of ​​​​post-experience replay: By reselecting several artificially set "virtual targets" g' different from the original target state g on the current trajectory τ, we re-examine this trajectory τ on this basis, use these "virtual targets" and the above reward function to re-give the reward of each single-step transfer tuple in this trajectory τ, and convert these new tuples (s t ,a t ,r t ,g′,s t+1 ,ag) is also stored in the replay experience buffer. The storage rules are the same as step 2. At this time, the replay experience buffer will contain a part of the delayed reward r t Single-step transfer tuples with positive values ​​will play a beneficial role in the learning of the agent; Step 4: Use the kernel density estimation method to estimate the density value of the agent's current realized state and calculate the corresponding intrinsic reward: By introducing intrinsic rewards To expand the reward mechanism, the delayed reward obtained by the agent consists of two parts: in is the intrinsic reward at the current time t, is the external reward at the current moment t, that is, the reward given by the environment. The goal of the agent is to maximize the value of the total delayed reward accumulated to the moment T; At the current time t, store all realized states ag in the replay experience buffer into the set G ag In the example, the traditional kernel density estimation is used to measure the novelty of the state. From the single-step transfer tuple, it can be seen that when the task is successfully completed, the current state in the last single-step transfer tuple is the state of task completion. At this time, the state ag reached by the agent is the same as g. Therefore, in the set G ag The state ag that has been achieved is the same as the original target state g. The density of a certain achieved state is defined as the number of times the achieved state is achieved and the set G ag The quotient of the number of realizations of all realized states in the set G ag The data in is the sample data, and then the kernel density estimation algorithm of the standard Gaussian kernel function is used to estimate the density of each state in the set. This kernel density estimation model is recorded as F ag , for the set G ag Each achieved goal in g i , density ρ i The calculation is as follows: r i =F ag (g i ) Therefore, the intrinsic reward design should be inversely proportional to the density of the current state. A state with a larger density means that it has been visited many times and is no longer desirable, so it should be ignored. A state with a smaller density is more novel to the agent and should be explored in the future. At the same time, considering the existence of the original target state that is expected to be reached at the beginning of training, the state that is closer to the expected target state and has a smaller density should be the focus of attention. Therefore, the kernel density estimation algorithm is used to estimate the current state s t In the collection G ag The density value in, and the gap d between the current state and the desired target state g, the intrinsic reward The design is as follows: Among them, f h (*) is a kernel density estimation function with a Gaussian kernel, which produces a smoothed density estimate of the state *, Indicates f h (*) Take the average value; ag j represents the jth realized state, η is a predefined hyperparameter, Indicates taking The Clip function is used to ensure the difference between the success state and the failure state; at the same time, the smaller the gap d, the greater the value of the intrinsic reward; As the agent's action selection strategy model training continues, the intrinsic reward Will gradually converge to 0; Step 5: Selection of dynamic initial state: In order to further improve the efficiency of exploring the environment, since the agent has acquired some capabilities, let the agent start exploring from the novel states it has experienced before; For each single-step transition tuple in the replay experience buffer, first use the intrinsic reward generated in step 4 Those intrinsic rewards First, select the single-step transition tuples with positive values; second, select the successful single-step transition tuples that reach the target state; finally, select the one with the largest intrinsic reward from these selected tuples, and use the ag in the tuple with the largest intrinsic reward as the initial state of the next interaction episode; Considering the balance between exploration and exploitation, before each interaction round, a number between [0,1] will be randomly selected. When the number is less than α, the novel state will be selected as the initial state of the interaction round. Otherwise, the fixed initial state at the beginning of each interaction round will be selected as the initial state of the interaction round. The ratio α is designed as follows: a=γe -2×succsss_rate Among them, × represents multiplication, γ is a hyperparameter, γ∈(0,1), and success_rate is the task success rate at the current moment; According to the above steps, the intelligent agent facing the continuous action space obtains sample data for training by interacting with the environment. The intelligent agent will automatically learn how to control and complete various tasks assigned by humans to achieve the solution of the intelligent agent control task.

2. The method for solving intelligent agent control tasks in continuous action space according to claim 1, characterized in that: The encoder network in step 1 is a neural network, and its output is a latent vector z, that is, the input is reduced in dimension and needs to obey the Gaussian distribution. The process of the encoder network outputting z is actually decomposed into two steps: 1) First, the encoder network outputs the parameter mean μ and variance σ of the Gaussian distribution. This parameter is different for each data point of the external environment observation data; 2) Fusing the noise with the Gaussian distribution, and sampling z from the Gaussian distribution fused with the noise; The decoder network is also a neural network, whose input is the latent vector z output by the encoder network, and whose output is the probability distribution of the data obtained by decoding the latent vector z into the original state space; The variational autoencoder consists of an encoder network and a decoder network. The input of the variational autoencoder is encoded by the encoder network, and the encoded result is decoded by the decoder network. The loss function of the variational autoencoder is a negative log-likelihood function with a regular term. Because there is no shared latent vector between all data points of the variational autoencoder's input data, the loss of each data point is l m are independent, the total loss is the sum of the losses of each data point, M represents the total number of data points, and data point x i The loss i It is expressed as: Among them, s (i) is the input data point x i The state of the agent at this time, β is a hyperparameter, and the first term D KL (q φ (z|s (i) )∥p(z)) is the regularizer, i.e. the encoder distribution q φ (z|s (i) ) and p(z). This parameter measures the discreteness, that is, how much information is lost in the data obtained by encoding the original data through the encoder and decoding it through the decoder compared with the original data; at the same time, p(z) represents a priori specified standard normal distribution, that is: p(z) = N(0,1), where N(0,1) represents a normal distribution with an expected value of 0 and a standard deviation of 1. is the expected negative log-likelihood-reconstruction loss, where p ψ (s (i) |z) represents the decoder distribution, Indicates that the encoder distribution is q φ (z|s (i) ) is used to obtain the expectation under the condition that the expectation is the distribution of the result obtained after the data is input into the encoder network for encoding.

3. The method for solving intelligent agent control tasks in continuous action space according to claim 2, characterized in that: The step 2 specifically includes: P1: Using the idea of ​​offline strategy algorithm DDPG, the agent interacts with the environment, generates sample data, and collects the sample data. The replay experience buffer contains several experience tuples (s t ,a t ,r t ,g,s t+1 ,ag), the achieved target set G ag , the desired target set G dg , where the action selection strategy is as follows, and H is the added noise: a t =π(s t ,g)+H Among them, π(s t ,g) means that in the action selection strategy π, when the parameters are state s t , the action probability distribution when the target is g; P2: Virtual target sampling, expansion of the replay experience buffer This step mainly expands the experience tuple sampled in step P1, replaces the initial target state g with the virtual target g′, and recalculates the reward under the virtual target g′ to obtain a new experience tuple, and then puts the new experience tuple into the replay experience buffer. For a trajectory τ randomly sampled in the replay experience buffer, k states on this trajectory and after the current state are randomly selected as new targets relative to the original trajectory. That is, if the current sample is (s t ,α t ,s t+1 ), then it will be in s t+1 …s T The target corresponding to k states is selected as the new target. In this case, the number of samples stored in the replay experience buffer is k times the number of samples actually sampled. P3: Kernel density estimation of the goals that the agent has achieved This step mainly performs the operation on the set G in the replay experience buffer after expansion in step P2. ag To reduce the computational complexity, a simple kernel density estimation algorithm is used to estimate the data density, thereby obtaining a kernel density estimation model F of the goal that the agent has achieved. ag , where the kernel function uses a smoother Gaussian kernel function, as shown below: Where K(u) represents the kernel function, and u is a parameter set in advance; Therefore, for the set G ag Each achieved goal in g i , its density value is calculated as: ρi=fh(gi) where f h (*) is the kernel density estimation function with a Gaussian kernel, and the Gaussian kernel is the above k(u); Next, the density values ​​are normalized as follows: in, Indicates the goal achieved i The normalized density value of N′ represents the number of achieved goals stored in the current replay experience buffer, that is, the set G ag The size of ρ min It represents the minimum value of the density in the current replay experience buffer. After normalization, the density values ​​of all achieved targets are between [0,1]; P4: Gap calculation At the beginning of each interaction round, the initial target state is set for the agent. Therefore, the low-density areas close to the target state are the areas that need to be focused on, which can guide the agent to explore more quickly. Therefore, in each round of interaction, for the set G ag Each state ag i , calculate the gap d between it and the initial desired target state g, the gap d is defined as the Euclidean distance: The smaller the gap, the closer the current state is to the target state, and the state should have greater value; P5: Design of Intrinsic Rewards The principle of intrinsic reward design is that the state with lower density and closer distance to the target state should have a larger reward value. Conversely, the state with higher density or farther distance should have a lower reward value. At the same time, it should also ensure that the intrinsic reward gradually decreases as the action selection strategy model training of the intelligent agent progresses.

4. The method for solving intelligent agent control tasks in continuous action space according to claim 3, characterized in that: Each state s in the successful single-step transition tuple that reaches the target state in step 5 i′ For example, i′ The probability of being selected is as follows: Indicates that in state s i′ The average delay reward under the condition, N″ is the state s in the tuple that successfully reaches the target state i′ The total quantity.

Images