Unmanned vehicle reinforcement learning method based on random network distillation exploration optimization

Abstract

The invention discloses an unmanned vehicle reinforcement learning method based on random network distillation exploration optimization, and the method obtains an internal reward through calculating an error between a prediction network and a target network, and meets the exploration demands of an intelligent agent in sparse rewards. Based on a fixed target network and a prediction network updated by using stochastic gradient descent, an internal reward is generated, and the internal reward is normalized in real time by using forward filtering and a dynamic normalization algorithm, so that the internal reward and an environment reward belong to the same order of magnitude, the stability of the internal reward is improved, and the prediction efficiency is improved. Therefore, the stability and generalization ability of the algorithm are improved, and the intelligent agent is promoted to better explore in the sparse reward environment. According to the method, the output error of the target network and the prediction network is used as an internal reward, so that the exploration capability of the intelligent agent is enhanced, and the environment exploration of the intelligent agent can be better promoted.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning, and in particular to a deep reinforcement learning method based on intrinsic reward to promote exploration. Background Art

[0002] Reinforcement learning (RL) methods work by maximizing the expected reward of a strategy. For example, in Atari games, the RL agent maximizes the game score according to the rules of the game to complete the game. When there are dense rewards in the environment and the agent can easily find these rewards by taking a random sequence of actions, it is easy for the RL agent to obtain rewards, thereby learning how to maximize the reward value within a certain period of time and finally find the optimal solution to the problem. But when rewards are sparse and difficult to find, it is difficult for the RL agent to learn by collecting rewards in the environment. In reality, it is usually impractical to design a dense reward function for each task that needs to be solved by the RL agent. In these cases, it is necessary to use a method that explores the environment in a directed manner.

[0003] There is a lot of research on the exploration strategy of reinforcement learning algorithms at home and abroad. In 2020, Michael Gimelfar proposed an ε-BMC (Epsilon-Bayesian Model Combination) algorithm to adaptively adjust the ε parameter in the ε-greedy exploration strategy in reinforcement learning. Experimental results show that ε-BMC effectively balances exploration and exploitation on different problems, and its performance is comparable or better than the best fixed annealing plan and other data-driven ε adaptation schemes. However, in this algorithm, ε is modeled based on the Beta distribution, and the algorithm is sensitive to the choice of prior parameters, so the agent may still not be able to fully explore. Through the study and design of the reward function, the agent can better obtain rewards and maximize the report, thereby solving this problem. For example, in 2021, Jiexin Wang et al. proposed a modular deep reinforcement learning method. By combining the value function of rewards and penalties, an automatic weight adjustment scheme was designed, and a sensor fusion network was introduced, which significantly improved the learning efficiency and safety of robot navigation tasks and performed well in a variety of maze environments. However, this method requires a sensor fusion network structure that combines lidar and image inputs. The system complexity is high, which may lead to increased computing resource requirements. In 2023, Jorge Ramirez et al. proposed an improved reinforcement learning from expert demonstrations (RLED) method, which can not only accelerate the learning process, but also find behaviors similar to expert teachers in high-dimensional or continuous space. In addition, this method has been successfully applied to redundant robot control, demonstrating the ability to accelerate the RL process by biasing the exploration of demonstration states and actions. The Chinese patent number CN202411251715.4 proposes a multi-platform collaborative behavior decision-making method and system based on deep reinforcement learning, in which an expert reward mechanism is used when building the reward function, and the platform is given specific expert knowledge before it starts learning to assist reinforcement learning. The above method optimizes the balance between utilization and exploration in the exploration of intelligent agents by setting exploration parameters or designing rewards. However, for environments with sparse rewards, existing algorithms cannot enable intelligent agents to explore the environment efficiently. Summary of the invention

[0004] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose a reinforcement learning method for unmanned vehicles based on random network distillation exploration optimization. The method is based on the unmanned vehicle reinforcement learning method for random network distillation (RND) exploration optimization, and uses the advantages of RND to use intrinsic rewards to encourage the agent to explore the environment, optimize the exploration performance of the agent in a sparse reward environment, and thus solve the problem that the agent is difficult to efficiently explore in an environment with sparse rewards. The present invention uses a method with low overhead and easy implementation to promote the exploration of the agent, uses the output error of the prediction network and the target network as the intrinsic reward, and combines the intrinsic reward with the reward obtained by the DQN agent from the environment, so that it can be fully explored in the learning process, thereby obtaining a more efficient strategy.

[0005] The technical solution adopted by the present invention to solve the technical problem is: an unmanned vehicle reinforcement learning method based on random network distillation exploration optimization, the method comprising the following steps:

[0006] Step 1: Initialize and deploy an unmanned vehicle based on the value RL algorithm. The unmanned vehicle is equipped with a pair of random networks, both of which are fully connected layer networks. This pair of random networks includes a fixed target network F and a prediction network that is updated using the stochastic gradient descent method. Both networks consist of three linear fully connected layers, and the input dimension of the network is D in , the output dimension is D out , the two networks have the same structure. The input is the observation result o, where o is the coordinates and speed of the unmanned vehicle. It represents the network's observation of the environment; the output is α, which is dimensional real vector, which has no physical meaning and is only used to calculate the output error of the two networks.

[0007] Step 2: Set the number of interactions N and the number of optimization steps N after each interaction opt , the length of the interaction T, the number of loop steps M to initialize β, where β is the observation specification parameter. The total number of time steps k (0≤k≤K) of global training is 0 at the beginning.

[0008] Step 3: Initialize the environment state s0, which follows the distribution of the initial state p(s0).

[0009] Step 4: Perform M steps of random exploration and initialize observation specification parameters. For the mth (1≤m≤M) step, the unmanned vehicle randomly selects an action a from the defined action set. (k) , that is, when selecting an action a (k) ~U(a (k)), the action set includes forward, backward, left, and right. Then observe the state s of the next time step (k+1) ,s (k+1) The transition probability distribution p(s (k+1) |s (k) ,a (k) ).

[0010] Step 5: Perform T-step interactions between the unmanned vehicle and the environment. For the t-th (1≤t≤T) interaction segment, first use the strategy π(a (k) |s (k) ) to get a (k) ,π(a (k) |s (k) ) means in state s (k) Take action a (k) The probability of , which is given by the agent's policy network. Then according to the joint probability distribution p(s (k+1) ,e (k) |s (k) ,a (k) ) Extract the next state s (k+1) and the reward e given by the environment at the current time step (k) Finally, calculate the intrinsic reward e i (k) , the calculation method is:

[0011]

[0012] Here θ is the prediction network Parameters.

[0013] Step 6: Use s (k) ,s (k+1) , a (k) , e (k) ,e i (k) Together they form the i-th optimization batch B i .

[0014] Step 7: Repeat steps 5-6, increasing t each time until it equals T.

[0015] Step 8: When k is greater than After that, every time k increases by 5, the intrinsic rewards in the optimization batch are normalized. During normalization, each intrinsic reward e in the reward buffer is first normalized. i (k) Perform forward filtering in a recursive manner, and the input of the filter is e i (k) , the output is the cumulative discounted reward I at time step k (k). Calculate the cumulative discounted reward from time step 0 to the current time step k. The specific method is as follows:

[0016]

[0017] Where γ is the discount factor, I (0) is initialized to 0. After forward filtering the intrinsic reward, the amount of data is The mean and standard deviation of the small batch intrinsic reward approximate the mean and standard deviation of the entire data with a data volume of δ, and the intrinsic reward is normalized. First, calculate the mean of the small batch intrinsic reward and variance s 2 , and then use the following formula to dynamically approximate the mean and standard deviation of all data:

[0018]

[0019]

[0020] Here μ and σ 2 They are the dynamic mean and variance calculated last time.

[0021] Then each intrinsic reward is normalized:

[0022]

[0023] Step 9: Perform N on the prediction network and DQN’s policy network opt Second optimization, firstly, according to the loss function of the reinforcement learning agent, the parameters θ of the policy network are π For optimization, the loss function is the temporal difference (TD) loss function:

[0024]

[0025] Q * (s (k) ,a (k) ) is the state s at time step k (k) and action a (k) The optimal Q value, r (k) is the immediate reward obtained at time step k, and is e (k) and e i (k) The sum of γ and maxQ is the discount factor in the temporal difference algorithm, which is used to measure the importance of future rewards relative to immediate rewards. * (s (k+1) ,A) is the maximum Q value of all possible actions at time step k+1, and A is the set of all actions.

[0026] Then the distillation loss is used to predict the parameters of the network To optimize:

[0027]

[0028] The Adam algorithm is used to optimize the parameters of the prediction network and the policy network for gradient descent, and the optimization goal is to minimize the loss function.

[0029] Step 10: Repeat steps 5-9 N times.

[0030] Beneficial effects:

[0031] 1. The present invention utilizes the output errors of the target network and the prediction network as intrinsic rewards, which enhances the exploration ability of the intelligent agent, thereby better promoting the intelligent agent to explore the environment.

[0032] 2. In the present invention, only one pair of neural networks is used as the generator of intrinsic rewards, so the computational cost of the method is low and the implementation is simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a schematic diagram of simulating the movement of an unmanned vehicle in a two-dimensional space in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The invention will be further described in detail below in conjunction with the accompanying drawings.

[0035] like Figure 1 As shown, the environment of the present invention simulates the movement of an unmanned vehicle in a two-dimensional space. The upper left corner represents the initial position of the unmanned vehicle, that is, the starting point. The yellow circle is the target position of the unmanned vehicle, that is, the end point. In the space, there are green rewards and black obstacles. The goal of the unmanned vehicle is to get more rewards and reach the end point while colliding with as few obstacles as possible. The position of the unmanned vehicle is represented by two-dimensional coordinates. The unmanned vehicle has 6 actions, which are to move one unit to the left, right, forward, and backward.

[0036] Step 1: Deploy an unmanned vehicle in the drone system. The unmanned vehicle agent uses the Deep QNetwork algorithm and configures a pair of neural networks for the agent: a fixed target network F and a prediction network updated using stochastic gradient descent. The two networks have the same structure, both consisting of three linear layers, using the Leaky_Relu activation function. The formula of the Leaky_Relu activation function is:

[0037]

[0038] Step 2: Set the number of interactions to N = 2000, and the number of optimization steps to be performed after each interaction to N opt = 10, the length of the interaction T = 50, the number of loop steps for initializing β M = 200, where β is the observation specification parameter. The total number of time steps k for global training is 0 at the beginning.

[0039] Step 3: Initialize the environment state s0.

[0040] Step 4: Perform 200 steps of random exploration and initialize observation specification parameters. For the mth (1≤m≤M) step, the unmanned vehicle randomly selects an action, such as moving forward. Then observe the state s of the next time step (k+1) ,s (k+1) is the 2D coordinate position of the unmanned vehicle at time step k+1. Finally, add one to k to complete the exploration of the current step.

[0041] Step 5: Next, the unmanned vehicle interacts with the environment. The interaction is T steps. For the tth (1≤t≤T)th interaction segment, firstly, according to the strategy π(a (k) |s (k) ) to get a t ,π(a (k) |s (k) ) means in state s (k) Take action a (k) Then according to the joint probability distribution p(s (k+1) ,e (k) |s (k) ,a (k) ) Extract the next state s (k+1) and the reward e given by the environment at the current time step (k) Finally, the intrinsic reward is calculated according to Formula 1.

[0042] Step 6: Use s (k) ,s (k+1) , a (k) , e (k) ,e i (k) Together they form the i-th optimization batch B i .

[0043] Step 7: Repeat steps 5-6, increasing t each time until it equals T.

[0044] Step 8: When k is greater than 200, the intrinsic rewards in the optimization batch are normalized every time k increases by 5. During normalization, each intrinsic reward e in the reward buffer is first normalized. i (k)Forward filtering is performed recursively, and the cumulative discounted reward from time step 0 to the current time step k is calculated by equation 2, where γ = 0.99. After forward filtering the intrinsic reward, the mean and standard deviation of all data with a data volume of n are approximated by equations 3 and 4, and the intrinsic reward is normalized by equation 5.

[0045] Step 9: Perform N on the prediction network and DQN’s policy network opt In the second optimization, we first optimize the policy network parameters θ according to the loss function of formula 6. π Then use formula (1.7) to predict the parameters of the network For optimization, the Adam algorithm is used for gradient descent for the optimization of the prediction network and policy network parameters.

[0046] Step 10: Repeat steps 5-9 N times.

[0047] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. An unmanned vehicle reinforcement learning method based on random network distillation exploration optimization, characterized in that: The following steps are involved: Step 1: Initialize the unmanned vehicle that deploys the value-based reinforcement learning algorithm, which is equipped with a fixed target network F and a prediction network The purpose is to generate intrinsic rewards based on the prediction errors of the two networks; Step 2: Set the number of interactions N and the number of optimization steps N after each interaction opt , the length of the interaction T, the number of loop steps M to initialize β, where β is the observation specification parameter, and the total time step k (0≤k≤K) of global training is 0 at the beginning; Step 3: Initialize the environment state s0, which follows the distribution of the initial state p(s0); Step 4: Perform M steps of random exploration, initialize observation specification parameters, and for the mth (1≤m≤M) step, the unmanned vehicle randomly selects an action a from the defined action set (k) , and then observe the state s of the next time step (k+1) ; Step 5: Perform T-step interactions between the unmanned vehicle and the environment. For the t-th (1≤t≤T) interaction segment, first use the strategy π(a (k) |s (k) ) to get a (k) , and then according to the joint probability distribution p(s (k+1) ,e (k) |s (k) ,a (k) ) Extract the next state s (k+1) and the reward e given by the environment at the current time step (k) , and finally calculate the intrinsic reward e i (k) ; Step 6: Use s (k) ,s (k+1) , a (k) , e (k) ,e i (k) Together they form the i-th optimization batch B i ; Step 7: Repeat steps 5-6, increasing t each time until it is equal to T; Step 8: When k is greater than After that, k increases Normalize the intrinsic rewards in the optimization batch once; Step 9: Perform N on the prediction network and DQN’s policy network opt Second optimization, firstly, according to the loss function of the reinforcement learning agent, the parameters θ of the policy network are π Optimize and then use distillation loss to predict the parameters of the network Optimize Step 10: Repeat steps 5-9 N times.

2. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that The present invention adds a target network and a prediction network to the deep reinforcement learning method to generate prediction errors and generate intrinsic rewards, thereby promoting intelligent agent exploration.

3. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that In step 5, the prediction error between the target network and the prediction network is used as the intrinsic reward. The calculation formula of the intrinsic reward is:

4. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that In step 8, for each intrinsic reward e in the reward buffer i (k) Perform forward filtering in a recursive manner, calculating the cumulative reward I during filtering (k) The method is:

5. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that In step 8, the dynamic mean and standard deviation are calculated in real time before normalizing the intrinsic reward. The formula for calculating the dynamic mean and standard deviation is:

6. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that In step 8, the intrinsic reward is normalized. The formula for normalizing the intrinsic reward is:

7. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that In step 9, the agent updates the policy network through the stochastic gradient descent algorithm to minimize the policy network loss function, which is:

8. According to the unmanned vehicle reinforcement learning method based on random network distillation exploration optimization as described in claim 1, it is characterized in that In step 9, the agent updates the prediction network through the stochastic gradient descent algorithm to minimize the prediction network loss function. The policy network loss function is:

Images