A method for generating intrinsic rewards for perceiving action loop decisions

Abstract

The present application belongs to the technical field of computer application, and particularly relates to a kind of internal reward generation method of perceiving action loop decision-making.The present application, first, builds state prediction network, and collects the environment state and agent action data in the process of agent training to predict modeling environment state change.Second, in the process of reinforcement learning training, the trained state prediction network model is used to predict environment state change and generate prediction difference intrinsic reward.Finally, the generated prediction difference intrinsic reward is combined with external reward to form a new reward for the policy update network to use.The present application constructs SGBRT network to predict environment state and generate intrinsic reward, and uses intrinsic reward to speed up the training process of policy update network and improve the training effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer application, and particularly relates to a method for generating internal reward of decision of a perception-action loop. BACKGROUND

[0002] When a human being performs a process task such as driving or fighting, a perception-action loop is usually formed, that is, perceiving a task environment, deciding an action to be taken, taking the action, changing the task environment, re-perceiving, until the task is completed or the task fails. With the rapid development of artificial intelligence in recent years, it has become a social demand to use artificial intelligence to replace human intelligence to complete process tasks in a real environment. In order to achieve this goal, artificial intelligence needs to complete two tasks, environment state perception and behavior decision. For behavior decision, artificial intelligence needs to use the perceived environment state to make a specific action decision and output the action to the environment to complete a loop in the perception-action loop, and the cycle continues until the process task is completed. The reinforcement learning method can solve the task of using the perceived state to make a behavior decision in the perception-action loop.

[0003] Reinforcement learning is an important branch of machine learning, which has two important components: environment and agent, and three main data in the agent training process: state, action and reward. The agent is the main body of reinforcement learning training and the main body of decision making, and the environment is a collection of external information of the agent involved in the task to be completed. The agent obtains available information from the environment as the state, makes an action according to the state and the agent's strategy, and obtains a reward according to the current state and the action made by the agent using the reward function designed for the environment, which can also be called external reward. The training process of reinforcement learning is a process of continuously updating the agent's strategy to maximize the external reward.

[0004] However, there is an important problem to be solved in reinforcement learning for perceptual behavior decision, which is the setting of reward function. The training goal of reinforcement learning is to find a policy for the agent that can effectively complete the task, and the policy update completely relies on the reward generated by the reward function for the agent. Therefore, the setting of the reward function needs to be consistent with the task in order to efficiently train a good policy. The reinforcement learning for perceptual behavior decision mostly uses a real environment, and for a real environment, the perceived state is generally complex, and it is also difficult to determine which behaviors in these complex states are beneficial to the completion of the overall task. Therefore, it is very difficult to design a dense reward function, and if only behaviors that are obviously beneficial to the task are rewarded, it will cause the reward to be sparse and make the training difficult. For similar problems, Pathak et al. designed a predictive difference intrinsic reward mechanism with an inverse environment model to accelerate the exploration of the agent in the environment; Bellemare et al. designed a virtual counting method to measure the novelty of the state; Florsensa et al. added auxiliary tasks to gradually increase the difficulty of the task to accelerate the training. However, due to the complexity of the state characteristics of some environments, the above methods are difficult to effectively predict the state changes of the environment, thereby slowing down the exploration of the agent in the environment. At the same time, when the state of the real environment has inherent randomness and cannot be predicted, using intrinsic rewards in this situation may cause the agent to perform useless exploration and circle in a series of purely random states. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, for the real environment with complex and unpredictable situations that need to be perceived, an intrinsic reward generation method for perceptual action loop decision is designed to assist in completing the task of the behavior decision part in the perceptual action loop. First, a state prediction network is built, and the environment state feature and agent action data during the agent training process are collected to model the state change of the environment. Second, the trained state prediction network model is used to predict the state change of the environment and generate a predictive difference intrinsic reward during the reinforcement learning training process of the behavior decision. Finally, the generated predictive difference intrinsic reward is combined with the external reward to form a new reward for the policy update network of the behavior decision. The SGBRT network is constructed to predict the state of the environment and generate an intrinsic reward, and the intrinsic reward is used to accelerate the training process of the policy update network and improve the training effect.

[0006] The present application is roughly divided into two parts:

[0007] (1) A state prediction network is constructed and intrinsic rewards are generated. Training process data for reinforcement learning of behavioral decisions is collected, including environmental state information perceived during the state perception process and actions taken by the reinforcement learning agent during the behavioral decision-making process. Environmental state information is predicted through supervised learning (GBRT regression tree model), and unsupervised learning (SOM clustering network) is combined to eliminate noise that may exist due to the inherent randomness of the perceived state during the prediction process. This network (referred to as SGBRT) is mainly composed of SOM clustering network and GBRT regression tree model. Using SOM for clustering can integrate similar state features and avoid the agent from making ineffective explorations due to the randomness of the environment itself, while GBRT can quickly and effectively predict the environmental state. The trained SGBRT network is introduced into the reinforcement learning training process to calculate the prediction error.

[0008] (2) The prediction error obtained by the pre-trained SGBRT network is used as the intrinsic reward to encourage the agent to effectively explore the environment, increase the likelihood of the agent obtaining external rewards, and accelerate training. The intrinsic and external rewards are standardized and fused in a certain proportion to ensure that the agent does not forget the original training objective while exploring effectively. The combined reward is used to replace the original pure external reward as the reward function used by the policy update network.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] An intrinsic reward generation method for decision-making in a perception-action loop, comprising the following steps:

[0011] Step (1) State prediction network construction and training

[0012] (1.1) First, the perceived environmental state s at each step is collected from the reinforcement learning process of training the behavior decision-making strategy update algorithm in N steps. i and the actions of the intelligent agent making the decision. i Combine them to get x i =[s i ,a i The SOM network has m sets of competing layer neuron weights, each with a dimension of D. During initialization, each neuron in the competing layer is given a small random initial value. Then, when the t-th sample x... t When used to train a network, the neuron c that is most similar to the sample is first selected from the competitive layer, as shown in the following formula:

[0013]

[0014] Where w is the weight of the neurons in the competition layer, w cis the weight of the selected competitive layer neuron. Equation (1) uses the Euclidean distance as the basis for judging the similarity between the neuron and the sample. After selecting the neuron, it is necessary to update the weights of the neuron and its surrounding neurons. The formula is as follows:

[0015] w t+1 = w t + s ct (x t - w t ) (2)

[0016] where w t and w t+1 are the weights of the competitive layer neurons at the t-th step and the upcoming update. 0 < s ct < 1 is a self-update control variable used to control the update amplitude of the neuron weights. Its determination method is as follows:

[0017]

[0018] where 0 < s0 < 1 is a hyperparameter that overall controls the weight update amplitude; σ t is used to control the weight update radius. As the SOM network training progresses, σ t will become smaller and smaller, and the weight update radius will also become smaller. The formula is as follows:

[0019]

[0020] where σ0 and τ are hyperparameters. σ0 is the initial value and also the maximum value of the weight update radius, and τ is used to control the speed at which the weight update radius shrinks. Equations (3) and (4) show that the neurons closer to the selected neuron c have a greater weight update degree, and as the training progresses, the weight update radius of the neurons gradually decreases.

[0021] (1.2) After performing the SOM clustering operation, a total of m clustering clusters are obtained. First, it is necessary to calculate the target vector for each clustering cluster. For the j-th clustering cluster h j calculate its target vector y j The formula is as follows:

[0022]

[0023] Using the clustering cluster target average as the target vector for training the prediction network helps to reduce the prediction error unrelated to exploration caused by the uncertainty of the environment itself and reduce the possibility of the agent's ineffective repeated exploration.

[0024] Subsequently, taking the weight information w of the competitive layer of the SOM network as the input, train the GBRT network. The prediction formula of the GBRT network is as follows:

[0025]

[0026] Where w j For cluster h j The corresponding SOM competition layer neuron weights, For the predicted cluster h j The target vector, f k Let Γ be a subtree in the GBRT network. The GBRT network consists of K subtrees, and Γ is the set of all subtrees. By training and ensembling each subtree, the GBRT network greatly reduces the instability of regression subtrees. The GBRT network adds subtrees to the set one by one, and the training of each subtree is aimed at minimizing the loss of the previous set. The formula is as follows:

[0027] F n (w)=F n-1 (w)+argminL(y,F n-1 (w)+f n (w)) (7)

[0028] Where F n-1 For subtree f n Add to the previous tree set, F n f n After adding the tree set, the GBRT network can be trained by inverse gradient updates through the introduction of an arbitrary loss function L. Through the above operations, the SGBRT network is obtained by training GBRT using the cluster weights of the SOM and the target vector as input.

[0029] Since SGBRT network training takes a long time, to prevent the network training from becoming a bottleneck in the overall reinforcement learning task, SGBRT network training and action decision reinforcement learning training are carried out in parallel. The prediction in reinforcement learning training is based on the pre-trained SGBRT network. While reinforcement learning is being trained, the SGBRT network also collects samples from the reinforcement learning training process data to train a new model. After training is completed, the new model replaces the pre-trained model in the reinforcement learning network, and samples are collected again.

[0030] Step (2) Intrinsic Reward Generation and Reward Combination

[0031] (2.1) In the k-th step of reinforcement learning training, its perceptual state and agent decision action are s, respectively. k and a k Inputting it into a pre-trained state prediction network yields the predicted state features. The prediction error at step k can then be calculated as the intrinsic reward. The formula is as follows:

[0032]

[0033] The intrinsic reward will be combined with the simple external reward generated by the reward function attached to the environment or designed by itself to encourage the agent to effectively explore the applicable environment of the perception-action loop.

[0034] (2.2) After obtaining the intrinsic reward obtained in step (2.1), the intrinsic reward and the external reward need to be combined for reinforcement learning training. However, since the sources of the internal reward and the external reward are different, the reward r needs to be standardized before the rewards are combined, and the formula is as follows:

[0035]

[0036] Where mean and std are the mean and standard deviation of all the rewards obtained, and the two values are dynamically calculated during the training process. The intrinsic reward r i and the external reward r e are calculated by formula (9) above, and the scales of the two rewards are similar, and and are obtained.

[0037]

[0038] Where 0≤ε≤1 is an adjustable hyperparameter for adjusting the role of the intrinsic reward in training, which specifically means that the larger the ε, the more the agent tends to optimize the strategy according to experience, and the smaller the ε, the more the agent tends to explore the environment. Subsequently, the combined reward can be input into the behavior decision task strategy update network for reinforcement learning strategy training. Since the results of the present application are coupled with the strategy update algorithm of reinforcement learning to a low degree, the use of different strategy update algorithms does not affect the use and final effect of the present application.

[0039] The beneficial effects of the present application are:

[0040] The purpose of the present application is to generate intrinsic rewards for perception-action loop reinforcement learning tasks to address the problem of complex and variable environmental state features, which are difficult to design reward functions that meet task objectives and are dense. Through the SGBRT network, the prediction ability of the environment state can be improved while reducing the invalid exploration of the agent caused by the unpredictability of the environment, generating high-quality intrinsic rewards to promote the agent to explore the environment state, accelerating the behavior decision reinforcement learning training process, and improving the task effect of the final obtained strategy network. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The overall framework for generating intrinsic rewards of the present application.

[0042] Figure 2 Fig. 1 is a schematic diagram of a state prediction network structure of the present application.

[0043] Fig. 3(a) is a flow chart of the task training process of the reinforcement learning part of the present application.

[0044] Fig. 3(b) is a flow chart of the training process of the SGBRT prediction network of the present application. DETAILED DESCRIPTION

[0045] The specific embodiments of the present application are further described below in conjunction with the accompanying drawings and technical solutions.

[0046] The present application can be used for the intrinsic reward generation of the reinforcement learning of the behavior decision task in the perception behavior loop, and the overall framework is as shown in Fig. 1, the structure of the state prediction network is as shown in Fig. 2, the overall workflow of the intrinsic reward generation and the additional policy training is as shown in Fig. 3(a) and Fig. 3(b), and the specific embodiments of the present application are described below in conjunction with an example. Figure 1 Figure 2 The present application can be used for the intrinsic reward generation of the reinforcement learning of the behavior decision task in the perception behavior loop, and the overall framework is as shown in Fig. 1, the structure of the state prediction network is as shown in Fig. 2, the overall workflow of the intrinsic reward generation and the additional policy training is as shown in Fig. 3(a) and Fig. 3(b), and the specific embodiments of the present application are described below in conjunction with an example.

[0047] The present embodiment scenario is the Atari video game pass strategy generation, and the specific implementation scheme discussed is only used for describing the implementation mode of the present application, and does not limit the scope of the present application.

[0048] The specific implementation scheme of the present application is described in detail below mainly for the Atari video game task strategy generation, which is mainly divided into two parts of the intrinsic reward dynamic generation and the agent policy update optimization and the state feature prediction network training, which is described in conjunction with the processes of Fig. 3(a) and Fig. 3(b):

[0049] (1) Intrinsic reward dynamic generation and agent policy update optimization:

[0050] ​The environment used in this example is the Atari video game environment of OpenAI Gym. Since the present application only involves the behavior decision part of the perception behavior loop, the simplest perception is performed on this environment, that is, the game screen is directly extracted as the state of the environment in the form of a 210x160x3 tensor. First, the state in the environment is feature extracted and dimensionally reduced. Since the state representation and feature extraction method of different application environments are different, the present application does not involve related content. In this example, a convolutional neural network with fixed random parameters is used to extract features from the image state. After the reinforcement learning network starts step-by-step training and obtains the state feature of the environment at one step and the action selected by the agent according to the strategy, it is necessary to determine whether the training sample queue of the prediction network reaches the maximum capacity. If not, the training data of this step is stored in the sample queue, which will be used to train the prediction network. Then the training data is input into the pre-trained prediction network, the competitive neurons of the SOM network are selected based on formula (1), and the weights of the selected prediction neurons are used as input to make predictions based on formula (6). Then the obtained results are used to calculate the intrinsic reward based on formula (8). Then take the external reward obtained from the environment, combine the intrinsic reward with the external reward based on formulas (9) and (10) to obtain the total reward. Finally, the obtained total reward is input into the policy update model to update the agent's policy. The present application is weakly coupled with the policy update algorithm and does not limit the selection of the policy update model. In this example, the nearest neighbor policy optimization model is used. Before reaching the pre-determined number of reinforcement learning steps, the above content is repeated.

[0051] (2) State feature prediction network training:

[0052] Before training the state feature prediction network, it is necessary to detect whether the number of samples in the network training sample queue reaches the minimum value required for training the prediction network. If the minimum value is reached, training begins. First, a copy of the sample queue is obtained and emptied to add new samples to the sample queue. Second, using the data in the sample queue, the SOM network is trained based on formulas (2), (3) and (4), and the target vector for each neuron in the SOM network is calculated based on formula (5). Then, the competitive neuron weights of the SOM network and the target vector are used to train the GBRT based on formula (7). Finally, the pre-trained network model in the first part is updated and it is re-determined whether the number of samples in the network sample queue reaches the minimum value, and then the new prediction network is re-trained.

[0053] After the above steps, the intrinsic reward can be generated and combined with the external reward generated by the reward function attached by the environment to form the total reward. Then the combined total reward can be brought into the reinforcement learning policy update algorithm to update the agent's policy to achieve the training goal of the reinforcement learning task.

Claims

1. A method for generating intrinsic rewards in a perception-action loop decision-making process, characterized in that, The steps are as follows: Step (1) State prediction network construction and training (1.1) First, the perceived environmental state at each step is collected from the reinforcement learning process of training the behavior decision-making strategy update algorithm in N steps. and the actions of intelligent agents , combined to obtain And let its dimension be D; when the number of clusters is m, the SOM network has m groups of competing layer neuron weights, and the dimension of each group of weights is D; during initialization, each neuron in the competing layer is given a random initial value; then when the t-th sample... When used to train a network, the neuron c that is most similar to the sample is first selected from the competitive layer, as shown in the following formula: （1）； in The weights of neurons in the competing layer. The weights of the selected competing layer neurons are given; Formula (1) uses Euclidean distance as the basis for judging the similarity between neurons and samples; After selecting neurons, the weights of the neurons and their surrounding neurons need to be updated, as shown in the following formula: （2）； in and Let t be the weights of the competing layer neurons in step t and the one about to be updated, 0 < <1 is a self-updating control variable used to control the update magnitude of neuron weights, and is determined as follows: （3）； Where, 0 < <1 is a hyperparameter that controls the overall weight update magnitude; Used to control the weight update radius, as the SOM training network progresses, The radius of the weight update will decrease as it gets smaller, as shown in the formula below: （4）； in and For hyperparameters, The initial value for updating the radius for the weights is also the maximum value. It is used to control the speed at which the weight update radius shrinks; Equations (3) and (4) show that the closer the neuron is to the selected neuron c, the greater the degree of weight update, and the neuron weight update radius gradually decreases as training progresses; (1.2) After performing the SOM clustering operation, a total of m clusters are obtained. First, it is necessary to calculate the target vector for each cluster. For the j-th cluster... Calculate its target vector The formula is as follows: （5）； Use the cluster target mean as the target vector for training the prediction network; Subsequently, the weight information of the competition layer of the SOM network is... The GBRT network is trained using this as input; the prediction formula for the GBRT network is as follows: （6）； in For clusters The corresponding SOM competition layer neuron weights, For the predicted clusters The target vector, This is a subtree in the GBRT network. The GBRT network consists of K subtrees. It is the set of all subtrees. The GBRT network trains and integrates each subtree. The GBRT network adds subtrees to the set one by one, and trains each subtree to minimize the loss of the previous set, as shown in the following formula: （7）； in For subtree Add to the previous tree set, for After adding the tree set, the GBRT network can be trained by introducing an arbitrary loss function L through inverse gradient update; through the above operation, the SGBRT network is obtained by training GBRT using the cluster weights of SOM and the target vector as input. The training of the SGBRT network and the reinforcement learning training for action decision are carried out in parallel. The prediction in the reinforcement learning training is based on the pre-trained SGBRT network. While the reinforcement learning is being trained, the SGBRT network also collects samples from the reinforcement learning training process data to train a new model. After the training is completed, the new model replaces the pre-trained model in the reinforcement learning network, and samples are collected again. Step (2) Intrinsic Reward Generation and Reward Combination (2.1) In the k-th step of reinforcement learning training, its perceptual state and agent decision action are respectively and Inputting it into a pre-trained state prediction network yields the predicted state features. The prediction error at step k is calculated as the intrinsic reward. The formula is as follows: （8）； (2.2) After obtaining the intrinsic reward obtained in step (2.1), the intrinsic reward and extrinsic reward are combined for reinforcement learning training; however, since the sources of intrinsic and extrinsic rewards are different, the reward r needs to be standardized before combining the rewards, as shown in the following formula: （9）； Where mean and std are the mean and standard deviation of all rewards obtained, respectively. These two values ​​are dynamically calculated during training, taking into account the intrinsic reward. and external rewards After calculation using formula (9), the two reward scales are similar, resulting in... and Then, add the two rewards together, using the following formula: （10）； in, This is an adjustable hyperparameter used to adjust the role of intrinsic reward during training, specifically manifested as follows: The larger the size, the more the agent tends to optimize its strategy based on experience. The smaller the value, the more the agent tends to explore the environment; subsequently, the combined reward input behavior decision-making task policy update network is trained with reinforcement learning policies.

Images