Layered reinforcement learning method and device based on potential Bayesian surprise

Abstract

The invention provides a hierarchical reinforcement learning method and device based on potential Bayesian surprise, and relates to the technical field of reinforcement learning. The method comprises the following steps: acquiring the state of an agent at the current moment and a final target, and inputting the state and the final target into an upper-layer strategy network to generate sub-targets; inputting the state of the agent at the current moment and the sub-target into a lower-layer strategy network, selecting an executed atomic action, obtaining the state of the agent at the next moment, and calculating an internal reward based on the completion degree of the sub-target; according to the state of the agent at the current moment, the state of the agent at the next moment and the executed atomic action, adopting a potential dynamic model to calculate a potential Bayesian surprise, and taking the KL divergence between the obtained potential prior distribution probability and the potential posterior distribution probability as an internal reward based on the potential Bayesian surprise; and according to the two internal rewards, obtaining an atomic action for guiding the intelligent agent to reach the final target. According to the invention, the exploration rate of a lower-layer policy network can be enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a hierarchical reinforcement learning method and device based on potential Bayesian surprise. Background Art

[0002] Hierarchical reinforcement learning is a method based on the divide-and-conquer idea and a hierarchical policy structure. It can decompose complex long-term decision-making tasks into simple subtasks. In reinforcement learning, rewards are the feedback from the environment to the agent and serve as supervisory signals. The agent optimizes its strategy based on the rewards. Goal-based hierarchical reinforcement learning is a mainstream method of hierarchical reinforcement learning. The upper-level strategy generates sub-goals based on the current state and the final goal. The lower-level strategy selects atomic actions to be executed based on the current state and sub-goals, and obtains certain intrinsic rewards when the sub-goals are achieved. Goal-based hierarchical reinforcement learning is an effective solution for solving long-term and complex tasks. The current hierarchical reinforcement learning method uses the form of task decomposition, and the lower-level strategy can obtain certain intrinsic rewards by achieving sub-goals, which alleviates the sparse reward problem to a certain extent, but this is not enough to guide the agent to conduct effective exploration at the lower level. Traditional exploration methods such as - Greedy algorithms and UCB algorithms can only explore the range near the current state, cannot effectively discover meaningful states, and have difficulty coping with continuous control tasks with sparse rewards. Hierarchical reinforcement learning requires more effective exploration methods to deal with the problems of slow training speed and low sample efficiency caused by low exploration efficiency. However, current hierarchical reinforcement learning algorithms focus more on the exploration of the upper layer, and strengthen the exploration of the upper layer by limiting the action space of the upper layer, but often ignore the exploration of the lower layer.

[0003] At present, the landmark-guided hierarchical reinforcement learning (HIGL) algorithm has made improvements in the upper layer to improve the exploration efficiency of the upper layer, which is caused by the large action space of the upper layer, resulting in poor overall exploration performance and low training efficiency. The HIGL algorithm constructs a high-level action space for the upper layer, which can be reached from the current state and can also be transferred to promising milestone states, effectively improving the exploration in the upper layer. However, the HIGL algorithm ignores the exploration in the lower layer, and the lower layer still faces problems such as slow sub-strategy convergence and low sample efficiency caused by low exploration efficiency. At present, in continuous control tasks with sparse rewards, most of the existing hierarchical reinforcement learning methods only consider the exploration of the upper layer, and ignore the direct exploration of the state by the lower layer. In the training of the lower-layer sub-strategy, the intrinsic reward is only designed based on the distance between the current state and the sub-goal, without considering how to improve the exploration rate of the lower layer, resulting in slow training of the lower-layer sub-strategy and low sample efficiency. Summary of the invention

[0004] In order to solve the problem that most of the existing hierarchical reinforcement learning methods in the prior art only consider the exploration of the upper layer, ignoring the direct exploration of the state by the lower layer, and in the training of the lower layer sub-strategy, only the intrinsic reward is designed according to the distance between the current state and the sub-goal, without considering how to improve the exploration rate of the lower layer, resulting in slow training of the lower layer sub-strategy and low sample efficiency, the embodiment of the present invention provides a hierarchical reinforcement learning method and device based on potential Bayesian surprise. The technical solution is as follows:

[0005] In one aspect, a hierarchical reinforcement learning method based on potential Bayesian surprise is provided, the method being implemented by a hierarchical reinforcement learning device based on potential Bayesian surprise, the method comprising:

[0006] S1. Obtain the current state of the agent and the ultimate goal of the agent; construct the upper strategy network and the lower strategy network according to the hierarchical reinforcement learning method;

[0007] S2, inputting the state of the agent at the current moment and the final goal into the upper-layer strategy network to generate sub-goals;

[0008] S3, inputting the state of the agent at the current moment and the sub-goal into the lower-layer strategy network, selecting an atomic action to be executed, and obtaining the state of the agent at the next moment;

[0009] S4, calculating an intrinsic reward based on the degree of completion of the sub-goal according to the executed atomic action, the state of the agent at the current moment, the state of the agent at the next moment, and the sub-goal;

[0010] S5. According to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action executed, a potential dynamics model is used to calculate the potential Bayesian surprise to obtain a potential prior distribution probability and a potential posterior distribution probability;

[0011] S6. Calculating an intrinsic reward based on a potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability;

[0012] S7. According to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise, obtain the atomic action that guides the intelligent agent to reach the final goal.

[0013] Optionally, the intrinsic reward based on the sub-goal completion degree of S4 is expressed by the following formula (1):

[0014] (1)

[0015] in, represents intrinsic rewards based on the completion of sub-goals; represents a sub-goal; Represents the state of the agent at the current moment; Represents an atomic action to be performed; Represents the state of the agent at the next moment.

[0016] Optionally, the step S5 calculates the potential Bayesian surprise using a potential dynamics model according to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action to obtain a potential prior distribution probability and a potential posterior distribution probability, including:

[0017] S51, according to the state of the agent at the current moment and the atomic action being executed, adopting a potential dynamics model and introducing potential variables to obtain a potential prior distribution probability;

[0018] S52. According to the state of the agent at the current moment, the atomic action executed, and the state at the next moment after the state transfer occurs, a latent dynamics model is adopted, and latent variables are introduced to obtain a potential posterior distribution probability.

[0019] Optionally, the potential Bayesian surprise is used to quantify curiosity; the value of the curiosity is the difference between the potential prior distribution probability and the potential posterior distribution probability.

[0020] Optionally, the step S6 of calculating the intrinsic reward based on potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability includes:

[0021] The information gain is calculated according to the KL divergence between the potential prior distribution probability and the potential posterior distribution probability; the information gain is used as an intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward based on the potential Bayesian surprise is expressed by the following formula (2):

[0022] (2)

[0023] in, represents the intrinsic reward based on the underlying Bayesian surprise; represents the potential prior distribution probability; represents the potential posterior distribution probability; represents latent variables; represents KL divergence; Indicates that the current state is known and select the action to perform Under the condition of and the next state The mutual information between them.

[0024] Optionally, the optimization process of the potential Bayesian surprise parameter includes: adopting the maximum log-likelihood function and minimizing the KL divergence between the potential prior distribution probability and the potential posterior distribution probability for joint optimization; wherein the optimization process is represented by the following formula (3):

[0025] (3)

[0026] in, It is indicated that in order to control the disentanglement in the latent representation; represents the maximum log-likelihood function; Represents the objective function.

[0027] Optionally, the step S7, according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise, obtains an atomic action that guides the agent to reach the final goal, including:

[0028] S71. Obtain the intrinsic reward of the lower policy network according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward of the lower policy network is expressed by the following formula (4):

[0029] (4)

[0030] in, represents the weight of the intrinsic reward based on the completion of the sub-goal; represents the weight of the intrinsic reward based on the underlying Bayesian surprise; Represents the intrinsic reward of the underlying policy network;

[0031] S72. Based on the intrinsic rewards of the underlying policy network, obtain the atomic actions that guide the agent to reach the final goal.

[0032] On the other hand, a hierarchical reinforcement learning device based on potential Bayesian surprise is provided, and the device is applied to a hierarchical reinforcement learning method based on potential Bayesian surprise, and the device comprises:

[0033] The acquisition and construction unit is used to obtain the current state of the intelligent agent and the ultimate goal of the intelligent agent; according to the hierarchical reinforcement learning method, the upper strategy network and the lower strategy network are constructed;

[0034] A generating unit, used for inputting the state of the agent at the current moment and the final goal into the upper strategy network to generate a sub-goal;

[0035] A first acquisition unit is used to input the state of the agent at the current moment and the sub-goal into the lower-layer strategy network, select an atomic action to be executed, and acquire the state of the agent at the next moment;

[0036] A first calculation unit, configured to calculate an intrinsic reward based on a degree of completion of a sub-goal according to the executed atomic action, the state of the agent at the current moment, the state of the agent at the next moment, and the sub-goal;

[0037] A second acquisition unit is used to calculate the Bayesian surprise using a potential dynamics model according to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action executed, to obtain a potential prior distribution probability and a potential posterior distribution probability;

[0038] A second computing unit, configured to compute an intrinsic reward based on a potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability;

[0039] The third acquisition unit is used to acquire the atomic action that guides the intelligent agent to reach the final goal according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise.

[0040] Optionally, the intrinsic reward based on the sub-goal completion degree is expressed by the following formula (1):

[0041] (1)

[0042] in, represents intrinsic rewards based on the completion of sub-goals; represents a sub-goal; Represents the state of the agent at the current moment; Represents an atomic action to be performed; Represents the state of the agent at the next moment.

[0043] Optionally, the second acquiring unit is used to:

[0044] According to the state of the agent at the current moment and the atomic action being executed, a potential dynamics model is adopted to obtain a potential prior distribution probability by introducing a potential variable;

[0045] According to the state of the agent at the current moment, the atomic action executed, and the state at the next moment after the state transfer occurs, a potential dynamics model is adopted to obtain the potential posterior distribution probability by introducing latent variables.

[0046] Optionally, the potential Bayesian surprise is used to quantify curiosity; the value of the curiosity is the difference between the potential prior distribution probability and the potential posterior distribution probability.

[0047] Optionally, the second computing unit is used to:

[0048] The information gain is calculated according to the KL divergence between the potential prior distribution probability and the potential posterior distribution probability; the information gain is used as an intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward based on the potential Bayesian surprise is expressed by the following formula (2):

[0049] (2)

[0050] in, represents the intrinsic reward based on the underlying Bayesian surprise; represents the potential prior distribution probability; represents the potential posterior distribution probability;

[0051] represents latent variables; represents KL divergence; Indicates that the current state is known and select the action to perform Under the condition of and the next state The mutual information between them.

[0052] Optionally, the optimization process of the potential Bayesian surprise parameter includes: adopting the maximum log-likelihood function and minimizing the KL divergence between the potential prior distribution probability and the potential posterior distribution probability for joint optimization; wherein the optimization process is represented by the following formula (3):

[0053] (3)

[0054] in, It is indicated that in order to control the disentanglement in the latent representation; represents the maximum log-likelihood function; Represents the objective function.

[0055] Optionally, the third acquiring unit is used to:

[0056] According to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise, the intrinsic reward of the lower policy network is obtained; wherein the intrinsic reward of the lower policy network is expressed by the following formula (4):

[0057] (4)

[0058] in, represents the weight of the intrinsic reward based on the completion of the sub-goal; represents the weight of the intrinsic reward based on the underlying Bayesian surprise; Represents the intrinsic reward of the underlying policy network;

[0059] Based on the intrinsic rewards of the underlying policy network, atomic actions are obtained to guide the agent to the final goal.

[0060] On the other hand, a hierarchical reinforcement learning device based on potential Bayesian surprise is provided, and the hierarchical reinforcement learning device based on potential Bayesian surprise comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned hierarchical reinforcement learning methods based on potential Bayesian surprise is implemented.

[0061] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned hierarchical reinforcement learning methods based on potential Bayesian surprise.

[0062] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0063] The embodiment of the present invention first obtains the state of the intelligent agent at the current moment and the final goal of the intelligent agent; constructs an upper policy network and a lower policy network according to a hierarchical reinforcement learning method; secondly, inputs the state of the intelligent agent at the current moment and the final goal into the upper policy network to generate sub-goals; inputs the state of the intelligent agent at the current moment and the sub-goals into the lower policy network, selects the atomic actions to be executed, and obtains the state of the intelligent agent at the next moment; calculates the intrinsic reward based on the completion of the sub-goals according to the executed atomic actions, the state of the intelligent agent at the current moment, the state of the intelligent agent at the next moment, and the sub-goals; calculates the potential Bayesian surprise using a potential dynamics model according to the state of the intelligent agent at the current moment, the state of the intelligent agent at the next moment, and the atomic actions executed, and obtains the potential prior distribution probability and the potential posterior distribution probability; calculates the intrinsic reward based on the potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability; finally, obtains the atomic action that guides the intelligent agent to reach the final goal according to the intrinsic reward based on the completion of the sub-goals and the intrinsic reward based on the potential Bayesian surprise.

[0064] The embodiment of the present invention simulates the cognitive function of information seeking driven by the reward area of ​​the brain, introduces curiosity as an intrinsic reward in the lower layer of hierarchical reinforcement learning to encourage the agent to explore in the lower layer; in the present invention, curiosity is quantified as potential Bayesian surprise, and by constructing a potential dynamics model, the prior distribution probability and the posterior distribution probability are calculated in the latent space, which greatly reduces the computational cost compared with other methods based on model parameters to calculate Bayesian surprise; the introduction of potential Bayesian surprise as an intrinsic reward in the lower layer does not introduce additional computational energy consumption. By introducing potential Bayesian surprise in the lower layer to calculate the information gain brought by entering a new state, the agent is effectively guided to explore in the lower layer, and the key state for solving the task can be found more quickly, thereby speeding up the training speed of the sub-strategy and improving the sample efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 It is a structural diagram of a hierarchical reinforcement learning based on potential Bayesian surprise provided by an embodiment of the present invention;

[0067] Figure 2 is a flow chart of a hierarchical reinforcement learning method based on potential Bayesian surprise provided by an embodiment of the present invention;

[0068] Figure 3 is a schematic diagram of an average success rate on a dot maze with sparse rewards provided by an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of an average success rate on an ant maze with sparse rewards provided by an embodiment of the present invention;

[0070] Figure 5 is a schematic diagram of the number of explored states on a point maze with sparse rewards provided by an embodiment of the present invention;

[0071] Figure 6 is a schematic diagram of the number of explored states in an ant maze with sparse rewards provided by an embodiment of the present invention;

[0072] Figure 7 It is a block diagram of a hierarchical reinforcement learning device based on potential Bayesian surprise provided by an embodiment of the present invention;

[0073] Figure 8It is a structural schematic diagram of a hierarchical reinforcement learning device based on potential Bayesian surprise provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0075] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0076] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0077] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0078] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0079] The embodiment of the present invention provides a hierarchical reinforcement learning method based on potential Bayesian surprise, which can be implemented by a hierarchical reinforcement learning device based on potential Bayesian surprise, and the hierarchical reinforcement learning device based on potential Bayesian surprise can be a terminal or a server. Figure 1 is a schematic diagram of a structure of hierarchical reinforcement learning based on potential Bayesian surprise provided by an embodiment of the present invention, comprising Figure 2 The flowchart implementation of the hierarchical reinforcement learning method based on latent Bayesian surprise is shown.

[0080] like Figure 2 The flowchart of the hierarchical reinforcement learning method based on potential Bayesian surprise is shown in FIG. 1 . The processing flow of the method may include the following steps:

[0081] S1. Obtain the current state of the agent and the ultimate goal of the agent; construct the upper-layer strategy network and the lower-layer strategy network according to the hierarchical reinforcement learning method.

[0082] In a feasible implementation, the training objective of the upper policy network is to maximize the cumulative extrinsic rewards directly given by the environment; the training objective of the lower policy network is to maximize the cumulative intrinsic rewards and.

[0083] In a feasible implementation, in the upper strategy network, a specific state is set as the target space; wherein, a target mapping function is used to map the state to the target; in most cases, the state space and the target space are the same. In the maze task, the state space contains multi-dimensional information. In the point maze, each state in the state space contains four-dimensional information: the x-coordinate, y-coordinate, linear velocity in the x-direction, and linear velocity in the y-direction; in the target space, each target only contains two-dimensional information: the x-coordinate and y-coordinate of the ball; therefore, a target mapping function is needed to map the state The information is mapped into two-dimensional information in the corresponding target space to facilitate calculation; in the present invention, the target spaces in the point maze and the ant maze are both the x-coordinate and y-coordinate of the agent.

[0084] Among them, the upper-level policy network comprehensively considers the novelty of the state and the coverage of the state, generates sub-goals, and passes the sub-goals to the lower-level policy network.

[0085] S2: Input the current state and final goal of the agent into the upper-level strategy network to generate sub-goals.

[0086] S3: Input the current state and sub-goals of the agent into the lower-level strategy network, select the atomic action to be executed, and obtain the state of the agent at the next moment.

[0087] S4. Calculate the intrinsic reward based on the completion of the sub-goal according to the atomic action executed, the state of the agent at the current moment, the state of the agent at the next moment, and the sub-goal.

[0088] In a feasible implementation, the lower-level policy network selects the atomic action to be executed based on the sub-goals and the current state of the agent, and the agent transfers from the current state to the state of the next moment. The lower-level policy network feeds back the intrinsic reward to the agent based on the agent's completion of the sub-goals.

[0089] Optionally, the intrinsic reward of S4 based on the sub-goal completion is expressed by the following formula (1):

[0090] (1)

[0091] in, represents intrinsic rewards based on the completion of sub-goals; represents a sub-goal; Represents the state of the agent at the current moment; Represents an atomic action to be performed; Represents the state of the agent at the next moment.

[0092] Among them, the intrinsic reward based on the completion of the sub-goal is inversely proportional to the distance between the state of the agent at the current moment and the sub-goal; the greater the distance between the state of the agent at the next moment and the sub-goal, the smaller the reward value obtained.

[0093] S5. According to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic actions performed, the potential dynamics model is used to calculate the potential Bayesian surprise to obtain the potential prior distribution probability and the potential posterior distribution probability.

[0094] In a feasible implementation, the present invention adopts a latent dynamics model and can reduce computational costs by performing calculations in a latent space.

[0095] Optionally, the specific implementation process of S5 may include S51-S52:

[0096] S51, according to the state of the agent at the current moment and the atomic actions performed, adopt the potential dynamics model and introduce latent variables to obtain the potential prior distribution probability;

[0097] Among them, the latent variable Contains the state of the generated agent at the next moment All information.

[0098] Among them, the agent performs atomic actions in the current state, undergoes state transition, and enters the next state. In the process of transition, it will bring new information to the potential dynamics model.

[0099] S52. According to the state of the agent at the current moment, the atomic actions executed, and the state at the next moment after the state transfer occurs, a latent dynamics model is adopted to obtain the potential posterior distribution probability by introducing latent variables.

[0100] Optionally, latent Bayesian surprise is used to quantify curiosity; the value of curiosity is the difference between the probability of the latent prior distribution and the probability of the latent posterior distribution.

[0101] S6. Calculate the intrinsic reward based on the potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability.

[0102] Among them, the information gain is calculated according to the KL divergence between the potential prior distribution probability and the potential posterior distribution probability, and the information gain is used as the intrinsic reward generated by the agent driven by curiosity, that is, the intrinsic reward based on potential Bayesian surprise.

[0103] Optionally, specific implementations of S6 include:

[0104] The information gain is calculated according to the KL divergence between the potential prior distribution probability and the potential posterior distribution probability; the information gain is used as an intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward based on the potential Bayesian surprise is expressed by the following formula (2):

[0105] (2)

[0106] in, represents the intrinsic reward based on the underlying Bayesian surprise; represents the potential prior distribution probability; represents the potential posterior distribution probability;

[0107] represents latent variables; represents KL divergence; Indicates that the current state is known and select the action to perform Under the condition of and the next state The mutual information between them.

[0108] Optionally, the optimization process of the latent Bayesian surprise parameter includes: adopting the maximum log-likelihood function and minimizing the KL divergence between the latent prior distribution probability and the latent posterior distribution probability for joint optimization; wherein the optimization process is expressed by the following formula (3):

[0109] (3)

[0110] in, It is indicated that in order to control the disentanglement in the latent representation; represents the maximum log-likelihood function; Represents the objective function.

[0111] S7. Obtain atomic actions that guide the agent to reach the final goal based on the intrinsic rewards based on the completion of sub-goals and the intrinsic rewards based on potential Bayesian surprises.

[0112] Optionally, the specific implementation process of S7 may include S71-S72:

[0113] S71. Obtain the intrinsic reward of the lower policy network according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward of the lower policy network is expressed by the following formula (4):

[0114] (4)

[0115] in, represents the weight of the intrinsic reward based on the completion of the sub-goal; represents the weight of the intrinsic reward based on the underlying Bayesian surprise; Represents the intrinsic reward of the underlying policy network;

[0116] Among them, the weight of the intrinsic reward based on the sub-goal completion is used to balance the intrinsic reward based on the sub-goal completion; the weight of the intrinsic reward based on the potential Bayesian surprise is used to balance the intrinsic reward based on the potential Bayesian surprise.

[0117] Among them, the greater the weight of the intrinsic reward based on potential Bayesian surprise, the more it encourages the agent to explore states that can produce greater information gain.

[0118] Among them, the intrinsic reward based on potential Bayesian surprise effectively improves the exploration of the intelligent agent in the high-dimensional state space and speeds up the training of the sub-strategy; in the present invention, setting , .

[0119] In a feasible implementation, after the upper policy network selects a sub-goal, the lower policy network will perform multiple actions to achieve this sub-goal. The reward of the upper policy network consists of the external cumulative reward directly fed back by the environment after executing multiple actions, which can be expressed by the following formula (5):

[0120] (5)

[0121] in, represents a trajectory segment of length m, including states, sub-goals and atomic actions at different time steps; represents the reward of the upper strategy; j represents the starting time of the trajectory segment; t represents the time step; Indicates that the lower policy network is in state The atomic action performed The external reward of environmental feedback obtained later.

[0122] S72. Based on the intrinsic rewards of the underlying policy network, obtain the atomic actions that guide the agent to reach the final goal.

[0123] In a feasible implementation, the present invention provides an embodiment including: testing the effectiveness of the present invention in two continuous control tasks, the implementation environment is a dot maze and an ant maze, both of which are maze navigation environments, and the rewards of the two environments are set to be sparse. Compared with discrete state space and action space, the state and action space in the continuous control task are continuous, which can increase the difficulty of strategy optimization. At the same time, in the case of sparse rewards, the agent only obtains significant reward signals in a few time steps, and the rewards obtained in most time steps are very few or the rewards obtained have no differentiation. The original rewards are difficult to guide the agent to optimize the strategy, which increases the difficulty of the agent's exploration.

[0124] The point maze is a U-shaped maze. The task of the environment is to let a small ball that is subjected to force along the Cartesian x-axis and y-axis directions start from the lower left corner of the maze and reach the end point in the upper left corner of the maze. The action space of the ball is the force applied in the x-axis and y-axis directions, and the state space is the x-axis coordinate and y-axis coordinate of the ball and the linear velocity of the ball on the x-axis and y-axis. Under the setting of sparse rewards, the ball will receive a reward of 0 when it reaches the end point, otherwise the reward will be -1; the ant maze is a complex U-shaped maze with an action space of 8 dimensions and a state space of 30 dimensions. Both the action space and the state space are continuous. The task of the environment is to let the four-legged ant start from the lower left corner of the maze and reach the end point in the upper left corner of the maze. Under the setting of sparse rewards, the ball will receive a reward of 0 when it reaches the end point, otherwise the reward will be -1.

[0125] In a feasible implementation, the experiment uses 1 million time steps for training, and the maximum step size of each scene is set to 500 steps. During the evaluation process, the currently trained strategy is tested 20 times to calculate the average success rate, and the maximum step size of each scene during the test is still 500 steps. In order to prove that the present invention has better performance and data utilization in continuous control tasks with sparse rewards, the present invention is compared with the recognized hierarchical reinforcement learning benchmark method: HIGL algorithm. The HIGL algorithm selects landmarks based on the novelty of the state and the coverage of the state, constructs a graph based on the current state, landmarks and final goal, and selects reachable sub-goals based on the shortest path planning algorithm.

[0126] In a feasible implementation, Figure 3 As well as Figure 4 As shown in the figure, the horizontal axis is the number of training steps, and the vertical axis is the success rate. For the convenience of observation, the experimental results are processed by exponential moving average to make the experimental results smoother and give greater weight to the data at the current moment. For the experiment in each environment, 3 different random seeds are used to run. Figure 3 and Figure 4The curve in is the average result of three experiments, and the shadow is the standard deviation. It can be seen from the experimental results that compared with the HIGL algorithm, the method proposed in the present invention can achieve a higher success rate, accelerate the convergence speed of the sub-strategy, and improve the sample efficiency.

[0127] In a feasible implementation, Figure 5 and Figure 6 The figure shows the change in the number of explored states in the dot maze and ant maze environments as the training process progresses. From the experimental results, it can be seen that the method proposed in the present invention can explore more novel states faster, effectively improving the exploration ability of the lower layers of hierarchical reinforcement learning in high-dimensional continuous space.

[0128] The embodiment of the present invention first obtains the state of the intelligent agent at the current moment and the final goal of the intelligent agent; constructs an upper policy network and a lower policy network according to a hierarchical reinforcement learning method; secondly, inputs the state of the intelligent agent at the current moment and the final goal into the upper policy network to generate sub-goals; inputs the state of the intelligent agent at the current moment and the sub-goals into the lower policy network, selects the atomic actions to be executed, and obtains the state of the intelligent agent at the next moment; calculates the intrinsic reward based on the completion of the sub-goals according to the executed atomic actions, the state of the intelligent agent at the current moment, the state of the intelligent agent at the next moment, and the sub-goals; calculates the potential Bayesian surprise using a potential dynamics model according to the state of the intelligent agent at the current moment, the state of the intelligent agent at the next moment, and the atomic actions executed, and obtains the potential prior distribution probability and the potential posterior distribution probability; calculates the intrinsic reward based on the potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability; finally, obtains the atomic action that guides the intelligent agent to reach the final goal according to the intrinsic reward based on the completion of the sub-goals and the intrinsic reward based on the potential Bayesian surprise.

[0129] The embodiment of the present invention simulates the cognitive function of information seeking driven by the reward area of ​​the brain, introduces curiosity as an intrinsic reward in the lower layer of hierarchical reinforcement learning to encourage the agent to explore in the lower layer; in the present invention, curiosity is quantified as potential Bayesian surprise, and by constructing a potential dynamics model, the prior distribution probability and the posterior distribution probability are calculated in the latent space, which greatly reduces the computational cost compared with other methods based on model parameters to calculate Bayesian surprise; the introduction of potential Bayesian surprise as an intrinsic reward in the lower layer does not introduce additional computational energy consumption. By introducing potential Bayesian surprise in the lower layer to calculate the information gain brought by entering a new state, the agent is effectively guided to explore in the lower layer, and the key state for solving the task can be found more quickly, thereby speeding up the training speed of the sub-strategy and improving the sample efficiency.

[0130] Figure 7is a block diagram of a hierarchical reinforcement learning device based on potential Bayesian surprise according to an exemplary embodiment, wherein the device is used for a hierarchical reinforcement learning method based on potential Bayesian surprise. Figure 7 The device includes an acquisition and construction unit 710, a generation unit 720, a first acquisition unit 730, a first calculation unit 740, a second acquisition unit 750, a second calculation unit 760 and a third acquisition unit 770. Among them:

[0131] The acquisition and construction unit 710 is used to acquire the state of the agent at the current moment and the final goal of the agent; construct an upper strategy network and a lower strategy network according to a hierarchical reinforcement learning method;

[0132] A generating unit 720, configured to input the state of the agent at the current moment and the final goal into the upper strategy network to generate a sub-goal;

[0133] The first acquisition unit 730 is used to input the state of the agent at the current moment and the sub-goal into the lower-layer strategy network, select an atomic action to be executed, and obtain the state of the agent at the next moment;

[0134] A first calculation unit 740 is used to calculate an intrinsic reward based on the completion degree of a sub-goal according to the executed atomic action, the state of the agent at the current moment, the state of the agent at the next moment, and the sub-goal;

[0135] A second acquisition unit 750 is used to calculate the Bayesian surprise using a potential dynamics model according to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action executed, to obtain a potential prior distribution probability and a potential posterior distribution probability;

[0136] A second calculation unit 760, configured to calculate an intrinsic reward based on a potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability;

[0137] The third acquisition unit 770 is used to acquire the atomic action that guides the intelligent agent to reach the final goal according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise.

[0138] Optionally, the intrinsic reward based on the sub-goal completion degree is expressed by the following formula (1):

[0139] (1)

[0140] in, represents intrinsic rewards based on the completion of sub-goals; represents a sub-goal; Represents the state of the agent at the current moment; Represents an atomic action to be performed; Represents the state of the agent at the next moment.

[0141] Optionally, the second acquiring unit 750 is configured to:

[0142] According to the state of the agent at the current moment and the atomic actions performed, the potential dynamics model is adopted to obtain the potential prior distribution probability by introducing latent variables;

[0143] According to the state of the agent at the current moment, the atomic action executed, and the state at the next moment after the state transfer occurs, a potential dynamics model is adopted to obtain the potential posterior distribution probability by introducing latent variables.

[0144] Optionally, the potential Bayesian surprise is used to quantify curiosity; the value of the curiosity is the difference between the potential prior distribution probability and the potential posterior distribution probability.

[0145] Optionally, the second calculating unit 760 is configured to:

[0146] The information gain is calculated according to the KL divergence between the potential prior distribution probability and the potential posterior distribution probability; the information gain is used as an intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward based on the potential Bayesian surprise is expressed by the following formula (2):

[0147] (2)

[0148] in, represents the intrinsic reward based on the underlying Bayesian surprise; represents the potential prior distribution probability; represents the potential posterior distribution probability;

[0149] represents latent variables; represents KL divergence; Indicates that the current state is known and select the action to perform Under the condition of and the next state The mutual information between them.

[0150] Optionally, the optimization process of the potential Bayesian surprise parameter includes: adopting the maximum log-likelihood function and minimizing the KL divergence between the potential prior distribution probability and the potential posterior distribution probability for joint optimization; wherein the optimization process is represented by the following formula (3):

[0151] (3)

[0152] in, It is indicated that in order to control the disentanglement in the latent representation; represents the maximum log-likelihood function; Represents the objective function.

[0153] Optionally, the third acquiring unit 770 is configured to:

[0154] According to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise, the intrinsic reward of the lower policy network is obtained; wherein the intrinsic reward of the lower policy network is expressed by the following formula (4):

[0155] (4)

[0156] in, represents the weight of the intrinsic reward based on the completion of the sub-goal; represents the weight of the intrinsic reward based on the underlying Bayesian surprise; Represents the intrinsic reward of the underlying policy network;

[0157] Based on the intrinsic rewards of the underlying policy network, atomic actions are obtained to guide the agent to the final goal.

[0158] The embodiment of the present invention first obtains the state of the intelligent agent at the current moment and the final goal of the intelligent agent; constructs an upper policy network and a lower policy network according to a hierarchical reinforcement learning method; secondly, inputs the state of the intelligent agent at the current moment and the final goal into the upper policy network to generate sub-goals; inputs the state of the intelligent agent at the current moment and the sub-goals into the lower policy network, selects the atomic actions to be executed, and obtains the state of the intelligent agent at the next moment; calculates the intrinsic reward based on the completion of the sub-goals according to the executed atomic actions, the state of the intelligent agent at the current moment, the state of the intelligent agent at the next moment, and the sub-goals; calculates the potential Bayesian surprise using a potential dynamics model according to the state of the intelligent agent at the current moment, the state of the intelligent agent at the next moment, and the atomic actions executed, and obtains the potential prior distribution probability and the potential posterior distribution probability; calculates the intrinsic reward based on the potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability; finally, obtains the atomic action that guides the intelligent agent to reach the final goal according to the intrinsic reward based on the completion of the sub-goals and the intrinsic reward based on the potential Bayesian surprise.

[0159] The embodiment of the present invention simulates the cognitive function of information seeking driven by the reward area of ​​the brain, introduces curiosity as an intrinsic reward in the lower layer of hierarchical reinforcement learning to encourage the agent to explore in the lower layer; in the present invention, curiosity is quantified as potential Bayesian surprise, and by constructing a potential dynamics model, the prior distribution probability and the posterior distribution probability are calculated in the latent space, which greatly reduces the computational cost compared with other methods based on model parameters to calculate Bayesian surprise; the introduction of potential Bayesian surprise as an intrinsic reward in the lower layer does not introduce additional computational energy consumption. By introducing potential Bayesian surprise in the lower layer to calculate the information gain brought by entering a new state, the agent is effectively guided to explore in the lower layer, and the key state for solving the task can be found more quickly, thereby speeding up the training speed of the sub-strategy and improving the sample efficiency.

[0160] Figure 8 is a schematic diagram of the structure of a hierarchical reinforcement learning device based on potential Bayesian surprise provided by an embodiment of the present invention, such as Figure 8 As shown, the hierarchical reinforcement learning device based on potential Bayesian surprise can include the above Figure 7 The hierarchical reinforcement learning device based on potential Bayesian surprise is shown. Optionally, the hierarchical reinforcement learning device based on potential Bayesian surprise 810 may include a first processor 2001.

[0161] Optionally, the hierarchical reinforcement learning device 810 based on potential Bayesian surprise may further include a memory 2002 and a transceiver 2003 .

[0162] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0163] Combine the following Figure 8 The components of the hierarchical reinforcement learning device 810 based on potential Bayesian surprise are specifically introduced:

[0164] The first processor 2001 is the control center of the hierarchical reinforcement learning device 810 based on potential Bayesian surprise, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0165] Optionally, the first processor 2001 can perform various functions of the hierarchical reinforcement learning device 810 based on potential Bayesian surprise by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0166] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0167] In a specific implementation, as an embodiment, the hierarchical reinforcement learning device 810 based on potential Bayesian surprise may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0168] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0169] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0170] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0171] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0172] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0173] It should be noted that Figure 8 The structure of the hierarchical reinforcement learning device 810 based on potential Bayesian surprise shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0174] In addition, the technical effects of the hierarchical reinforcement learning device 810 based on potential Bayesian surprise can refer to the technical effects of the hierarchical reinforcement learning method based on potential Bayesian surprise described in the above method embodiment, which will not be repeated here.

[0175] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0176] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0177] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0178] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0179] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0180] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0181] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0182] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0183] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0186] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0187] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A hierarchical reinforcement learning method based on potential Bayesian surprise, characterized in that: The method comprises: S1. Obtain the current state of the agent and the ultimate goal of the agent; construct the upper strategy network and the lower strategy network according to the hierarchical reinforcement learning method; S2, inputting the state of the agent at the current moment and the final goal into the upper-layer strategy network to generate sub-goals; S3, inputting the state of the agent at the current moment and the sub-goal into the lower-layer strategy network, selecting an atomic action to be executed, and obtaining the state of the agent at the next moment; S4, calculating an intrinsic reward based on the degree of completion of the sub-goal according to the executed atomic action, the state of the agent at the current moment, the state of the agent at the next moment, and the sub-goal; S5. According to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action executed, a potential dynamics model is used to calculate the potential Bayesian surprise to obtain a potential prior distribution probability and a potential posterior distribution probability; S6. Calculating an intrinsic reward based on a potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability; S7. According to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise, obtain the atomic action that guides the intelligent agent to reach the final goal.

2. The hierarchical reinforcement learning method based on potential Bayesian surprise according to claim 1, characterized in that: The intrinsic reward of S4 based on the completion of sub-goals is expressed by the following formula (1): （1） in, represents intrinsic rewards based on the completion of sub-goals; represents a sub-goal; Represents the state of the agent at the current moment; Represents an atomic action to be performed; Represents the state of the agent at the next moment.

3. The hierarchical reinforcement learning method based on potential Bayesian surprise according to claim 1, characterized in that: The step S5 calculates the potential Bayesian surprise using a potential dynamics model according to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action being executed, to obtain a potential prior distribution probability and a potential posterior distribution probability, including: S51, according to the state of the agent at the current moment and the atomic action being executed, adopting a potential dynamics model and introducing potential variables to obtain a potential prior distribution probability; S52. According to the state of the agent at the current moment, the atomic action executed, and the state at the next moment after the state transfer occurs, a latent dynamics model is adopted to obtain a potential posterior distribution probability by introducing latent variables.

4. The hierarchical reinforcement learning method based on potential Bayesian surprise according to claim 3, characterized in that: The potential Bayesian surprise is used to quantify curiosity; the value of the curiosity is the difference between the potential prior distribution probability and the potential posterior distribution probability.

5. The hierarchical reinforcement learning method based on potential Bayesian surprise according to claim 3, characterized in that: The step S6 calculates the intrinsic reward based on the potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability, including: The information gain is calculated according to the KL divergence between the potential prior distribution probability and the potential posterior distribution probability; the information gain is used as an intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward based on the potential Bayesian surprise is expressed by the following formula (2): （2） in, represents the intrinsic reward based on the underlying Bayesian surprise; represents the potential prior distribution probability; represents the potential posterior distribution probability; represents latent variables; represents KL divergence; Indicates that the current state is known and select the action to perform Under the condition of and the next state The mutual information between them.

6. The hierarchical reinforcement learning method based on potential Bayesian surprise according to claim 5, characterized in that: The optimization process of the latent Bayesian surprise parameter includes: using the maximum log-likelihood function and minimizing the KL divergence between the latent prior distribution probability and the latent posterior distribution probability for joint optimization; wherein the optimization process is represented by the following formula (3): （3） in, It is indicated that in order to control the disentanglement in the latent representation; represents the maximum log-likelihood function; Represents the objective function.

7. The hierarchical reinforcement learning method based on potential Bayesian surprise according to claim 1, characterized in that: The step S7 obtains an atomic action that guides the agent to reach the final goal according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise, including: S71. Obtain the intrinsic reward of the lower policy network according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise; wherein the intrinsic reward of the lower policy network is expressed by the following formula (4): （4） in, represents the weight of the intrinsic reward based on the completion of the sub-goal; represents the weight of the intrinsic reward based on the underlying Bayesian surprise; Represents the intrinsic reward of the underlying policy network; S72. Based on the intrinsic rewards of the underlying policy network, obtain the atomic actions that guide the agent to reach the final goal.

8. A hierarchical reinforcement learning device based on potential Bayesian surprise, wherein the hierarchical reinforcement learning device based on potential Bayesian surprise is used to implement the hierarchical reinforcement learning method based on potential Bayesian surprise as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The acquisition and construction unit is used to obtain the current state of the intelligent agent and the ultimate goal of the intelligent agent; according to the hierarchical reinforcement learning method, the upper strategy network and the lower strategy network are constructed; A generating unit, used for inputting the state of the agent at the current moment and the final goal into the upper strategy network to generate a sub-goal; A first acquisition unit is used to input the state of the agent at the current moment and the sub-goal into the lower-layer strategy network, select an atomic action to be executed, and acquire the state of the agent at the next moment; A first calculation unit, configured to calculate an intrinsic reward based on a degree of completion of a sub-goal according to the executed atomic action, the state of the agent at the current moment, the state of the agent at the next moment, and the sub-goal; A second acquisition unit is used to calculate the Bayesian surprise using a potential dynamics model according to the state of the agent at the current moment, the state of the agent at the next moment, and the atomic action executed, to obtain a potential prior distribution probability and a potential posterior distribution probability; A second computing unit, configured to compute an intrinsic reward based on a potential Bayesian surprise according to the potential prior distribution probability and the potential posterior distribution probability; The third acquisition unit is used to acquire the atomic action that guides the intelligent agent to reach the final goal according to the intrinsic reward based on the sub-goal completion and the intrinsic reward based on the potential Bayesian surprise.

9. A hierarchical reinforcement learning device based on potential Bayesian surprise, characterized in that The hierarchical reinforcement learning device based on potential Bayesian surprise includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images