Method and device for determining exploration strategy of agent, agent and storage medium

Abstract

The present disclosure relates to a method and device for determining an exploration strategy of an agent, the agent and a storage medium, applied to the technical field of machine learning, for determining an exploration strategy of an agent, the exploration strategy including an exploration strategy based on a preset network model. The method for determining the exploration strategy of the agent includes: obtaining a historical trajectory set of the agent; determining a predicted trajectory set of the agent based on preset network parameters of the preset network model; determining a distance parameter between the predicted trajectory set and the historical trajectory set; determining a target value of the preset network parameters based on the distance parameter; and determining a target exploration strategy of the agent according to the target value of the preset network parameters. In the present disclosure, the preset network parameters are determined through the distance parameter between the predicted trajectory set and the historical trajectory set, which can enable the agent to explore more unknown areas, thereby enabling the agent to break away from a local optimal strategy, learn a global optimal strategy, and greatly improve the exploration ability of the agent for new trajectories.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of reinforcement learning and agent evolution strategy, and particularly relates to a method and device for determining an exploration strategy of an agent, an agent and a storage medium. BACKGROUND

[0002] With the development of deep reinforcement learning algorithms in various fields, an agent can obtain complex manipulation and motor skills from raw sensory data, such as playing Atari games with raw pixel input, mastering Go games, and the like. However, these deep reinforcement learning algorithms have a poor performance in tasks with sparse or delayed rewards, large state spaces, and deceptive local optimization. For the problem of large-scale sparse rewards, in the related art, in order to encourage the agent to fully explore the environment, noise sampled from a random distribution such as a Gaussian distribution or an Ornstein-Unlenbeck process is added to the action output by the policy network, so as to encourage the agent to visit states that it rarely or never visits, so as to guide the agent to enter unknown areas; or a maximum entropy method is used to take new actions in the existing state, allowing the agent to better explore the environment by encouraging a high-entropy distribution of actions in a given state. However, the method in the related art may lead to suboptimal behavior in exploration tasks with deceptive or sparse rewards, and therefore does not achieve satisfactory results. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides a method and device for determining an exploration strategy of an agent, an agent and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for determining an exploration strategy of an agent is provided, the exploration strategy including an exploration strategy based on a preset network model, and the determining method includes:

[0005] obtaining a historical trajectory set of the agent, the historical trajectory set being a historical state-action distribution set;

[0006] determining a predicted trajectory set of the agent based on preset network parameters of the preset network model, the predicted trajectory set being a predicted state-action distribution set;

[0007] determining a distance parameter between the predicted trajectory set and the historical trajectory set;

[0008] determining a target value of the preset network parameters based on the distance parameter;

[0009] determining a target exploration strategy of the agent according to the target value of the preset network parameters.

[0010] In an example embodiment, the determining the target value of the preset network parameter based on the distance parameter comprises:

[0011] determining a target function of the preset network model according to the distance parameter and a reward function of the preset network model; the target function and the reward function are both functions based on the preset network parameter;

[0012] determining the target value of the preset network parameter as a value of the preset network parameter when the target function reaches a maximum value.

[0013] In an example embodiment, the determining the target function of the preset network model according to the distance parameter and the reward function of the preset network model comprises:

[0014] the target function is determined by the following formula:

[0015]

[0016] wherein θ represents the preset network parameter, L(θ) represents the target function, represents the reward function, ρ π represents the predicted state-action distribution, ρ μ represents the historical state-action distribution, D MMD (ρ π , ρ μ ) represents the distance parameter, and α represents a weight.

[0017] In an example embodiment, the determining the target value of the preset network parameter as the value of the preset network parameter when the target function reaches the maximum value comprises:

[0018] calculating a gradient of the target function;

[0019] determining the value of the preset network parameter when the target function reaches the maximum value based on a gradient estimation algorithm.

[0020] In an example embodiment, the determining the predicted trajectory set of the agent based on the preset network parameter of the preset network model comprises:

[0021] sampling the preset network parameter multiple times to obtain multiple sampling values of the preset network parameter;

[0022] inputting the multiple sampling values of the preset network parameter into the preset network model to obtain the predicted trajectory set.

[0023] In an example embodiment, the sampling the preset network parameter multiple times to obtain a sampling value of the preset network parameter comprises:

[0024] Sampling based on a normal distribution:

[0025]

[0026] wherein, represents the normal distribution, σ represents a standard deviation of the normal distribution, I represents a unit matrix, σ 2 I represents a covariance matrix of the normal distribution, n represents a number of samplings, ∈ i represents an i-th sampling value;

[0027] The sampling value of the preset network parameter is determined by the following formula:

[0028] θ i = θ + ∈ i

[0029] wherein, θ i represents a sampling value of the preset network parameter in the i-th sampling, and θ represents the preset network parameter.

[0030] In an example embodiment, the determining the distance parameter between the predicted trajectory set and the historical trajectory set comprises:

[0031] Determining the distance parameter between each predicted state-action distribution of the predicted trajectory set and the historical trajectory set.

[0032] According to a second aspect of the embodiments of the present disclosure, a determination apparatus of an exploration strategy of an agent is provided, the exploration strategy comprising an exploration strategy based on a preset network model, and the determination apparatus comprises:

[0033] An acquisition module configured to acquire a historical trajectory set of the agent, the historical trajectory set comprising a historical state-action distribution set;

[0034] A first determination module configured to determine a predicted trajectory set of the agent based on a preset network parameter of the preset network model, the predicted trajectory set comprising a predicted state-action distribution set;

[0035] A second determination module configured to determine a distance parameter between the predicted trajectory set and the historical trajectory set;

[0036] A third determination module configured to determine a target value of the preset network parameter based on the distance parameter;

[0037] A fourth determining module is configured to determine a target exploration strategy of the agent according to a target value of the preset network parameter.

[0038] In an example embodiment, the third determining module is further configured to:

[0039] determine a target function of the preset network model according to the distance parameter and a reward function of the preset network model; the target function and the reward function are both functions based on the preset network parameter;

[0040] determine a target value of the preset network parameter when the target function has a maximum value as the target value of the preset network parameter.

[0041] In an example embodiment, the third determining module is further configured to:

[0042] determine the target function by the following formula:

[0043]

[0044] wherein θ represents the preset network parameter, L(θ) represents the target function, represents the reward function, ρ π represents the predicted state-action distribution, ρ μ represents the historical state-action distribution, D MMD (ρ π , ρ μ ) represents the distance parameter, and α represents a weight.

[0045] In an example embodiment, the third determining module is further configured to:

[0046] calculate a gradient of the target function;

[0047] determine a value of the preset network parameter when the target function has a maximum value based on a gradient estimation algorithm.

[0048] In an example embodiment, the first determining module is further configured to:

[0049] sample the preset network parameter multiple times to obtain multiple sampling values of the preset network parameter;

[0050] input the multiple sampling values of the preset network parameter into a preset network model to obtain the set of predicted trajectories.

[0051] In an example embodiment, the first determining module is further configured to:

[0052] sample based on a normal distribution:

[0053]

[0054] wherein, represents the normal distribution, σ represents a standard deviation of the normal distribution, I represents a unit matrix, σ 2 I represents a covariance matrix of the normal distribution, n represents a number of samplings, ∈ i represents an i-th sampling value of the preset network parameter;

[0055] The sampling value of the preset network parameter is determined by the following formula:

[0056] θ i = θ + ∈ i

[0057] wherein, θ i represents an i-th sampling value of the preset network parameter, θ represents the preset network parameter.

[0058] In an example embodiment, the second determining module is further configured to:

[0059] determine a distance parameter between each of the predicted state-action distributions of the set of predicted trajectories and the set of historical trajectories.

[0060] According to a third aspect of embodiments of the present disclosure, an agent is provided, comprising:

[0061] a processor;

[0062] a memory for storing processor-executable instructions;

[0063] wherein the processor is configured to perform the method according to any one of the first aspect of embodiments of the present disclosure.

[0064] According to a fourth aspect of embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, when instructions in the storage medium are executed by a processor of an agent, the agent is enabled to perform the method according to any one of the first aspect of embodiments of the present disclosure.

[0065] The above method of the present disclosure has the following beneficial effects: in the present disclosure, the preset network parameter is determined by the distance parameter between the set of predicted trajectories and the set of historical trajectories, which can enable the agent to explore more unknown areas, so as to make the agent get rid of the local optimal strategy, learn the global optimal strategy, and greatly improve the exploration ability of the agent to new trajectories.

[0066] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0068] Figure 1 is a flow chart of a method for determining an exploration strategy of an agent according to an example embodiment;

[0069] Figure 2 is a schematic diagram of an exploration environment of an agent according to an example embodiment;

[0070] Figure 3 is a flow chart of a method for determining a set of predicted trajectories of an agent based on preset network parameters of a preset network model in step S102 according to an example embodiment;

[0071] Figure 4 is a schematic diagram of an exploration environment of an agent according to an example embodiment;

[0072] Figure 5 is a flow chart of a method for determining a target value of the preset network parameters based on a distance parameter in step S104 according to an example embodiment;

[0073] Figure 6 is a schematic diagram of an exploration trajectory of an agent according to an example embodiment;

[0074] Figure 7 is a flow chart of a method for determining a value of the preset network parameters as a target value of the preset network parameters when a value of a target function is maximum in step S302 according to an example embodiment;

[0075] Figure 8 is a block diagram of a device for determining an exploration strategy of an agent according to an example embodiment;

[0076] Figure 9 is a block diagram of an agent according to an example embodiment. DETAILED DESCRIPTION

[0077] The example embodiments will now be described in detail with reference to the accompanying drawings. If the description of the embodiments contained herein refers to arrangements and devices other than those depicted in the drawings, it will be understood that such references are made by analogy with the embodiments depicted in the drawings. The following description of example embodiments is not representative of all embodiments consistent with the application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the application as detailed in the appended claims.

[0078] In the exemplary embodiments of the present disclosure, in order to overcome the problems in the related art, a method for determining an exploration strategy of an agent is provided, a historical trajectory set of the agent is obtained, the historical trajectory set is a historical state-action distribution set, a predicted trajectory set of the agent is determined based on preset network parameters of a preset network model, the predicted trajectory set is a predicted state-action distribution set, a distance parameter between the predicted trajectory set and the historical trajectory set is determined, a target value of the preset network parameters is determined based on the distance parameter, and a target exploration strategy of the agent is determined according to the target value of the preset network parameters. In the method of the present disclosure, the distance parameter between the predicted trajectory set and the historical trajectory set is used to determine the preset network parameters, so that the agent can explore more unknown areas, thereby enabling the agent to get rid of the local optimal strategy and learn the global optimal strategy, and greatly improving the exploration ability of the agent for new trajectories.

[0079] In the exemplary embodiments of the present disclosure, a method for determining an exploration strategy of an agent is provided, wherein the exploration strategy includes an exploration strategy based on a preset network model, Figure 1 is a flowchart of a method for determining an exploration strategy of an agent according to an exemplary embodiment, as Figure 1 shown, comprising the following steps:

[0080] In step S101, a historical trajectory set of the agent is obtained, and the historical trajectory set is a historical state-action distribution set.

[0081] In step S102, a predicted trajectory set of the agent is determined based on preset network parameters of a preset network model, and the predicted trajectory set is a predicted state-action distribution set.

[0082] In step S103, a distance parameter between the predicted trajectory set and the historical trajectory set is determined.

[0083] In step S104, a target value of the preset network parameters is determined based on the distance parameter.

[0084] In step S105, a target exploration strategy of the agent is determined according to the target value of the preset network parameters.

[0085] The preset network model is a reinforcement learning network model, which is composed of an Actor network in an Actor-Critic algorithm, the position of the agent is taken as the state s of the agent, the direction to be moved by the agent is taken as the action a of the agent, the Actor network constructs a neural network with the dimensions of the state space and the action space of the agent as the number of input layer and output layer neurons respectively, the Actor network can output actions in combination with the environment of the agent, and the agent interacts with the environment through the actions, wherein the state-action distribution of the agent is the exploration strategy of the agent. Figure 2 is a schematic diagram of an agent exploring an environment according to an exemplary embodiment, asFigure 2 As shown, the exploration environment is a Humanoid-v1 scene in a 3D simulation environment Mujoco, the state of the agent 201 at the current time is input into the preset network model, the preset network model can predict the feasible moving direction of the agent according to the position of the obstacle 202 in the environment, as the action output of the agent at the current time, so as to obtain the exploration strategy of the agent, the agent moves according to the exploration strategy, the position after moving is the state of the agent at the next time, and the action of the agent at the next time is predicted, and the agent 201 can complete the exploration task of avoiding the obstacle 202 in the exploration environment.

[0086] In step S101, the historical state-action distribution of the agent is the state-action distribution before the current time of the agent, that is, the actual moving track generated by the historical strategy of the agent, and the historical track set is composed of the set of historical state-action distributions at different times. The historical strategy of the agent is denoted as μ(a|s), wherein a represents an action, and s represents a state, and the historical state-action distribution is denoted as ρ μ (s, a), and the historical track set is formed by the set of historical state-action distributions

[0087] In step S102, the preset network parameters in the preset network model are denoted as θ, the current state of the agent is input into the preset network model, and the action of the agent is predicted based on the preset network parameters, so as to generate a prediction strategy, obtain a prediction state-action distribution, and obtain the set of prediction state-action distributions corresponding to different values of the preset network parameters by continuously adjusting the values of the preset network parameters, and the prediction track set is composed of the set of prediction state-action distributions. The prediction strategy of the agent is denoted as π(a|s), wherein a represents an action, and s represents a state, and the prediction state-action distribution is denoted as ρ π (s, a), and the prediction track set is formed by the set of prediction state-action distributions

[0088] In step S103, since the prediction track set is a set of prediction state-action distributions, and the historical track set is a set of historical state-action distributions, the distance parameter between the prediction track set and the historical track set is determined, that is, the difference between the prediction state-action distribution and the historical state-action distribution is determined, the prediction state-action distribution is denoted as ρ π (s, a), the historical state-action distribution is denoted as ρ μ (s, a), and the distance parameter between the prediction track set and the historical track set is denoted as D MMD (ρ π , ρ μ ).

[0089] In step S104, the set of predicted trajectories is determined based on the preset network parameters, and the distance parameter between the set of predicted trajectories and the set of historical trajectories is also determined based on the preset network parameters. The value of the distance parameter can be used to determine the value of the preset network parameters. Therefore, the value of the preset network parameters determined based on the value expected to satisfy the distance parameter can be used as the target value of the preset network parameters.

[0090] In step S105, when the value of the preset network parameters is the target value, it indicates that the action predicted by the preset network model is the expected action of the agent. The exploration strategy determined based on the current state of the agent and the expected action is the target exploration strategy of the agent. In order to improve the exploration ability of the agent for new trajectories and learn the globally optimal strategy, the agent is expected to explore more unknown areas, i.e., the agent is expected to learn a prediction strategy different from the historical strategy μ. When the prediction strategy π is significantly different from the historical strategy μ, the difference between the predicted state-action distribution and the historical state-action distribution will also be large, i.e., the distance parameter between the set of predicted trajectories and the set of historical trajectories will have a large value. Therefore, the value of the preset network parameters corresponding to the maximum value of the distance parameter is used as the target value of the preset network parameters. The target exploration strategy corresponding to the target value of the preset network parameters is the exploration strategy that is most different from the historical strategy.

[0091] In the exemplary embodiments of the present disclosure, the set of historical trajectories of the agent is obtained, the set of predicted trajectories of the agent is determined based on the preset network parameters of the preset network model, the distance parameter between the set of predicted trajectories and the set of historical trajectories is determined, the target value of the preset network parameters is determined based on the distance parameter, the target exploration strategy of the agent is determined based on the target value of the preset network parameters, and the agent can explore more unknown areas according to the target exploration strategy. Therefore, the agent can break out of the local optimal strategy and learn the globally optimal strategy, and the exploration ability of the agent for new trajectories can be greatly improved.

[0092] In an exemplary embodiment, Figure 3 FIG. 2 shows a flowchart of a method for determining the set of predicted trajectories of the agent based on the preset network parameters of the preset network model in step S102 according to an exemplary embodiment. As shown in FIG. 2, the method comprises the following steps: Figure 3

[0093] Step S301: The preset network parameters are sampled multiple times to obtain multiple sampling values of the preset network parameters.

[0094] Step S302: The multiple sampling values of the preset network parameters are input into the preset network model to obtain the set of predicted trajectories.

[0095] ​The sampling method of the preset network parameter can be any sampling method, such as inverse transform sampling, rejection sampling, normal distribution sampling, and the like.

[0096] In an example, sampling is performed based on a normal distribution:

[0097]

[0098] wherein, represents a normal distribution, σ represents a standard deviation of the normal distribution, I represents a unit matrix, σ 2 I represents a covariance matrix of the normal distribution, n represents a number of samplings, ∈ i represents an i-th sampling value.

[0099] The sampling value of the preset network parameter is determined by the following formula:

[0100] θ i = θ + ∈ i

[0101] wherein, θ i represents an i-th sampling value of the preset network parameter, and θ represents the preset network parameter.

[0102] The sampling value of each preset network parameter is input into the preset network model, and a predicted action corresponding to a current state of the agent is output, at this time, represents that one round of training of the preset network model is completed, and the preset network parameter is updated to a next sampling value of the preset network parameter when the next round of training is performed. The number of training rounds of the preset network model is related to the number of sampling values of the preset network parameter, and the sampling values of all preset network parameters correspond to a state-action distribution to form a predicted trajectory set.

[0103] It should be noted that the number of preset network parameters can be set according to actual requirements, and the M preset network parameters are denoted as θ1, θ2... θ M In each round of training of the preset network model, the M preset network parameters need to be sampled and updated. In order to improve the determination time of the target exploration strategy, parallel computing is used for updating the multiple preset network parameters. When the M preset network parameters are set, the distribution of M diversified and efficient strategies can be optimized, and diversified agents are realized, for example, Figure 4 is a schematic diagram of an exploration environment of an agent according to an example embodiment, as Figure 4 shown, the exploration environment has various complex roads, and the agent needs to master different skills, so a diversified agent with multiple preset network parameters is needed to complete the exploration task in the exploration environment.

[0104] In an example embodiment, Figure 5is a method flow chart for determining the target value of the preset network parameter based on the distance parameter in step S104 according to an exemplary embodiment, as shown in Figure 5 includes the following steps:

[0105] Step S501, determining a target function of the preset network model according to the distance parameter and a reward function of the preset network model; the target function and the reward function are both functions based on the preset network parameter;

[0106] Step S502, determining the value of the preset network parameter when the value of the target function is the maximum value as the target value of the preset network parameter.

[0107] In the exploration process of the agent in the environment, the current state of the agent is input into the preset network model, and the action corresponding to the current state is determined based on the preset network parameter. The agent selects an action, and the environment responds to the action to make the agent move to a new state, and an environment reward signal is generated. The environment reward signal is usually a numerical value. When the numerical value of the environment reward signal is the maximum, it means that the corresponding action has the maximum return. Therefore, the agent usually maximizes the numerical value of the environment reward signal in action selection. In the process of training the preset network model, the reward function is determined based on the preset network parameter, and the value of the preset network parameter corresponding to the maximum value of the reward function is the target value of the preset network parameter. The action corresponding to the target value is the action with the maximum return in the exploration process of the agent. The process of obtaining the target value of the preset network parameter is called the optimization process of the reward function.

[0108] Since the distance parameter between the prediction trajectory set and the historical trajectory set is also based on the preset network parameter, and the distance parameter between the prediction trajectory set and the historical trajectory set is the maximum value, the prediction strategy π and the historical strategy μ are most different, which can make the agent explore more unknown areas. Therefore, the target function of the preset network model is determined according to the reward function and the distance parameter. When the value of the target function is the maximum value, the reward function and the distance parameter are both the maximum value. The value of the preset network parameter at this time is determined as the target value of the preset network parameter, which can optimize the reward function while making the agent not repeat the original movement trajectory as much as possible to improve the exploration ability of the agent to explore new trajectories and avoid the agent falling into local optimum.

[0109] For example, Figure 6 is a schematic diagram of the exploration trajectory of the agent according to an exemplary embodiment, as shown in Figure 6As shown, the star-shaped figure 1 represents a suboptimal target, the star-shaped figure 2 represents an optimal target, the trajectory a represents an exploration trajectory of the agent reaching the suboptimal target without adding the distance parameter in the target function of the preset network model, and the trajectory b represents an exploration trajectory of the agent reaching the optimal target with the distance parameter added in the target function of the preset network model. Adding the distance parameter in the target function of the preset network model is equivalent to limiting the exploration trajectory of the agent with the red circle, so that the agent explores the unknown area, thereby avoiding entering the local optimal target, i.e., the suboptimal target, and reaching the global optimal target.

[0110] In an example embodiment, the step S301 of determining the target function of the preset network model according to the distance parameter and the reward function of the preset network model comprises:

[0111] The target function is determined by the following formula:

[0112]

[0113] wherein θ represents the preset network parameter, L(θ) represents the target function, represents the reward function, ρ π represents the predicted state-action distribution, ρ μ represents the historical state-action distribution, D MMD (ρ π , ρ μ ) represents the distance parameter, and α represents the weight. The weight α determines the proportion of the distance parameter in the target function, and adjusting the weight α can determine whether the agent is more inclined to pursue a higher environmental reward or a different movement trajectory.

[0114] In an example embodiment, Figure 7 is a method flowchart for determining the value of the preset network parameter when the value of the target function is the maximum value as the target value of the preset network parameter in the step S302 according to an example embodiment, as shown in Figure 7 The method comprises the following steps:

[0115] Step S701, calculating the gradient of the target function;

[0116] Step S702, determining the value of the preset network parameter when the value of the target function is the maximum value based on the gradient estimation algorithm.

[0117] The number of preset network parameters is determined according to the preset network model, and M preset network parameters are denoted as θ1, θ2... θ M , and they are initialized, and the historical trajectory set D is also initialized. The target function of the preset network model is represented as:

[0118]

[0119] wherein θ represents a preset network parameter, L(θ) represents a target function, represents a reward function, ρ π represents a predicted state-action distribution, ρ μ represents a historical state-action distribution, D MMD (ρ π , ρ μ ) represents a distance parameter, and α represents a weight.

[0120] The gradient of the target function is calculated as follows:

[0121]

[0122] wherein N represents a normal distribution, σ represents a standard deviation of the normal distribution, I represents a unit matrix, and ∈ represents a sampling value of the normal distribution;

[0123] After sampling ∈ for n times, the gradient estimation is performed as follows:

[0124]

[0125] wherein ∈ i is the i-th sampling value, and θ i = θ + σ ∈ i . The value of the target function corresponding to the value of the preset network parameter of each sampling is calculated, and the value of the preset network parameter corresponding to the maximum value of the target function is selected as the target value of the preset network parameter.

[0126] In an exemplary embodiment, the distance parameter between the predicted trajectory set and the historical trajectory set is determined by determining the distance parameter between each predicted state-action distribution of the predicted trajectory set and the historical trajectory set in step S103.

[0127] Since the exploration environment gives the same reward to all state-action distributions in the same trajectory, when determining the distance parameter between the predicted trajectory set and the historical trajectory set, the distance between each predicted state-action distribution in the predicted trajectory set and the historical trajectory set is calculated, i.e.:

[0128]

[0129] wherein x represents a predicted state-action distribution in the predicted trajectory set, represents a set of functions of a certain type on the state space, and f(x) represents a function set on the distribution space.

[0130] ​​In the illustrative embodiments of the present disclosure, a device for determining an exploration strategy of an agent is provided, the exploration strategy including an exploration strategy based on a preset network model. Figure 8 is a block diagram of a device for determining an exploration strategy of an agent according to an illustrative embodiment, as shown in Figure 8 , comprising:

[0131] The acquisition module 801 is configured to acquire a historical trajectory set of the agent, the historical trajectory set including a historical state-action distribution set;

[0132] The first determination module 802 is configured to determine a predicted trajectory set of the agent based on preset network parameters of the preset network model, the predicted trajectory set including a predicted state-action distribution set;

[0133] The second determination module 803 is configured to determine a distance parameter between the predicted trajectory set and the historical trajectory set;

[0134] The third determination module 804 is configured to determine a target value of the preset network parameters based on the distance parameter;

[0135] The fourth determination module 805 is configured to determine a target exploration strategy of the agent according to the target value of the preset network parameters.

[0136] In an illustrative embodiment, the third determination module 804 is further configured to:

[0137] determine a target function of the preset network model according to the distance parameter and a reward function of the preset network model; the target function and the reward function are both functions based on the preset network parameters;

[0138] determine the target value of the preset network parameters as a value of the preset network parameters when the target function takes a maximum value.

[0139] In an illustrative embodiment, the third determination module 804 is further configured to:

[0140] determine the target function by the following formula:

[0141]

[0142] wherein θ represents the preset network parameters, L(θ) represents the target function, represents the reward function, ρ π represents the predicted state-action distribution, ρ μ represents the historical state-action distribution, D MMD (ρ π , ρ μ) represents the distance parameter, and σ represents a weight.

[0143] In an example embodiment, the third determining module 804 is further configured to:

[0144] calculate a gradient of the target function;

[0145] determine the value of the preset network parameter when the value of the target function is maximum based on a gradient estimation algorithm.

[0146] In an example embodiment, the first determining module 802 is further configured to:

[0147] sample the preset network parameter multiple times to obtain multiple sample values of the preset network parameter;

[0148] input the multiple sample values of the preset network parameter into a preset network model to obtain the set of predicted trajectories.

[0149] In an example embodiment, the first determining module 802 is further configured to:

[0150] sample based on a normal distribution:

[0151]

[0152] wherein, represents the normal distribution, σ represents a standard deviation of the normal distribution, I represents a unit matrix, and σ 2 I represents a covariance matrix of the normal distribution, n represents a number of samplings, and ∈ i represents an i-th sample value;

[0153] determine the sample value of the preset network parameter by the following formula:

[0154] θ i = θ + ∈ i

[0155] wherein, θ i represents a sample value of the preset network parameter sampled for the i-th time, and θ represents the preset network parameter.

[0156] In an example embodiment, the second determining module 803 is further configured to:

[0157] determine a distance parameter between each of the predicted state-action distributions of the set of predicted trajectories and the set of historical trajectories.

[0158] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0159] Figure 9 is a block diagram of an intelligent entity 900 illustrated according to an exemplary embodiment.

[0160] Referring to Figure 9 , the intelligent entity 900 can include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0161] The processing component 902 usually controls overall operations of the intelligent entity 900, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 902 can include one or more processors 920 to execute instructions to complete all or part of steps of the above methods. In addition, the processing component 902 can include one or more modules to facilitate interaction between the processing component 902 and other components. For example, the processing component 902 can include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0162] The memory 904 is configured to store various types of data to support operations of the intelligent entity 900. Examples of these data include instructions for any application or method operating on the intelligent entity 900, contact data, phonebook data, messages, pictures, videos, and the like. The memory 904 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0163] The power supply component 906 supplies various components of the intelligent entity 900 with power. The power supply component 906 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the intelligent entity 900.

[0164] The multimedia component 908 includes a screen to provide an output interface between the smart object 900 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 908 includes a front camera and / or a back camera. The front and / or back camera can receive external multimedia data when the smart object 900 is in an operation mode, such as a shooting mode or a video mode. Each of the front and back camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0165] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC) to receive an external audio signal when the smart object 900 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 also includes a speaker to output audio signals.

[0166] The I / O interface 912 provides an interface between the processing component 902 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0167] The sensor component 914 includes one or more sensors to provide various state assessments for the smart object 900. For example, the sensor component 914 can detect an open / closed state of the smart object 900, relative positioning of components, such as a display and a keypad of the smart object 900, a change in position of the smart object 900 or a component of the smart object 900, presence or absence of user contact with the smart object 900, an orientation or acceleration / deceleration / g-force and a temperature change of the smart object 900. The sensor component 914 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 914 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 914 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0168] The communication component 916 is configured to facilitate wired or wireless communication between the agent 900 and other devices. The agent 900 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0169] In an exemplary embodiment, the agent 900 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components, for performing the above-described methods.

[0170] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 904 including instructions, is also provided, which can be executed by the processor 920 of the agent 900 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0171] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an apparatus, enables the apparatus to perform a method for determining an agent exploration policy, the method including any of the above-described methods.

[0172] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application indicated by the following claims.

[0173] It is to be understood that the application is not limited to the precise construction herein described and as shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is to be indicated by the appended claims, rather than the examples given above.

Claims

1. A method for determining an agent's exploration strategy, characterized in that, The exploration strategy comprises an exploration strategy based on a preset network model, and the determination method comprises: obtaining a historical trajectory set of the agent, the historical trajectory set being a historical state-action distribution set; determining a predicted trajectory set of the agent based on preset network parameters of the preset network model, the predicted trajectory set being a predicted state-action distribution set; determining a distance parameter between the predicted trajectory set and the historical trajectory set; determining a target value of the preset network parameters based on the distance parameter; determining a target exploration strategy of the agent according to the target value of the preset network parameters; The determination of the target value of the preset network parameters based on the distance parameter comprises: determining a target function of the preset network model according to the distance parameter and a reward function of the preset network model; the target function and the reward function are both functions based on the preset network parameters; calculating a gradient of the target function; determining the value of the preset network parameters when the value of the target function is maximum based on a gradient estimation algorithm. 2.The method of claim 1, wherein, The determination of the target function of the preset network model according to the distance parameter and the reward function of the preset network model comprises: determining the target function by the following formula: wherein, denotes the preset network parameter, denotes the target function, denotes the reward function, denotes the predicted state-action distribution, denotes the historical state-action distribution, denotes the distance parameter, denotes the weight.

3. The method of claim 1, wherein, The determination of the predicted trajectory set of the agent based on the preset network parameters of the preset network model comprises: sampling the preset network parameters multiple times to obtain multiple sampling values of the preset network parameters; inputting the multiple sampling values of the preset network parameters into the preset network model to obtain the predicted trajectory set.

4. The method of claim 3, wherein, The sampling of the preset network parameters multiple times to obtain the multiple sampling values of the preset network parameters comprises: sampling based on a normal distribution: in, This represents the normal distribution. This represents the standard deviation of the normal distribution. Represents the identity matrix. Let the covariance matrix of the normal distribution be represented. Indicates the number of samples. Indicates the first Sub-sample value; determining the multiple sampling values of the preset network parameters by the following formula: wherein, represents the first sampling value of the preset network parameter, represents the preset network parameter.

5. The method of claim 1, wherein, The determination of the distance parameter between the predicted trajectory set and the historical trajectory set comprises: determining a distance parameter between each predicted state-action distribution of the predicted trajectory set and the historical trajectory set.

6. A device for determining an agent's exploration strategy, characterized in that, The exploration strategy comprises an exploration strategy based on a preset network model, and the determination device comprises: an obtaining module configured to obtain a historical trajectory set of the agent, the historical trajectory set comprising a historical state-action distribution set; a first determining module configured to determine a predicted trajectory set of the agent based on preset network parameters of the preset network model, the predicted trajectory set comprising a predicted state-action distribution set; a second determining module configured to determine a distance parameter between the predicted trajectory set and the historical trajectory set; a third determining module configured to determine a target value of the preset network parameters based on the distance parameter; a fourth determining module configured to determine a target exploration strategy of the agent according to the target value of the preset network parameters; The third determining module is further configured to: determine a target function of the preset network model according to the distance parameter and a reward function of the preset network model; the target function and the reward function are both functions based on the preset network parameters; calculate a gradient of the target function; Based on a gradient estimation algorithm, a value of the preset network parameter when a value of the target function is a maximum value is determined.

7. An agent, characterized in that Comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium, comprising: When the instructions in the storage medium are executed by the processor of the intelligent agent, the intelligent agent is enabled to perform the method of any one of claims 1-5.

Images