Intelligent agent guiding method and device, electronic equipment and storage medium

Abstract

The invention relates to an agent guiding method and device, electronic equipment and a storage medium, and the method comprises the steps: adding Gaussian noise to each original state in a target data set, so as to obtain a disturbed state; at least one constraint condition is utilized to constrain the initial training strategy to obtain an actual training strategy, and transition distribution caused by the actual training strategy on the disturbed state is aligned with value-based state transition distribution in the original state; and starting from the disturbed state, guiding the intelligent agent to transfer to a state in high-value distribution in value-based state transfer distribution based on an actual training strategy. Therefore, the technical problems that in the related technology, the calculation efficiency is low, the simplicity of the algorithm is poor, and the strategy quality is easily affected under the condition that the data set contains a large number of suboptimal states are solved.

Description

Technical Field

[0001] The present application relates to the technical field of offline reinforcement learning, and in particular to a method, device, electronic device and storage medium for guiding an intelligent agent. Background Art

[0002] Deep reinforcement learning shows potential in solving sequential decision-making problems and has attracted increasing attention in real-world applications. However, deploying reinforcement learning algorithms in a wide range of scenarios still faces persistent challenges such as risk-sensitive exploration and time-consuming episodic data collection.

[0003] Offline reinforcement learning is seen as a solution to these challenges. Offline reinforcement learning aims to learn policies from a fixed dataset without further interaction. It is able to leverage existing large-scale datasets to achieve safe and efficient learning. In offline reinforcement learning research, there is an out-of-distribution state problem that impairs performance. The out-of-distribution state problem refers to the situation where the agent encounters states outside the offline dataset (out-of-distribution states) during the policy deployment phase (i.e., the test phase). The emergence of out-of-distribution states may arise from out-of-distribution actions, random environment dynamics, and perturbations in the real world. Since typical offline reinforcement learning algorithms do not involve policy training in out-of-distribution states, once the agent enters the out-of-distribution state during the test phase, it often exhibits uncontrolled behavior. This situation may further aggravate the deviation of its state from the offline dataset, resulting in a severe degradation in performance.

[0004] In terms of alleviating the out-of-distribution state problem, related technologies attempt to correct the agent from the out-of-distribution state back to the in-distribution state through training strategies. Technically, one attempt builds a dynamics model and a state transition model and aligns them to guide the agent back to the in-distribution area, while another attempt uses an inverse dynamics model for policy constraints.

[0005] However, when dealing with out-of-distribution states and out-of-distribution actions respectively, related technologies require additional out-of-distribution action suppression components and complex distribution modeling, which have low computational efficiency and poor algorithm simplicity. Especially when the dataset contains a large number of suboptimal states, correcting the agent from out-of-distribution states back to all in-distribution states impartially can easily lead to problems such as strategy solidification and overfitting that affect strategy quality, and this needs to be improved. Summary of the invention

[0006] The present application provides an intelligent body guidance method, device, electronic device and storage medium to solve the technical problems in the related technology, such as low computing efficiency and poor algorithm simplicity, which are easy to affect the quality of strategy when the data set contains a large number of suboptimal states.

[0007] The first aspect of the present application provides a method for guiding an intelligent agent, comprising the following steps: adding Gaussian noise to each original state in a target data set to obtain a perturbed state; using at least one constraint condition to constrain an initial training strategy to obtain an actual training strategy, wherein the transfer distribution induced by the actual training strategy on the perturbed state is aligned with the value-based state transfer distribution in the original state; starting from the perturbed state, guiding the intelligent agent to transfer to a high-value distribution state in the value-based state transfer distribution based on the actual training strategy.

[0008] Optionally, in one embodiment of the present application, before using at least one constraint condition to constrain the initial training strategy, it also includes: constructing a value-based state transition distribution optimization problem based on the target data set; and calculating an analytical solution to the expected state transition distribution of the state transition distribution optimization problem.

[0009] Optionally, in one embodiment of the present application, the expression of the state transition distribution optimization problem is:

[0010]

[0011] Among them, α is a hyperparameter for balancing the two terms, is the target data set, s is the state in the target data set, V is the value function, D KL is the KL divergence, N(·|s) is the state transition distribution in the target data set, N * (·|s) is the value-based state transition distribution.

[0012] Optionally, in one embodiment of the present application, the use of at least one constraint condition to constrain the initial training strategy to obtain the actual training strategy includes: using KL divergence to constrain the initial training strategy; using the analytical solution and the empirical normalization term to optimize the initial training strategy, and converting the optimized initial training strategy into a Jensen inequality; solving the Jensen inequality to obtain the actual training strategy that satisfies a preset stability condition.

[0013] Optionally, in one embodiment of the present application, the expression of the actual training strategy is:

[0014]

[0015] Where exp(αV(s)) is the normalization term of experience, π is the initial training strategy, s′ is the state in the high value distribution, is the out-of-distribution state after disturbance, is Gaussian noise, σ 2 is the variance of the Gaussian noise, and M is the state transition dynamics function.

[0016] Optionally, in one embodiment of the present application, the use of at least one constraint condition to constrain the initial training strategy to obtain the actual training strategy includes: using the target data set to train the dynamics model to obtain a trained deterministic dynamics model; obtaining a corresponding policy network based on the target data set and the policy evaluation loss function; and obtaining the actual training strategy based on the deterministic dynamics model, the policy network and the initial training strategy.

[0017] Optionally, in one embodiment of the present application, the actual training strategy expression is:

[0018]

[0019] Among them, π φ is the policy network, M ω is the deterministic kinetic model, V θ (s) is a function of the state policy network without gradient backpropagation.

[0020] The second aspect of the present application provides a guiding device for an intelligent agent, comprising: a perturbation module, used to add Gaussian noise to each original state in a target data set to obtain a perturbed state; a constraint module, used to constrain an initial training strategy using at least one constraint condition to obtain an actual training strategy, wherein the transfer distribution induced by the actual training strategy on the perturbed state is aligned with the value-based state transfer distribution in the original state; and a guiding module, used to guide the intelligent agent to transfer to a high-value distribution state in the value-based state transfer distribution based on the actual training strategy, starting from the perturbed state.

[0021] Optionally, in one embodiment of the present application, it also includes: a construction module for constructing a value-based state transition distribution optimization problem based on the target data set; and a calculation module for calculating an analytical solution to the expected state transition distribution of the state transition distribution optimization problem.

[0022] Optionally, in one embodiment of the present application, the expression of the state transition distribution optimization problem is:

[0023]

[0024] Among them, α is a hyperparameter for balancing the two terms, is the target data set, s is the state in the target data set, V is the value function, D KL is the KL divergence, N(·|s) is the state transition distribution in the target data set, N * (·|s) is the value-based state transition distribution.

[0025] Optionally, in one embodiment of the present application, the constraint module includes: a constraint unit, used to utilize KL divergence to constrain the initial training strategy; an optimization unit, used to optimize the initial training strategy using the analytical solution and the empirical normalization term, and convert the optimized initial training strategy into a Jensen inequality; a first calculation unit, used to solve the Jensen inequality to obtain the actual training strategy that satisfies a preset stability condition.

[0026] Optionally, in one embodiment of the present application, the expression of the actual training strategy is:

[0027]

[0028] Where exp(αV(s)) is the normalization term of experience, π is the initial training strategy, s′ is the state in the high value distribution, is the out-of-distribution state after disturbance, is Gaussian noise, σ 2 is the variance of the Gaussian noise, and M is the state transition dynamics function.

[0029] Optionally, in one embodiment of the present application, the constraint module includes: a training unit, used to train the dynamic model using the target data set to obtain a trained deterministic dynamic model; a second computing unit, used to obtain a corresponding policy network based on the target data set and the policy evaluation loss function; and a third computing unit, used to obtain the actual training strategy based on the deterministic dynamic model, the policy network and the initial training strategy.

[0030] Optionally, in one embodiment of the present application, the actual training strategy expression is:

[0031]

[0032] Among them, π φ is the policy network, M ω is the deterministic kinetic model, V θ (s) is a function of the state policy network without gradient backpropagation.

[0033] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the guiding method of the intelligent body as described in the above embodiment.

[0034] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the agent booting method as described in the above embodiment.

[0035] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned agent guidance method.

[0036] The embodiment of the present application can perturb each original state in the target data set to obtain the perturbed state, and constrain the training strategy to guide the agent to start from the perturbed state, and transfer the constraint-based training strategy to the high-value in-distribution state in the value-based state transfer distribution, formulate a simple and effective strategy regularizer for offline reinforcement learning, unify the out-of-distribution state correction and out-of-distribution action suppression, realize the value-based out-of-distribution state correction, and correct the agent from the out-of-distribution state to the high-value in-distribution state. In this way, the technical problems in the related technology that the computational efficiency is low and the simplicity of the algorithm is poor, and the quality of the strategy is easily affected when the data set contains a large number of suboptimal states, are solved.

[0037] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 A flowchart of a guiding method of an intelligent agent provided according to an embodiment of the present application;

[0040] Figure 2 is a flow chart of a guiding method of an intelligent agent according to an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a guiding device for an intelligent body provided according to an embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The following describes the guiding method, device, electronic device and storage medium of the intelligent agent of the embodiment of the present application with reference to the accompanying drawings. In view of the technical problems that the computational efficiency is low and the simplicity of the algorithm is poor in the related technologies mentioned in the above background technology, and the quality of the strategy is easily affected when the data set contains a large number of suboptimal states, the present application provides a guiding method for an intelligent agent, in which each original state in the target data set can be disturbed to obtain the state after the disturbance, and the training strategy is constrained to guide the intelligent agent to start from the state after the disturbance, and transfer the constraint-based training strategy to the high-value distribution state in the value-based state transfer distribution, and formulate a simple and effective strategy regularizer for offline reinforcement learning, unify the out-of-distribution state correction and out-of-distribution action suppression, and realize the value-based out-of-distribution state correction, and correct the intelligent agent from the out-of-distribution state to the high-value distribution state. In this way, the technical problems that the computational efficiency is low and the simplicity of the algorithm is poor in the related technologies, and the quality of the strategy is easily affected when the data set contains a large number of suboptimal states, are solved.

[0045] Specifically, Figure 1 A flowchart of a method for guiding an intelligent agent provided in an embodiment of the present application.

[0046] like Figure 1 As shown, the agent guidance method includes the following steps:

[0047] In step S101, Gaussian noise is added to each original state in the target data set to obtain a disturbed state.

[0048] In the actual execution process, in order to generate a large number of out-of-distribution states, the embodiment of the present application can Gaussian noise is added to each state s in Among them, σ 2 is the variance of the Gaussian distribution, and the out-of-distribution state after the disturbance is obtained It is worth noting that the data set used for reinforcement learning training remains unchanged, and the state is perturbed in the embodiment of the present application only to construct a regularization term.

[0049] In step S102, at least one constraint condition is used to constrain the initial training strategy to obtain an actual training strategy, wherein the transition distribution induced by the actual training strategy in the perturbed state is aligned with the value-based state transition distribution in the original state.

[0050] As a possible implementation method, the embodiment of the present application can use the KL divergence constraint training strategy π to make π in the out-of-distribution state after the disturbance The transition distribution induced by is aligned with the value-based state transition distribution under the original state s.

[0051] Optionally, in one embodiment of the present application, before using at least one constraint condition to constrain the initial training strategy, the method further includes: constructing a value-based state transition distribution optimization problem based on the target data set; and calculating an analytical solution to the expected state transition distribution of the state transition distribution optimization problem, wherein the expression of the state transition distribution optimization problem is:

[0052]

[0053] Among them, α is a hyperparameter for balancing the two terms, is the target data set, s is the state in the target data set, V is the value function, D KL is the KL divergence, N(·|s) is the state transition distribution in the target data set, and N * (·|s) is the value-based state transition distribution.

[0054] Before using at least one constraint condition to constrain the initial training strategy, the embodiments of the present application may consider the following issues:

[0055]

[0056] Among them, α is a hyperparameter that balances the two terms, is a data set, s is the state in the data set, V is the value function, and D KL is the KL divergence, and N(·|s) is the state transition distribution in the dataset.

[0057] The above optimization problem has an analytical solution:

[0058]

[0059] Where Z(s) = ∑ s′ exp(αV(s′))N(s′|s) is the normalization factor. As α increases, N * (·|s) will be more inclined to the optimal state within the support set of N(·|s).

[0060] Optionally, in one embodiment of the present application, at least one constraint condition is used to constrain the initial training strategy to obtain the actual training strategy, including: using KL divergence to constrain the initial training strategy; optimizing the initial training strategy using an analytical solution and an empirical normalization term, and converting the optimized initial training strategy into a Jhinsen inequality; solving the Jhinsen inequality to obtain an actual training strategy that satisfies a preset stability condition, wherein the expression of the actual training strategy is:

[0061]

[0062] Among them, exp(αV(s)) is the normalization term of experience, π is the initial training strategy, s′ is the state in the high value distribution, is the out-of-distribution state after disturbance, is Gaussian noise, σ 2 is the variance of Gaussian noise, and M is the state transition dynamics function.

[0063] In some embodiments, the embodiments of the present application may constrain the initial training strategy π by minimizing the following regularization term:

[0064]

[0065] Where M is the state transition dynamics function.

[0066] By replacing N in formula (2) * Substituting the analytical solution of into KL divergence, we get:

[0067]

[0068] Where N is the state transition distribution in the data set, so Equivalent to Therefore, minimizing (3) is equivalent to maximizing the following regularization term:

[0069]

[0070] Therefore, R(π) effectively eliminates the need for pre-trained multimodal state transition models (N or N * ) and can be directly sampled from the dataset for optimization. However, the normalization factor Z(s) in R(π) may be difficult to calculate.

[0071] It should be noted that the regularization term R(π) is derived from the minimization of the KL divergence in equation (3). Each state s in minimizes the KL divergence, and Z(s) only affects the relative weights of different states s, so it is not so important to accurately recover the correct relative weights by calculating Z(s), especially Z(s) is difficult to estimate empirically and may bring more instability.

[0072] Therefore, for the stability of calculation, in the embodiment of the present application, Z(s) can be replaced by the empirical normalization term exp(αV(s)) in R(π):

[0073]

[0074] The embodiment of the present application can shift the focus to the optimization of R1(π). According to Jensen's inequality, the expectation of strategy π can be moved outside the logarithmic function:

[0075]

[0076] When π is a deterministic strategy, the inequality becomes an equality. For convenience, the embodiment of the present application can use the reparameterization technique to maximize the lower bound of the inequality in formula (7). However, in order to ensure the equality in formula (7), the embodiment of the present application can choose to train a deterministic strategy π. In this case, the embodiment of the present application can calculate the gradient of π by automatic differentiation and directly maximize R1(π).

[0077] Optionally, in one embodiment of the present application, at least one constraint condition is used to constrain the initial training strategy to obtain the actual training strategy, including: using the target data set to train the dynamics model to obtain a trained deterministic dynamics model; obtaining a corresponding policy network based on the target data set and the policy evaluation loss function; and obtaining the actual training strategy based on the deterministic dynamics model, the policy network and the initial training strategy. The actual training strategy expression is:

[0078]

[0079] Among them, π φ is the policy network, M ω is a deterministic kinetic model, V θ (s) is a function of the state policy network without gradient backpropagation.

[0080] In other embodiments, the present application can combine the dynamic model pre-training and the strategy training of the Actor-Critic framework to perform training strategy constraints, and use a neural network to parameterize the dynamic network M. ω , policy network π φ ,Q NetworkQ θ , and the target Q network Q θ′ .

[0081] Among them, dynamics model pre-training:

[0082] The present application embodiment can adopt the deterministic kinetic model M ω , learn by minimizing the mean square error. Training M ω The loss function is:

[0083]

[0084] in, is the target data set, s, a, s′ are the state, action and next state in the target data set respectively.

[0085] Strategy training of Actor-Critic framework:

[0086] The embodiments of the present application can alternately optimize the following objective functions to alternately perform strategy evaluation and strategy improvement.

[0087] Strategy Evaluation:

[0088] In the strategy evaluation, the embodiment of the present application can minimize the standard strategy evaluation loss function to learn the Q function Q θ .

[0089]

[0090] Among them, Q θ is the Q function network, R is the reward function, γ is the discount factor, π φ is the policy network, Q θ′ is a target Q function network updated by Pavel averaging.

[0091] Strategy Improvements:

[0092] When using a deterministic dynamic model, the embodiment of the present application can replace the log-likelihood in R1(π) with the mean square error. This is a common practice of converting the maximum likelihood estimation problem into a regression problem when dealing with a Gaussian distribution with a fixed variance.

[0093] The embodiment of the present application may also adopt a deterministic strategy model π φ Therefore, the embodiment of the present application can obtain the following policy optimization regularization term:

[0094]

[0095] in, represents π without gradient backpropagation φ Using a deterministic strategy also simplifies the training process because there is no need to learn V separately θFunction. R2(π φ ) combined with the standard policy improvement goals, the embodiments of the present application can update the policy by maximizing the following goals:

[0096]

[0097] Among them, λ∈(0,1) is a hyperparameter used to balance these two items. In addition, the embodiment of the present application can also adjust the Q in the first item in each small batch. θ Normalization is performed to maintain a balanced scale across tasks.

[0098] In step S103, starting from the perturbed state, the agent is guided to transfer to a high-value in-distribution state in the value-based state transition distribution based on the actual training strategy.

[0099] As a possible implementation method, the embodiment of the present application anticipates the following value-based out-of-distribution state correction scenario: the agent is in an out-of-distribution state after the disturbance Start by following the initial training strategy π and transfer to distribution N * (·|s) is a high-value distribution state s′. To this end, the embodiment of the present application constrains the initial training strategy π by KL divergence, so that π is in the perturbation state The transition distribution induced by is aligned with the value-based state transition distribution under the original state s.

[0100] Combination Figure 2 As shown, the working principle of the guiding method of the intelligent body of the embodiment of the present application is described in detail with an embodiment.

[0101] Step S201: Consider an optimization problem based on a target data set and calculate an analytical solution to the optimization problem.

[0102] For the state distribution within the distribution to which the agent is corrected, the embodiment of the present application expects it to be a value-based state transition distribution N * (·|s), which is within the support set of the state transition distribution N(·|s) of the data set, but is biased towards the next state s′ with high value. In order to ensure stability and avoid modeling complex distributions, the embodiment of the present application seeks its soft optimal version. To this end, the following issues can be considered:

[0103]

[0104] Among them, α is a hyperparameter that balances the two terms, is a data set, s is the state in the data set, V is the value function, and D KL is the KL divergence, and N(·|s) is the state transition distribution in the dataset.

[0105] The above optimization problem has an analytical solution:

[0106]

[0107] Where Z(s) = ∑ s′ exp(αV(s′))N(s′|s) is the normalization factor. As α increases, N * (·|s) will be more inclined to the optimal state within the support set of N(·|s).

[0108] Step S202: perturb the original state and constrain the training strategy.

[0109] In order to generate a large number of out-of-distribution states, the embodiment of the present application can add Gaussian noise to each state s in the target data set D. Among them, σ 2 is the variance of the Gaussian distribution, and the state after disturbance is obtained It is worth noting that the data set used for reinforcement learning training remains unchanged, and the state is perturbed in the embodiment of the present application only to construct a regularization term.

[0110] The present application embodiment anticipates the following value-based out-of-distribution state correction scenario: the agent is corrected from the out-of-distribution state after the disturbance Start by following the initial training strategy π and transfer to distribution N * (·|s) in the high-value distribution state s′. To this end, the embodiment of the present application constrains the initial training strategy π by KL divergence, so that π is in the distribution state after the disturbance The transition distribution induced by is aligned with the value-based state transition distribution under the original state s.

[0111] The embodiment of the present application can constrain the initial training strategy π by minimizing the following regularization term:

[0112]

[0113] Where M is the state transition dynamics function.

[0114] By replacing N in formula (2) * Substituting the analytical solution of into KL divergence, we get:

[0115]

[0116] Where N is the state transition distribution in the data set, so s′~N(·|s) is equivalent to Therefore, minimizing (3) is equivalent to maximizing the following regularization term:

[0117]

[0118] Therefore, R(π) effectively eliminates the need for pre-trained multimodal state transition models (N or N * ) and can be directly sampled from the dataset for optimization. However, the normalization factor Z(s) in R(π) may be difficult to calculate.

[0119] It should be noted that the regularization term R(π) is derived from the minimization of the KL divergence in equation (3). Each state s in minimizes the KL divergence, and Z(s) only affects the relative weights of different states s, so it is not so important to accurately recover the correct relative weights by calculating Z(s), especially Z(s) is difficult to estimate empirically and may bring more instability.

[0120] Therefore, for the stability of calculation, in the embodiment of the present application, Z(s) can be replaced by the empirical normalization term exp(αV(s)) in R(π):

[0121]

[0122] The embodiment of the present application can shift the focus to the optimization of R1(π). According to Jensen's inequality, the expectation of strategy π can be moved outside the logarithmic function:

[0123]

[0124] When π is a deterministic strategy, the inequality becomes an equality. For convenience, the embodiment of the present application can use the reparameterization technique to maximize the lower bound of the inequality in formula (7). However, in order to ensure the equality in formula (7), the embodiment of the present application can choose to train a deterministic strategy π. In this case, the embodiment of the present application can calculate the gradient of π by automatic differentiation and directly maximize R1(π).

[0125] Furthermore, in the actual implementation process, the embodiment of the present application can also combine the dynamic model pre-training and the strategy training of the Actor-Critic framework to perform training strategy constraints, and use the neural network parameterized dynamic network M ω , policy network π φ ,Q NetworkQ θ , and the target Q network Q θ′ .

[0126] Among them, dynamics model pre-training:

[0127] The present application embodiment can adopt the deterministic kinetic model M ω , learn by minimizing the mean square error. Training M ωThe loss function is:

[0128]

[0129] in, is the target data set, s, a, s′ are the state, action and next state in the target data set respectively.

[0130] Strategy training of Actor-Critic framework:

[0131] The embodiments of the present application can alternately optimize the following objective functions to alternately perform strategy evaluation and strategy improvement.

[0132] Strategy Evaluation:

[0133] In the strategy evaluation, the embodiment of the present application can minimize the standard strategy evaluation loss function to learn the Q function Q θ .

[0134]

[0135] Among them, Q θ is the Q function network, R is the reward function, γ is the discount factor, π φ is the policy network, Q θ′ is a target Q function network updated by Pavel averaging.

[0136] Strategy Improvements:

[0137] When using a deterministic dynamic model, the embodiment of the present application can replace the log-likelihood in R1(π) with the mean square error. This is a common practice of converting the maximum likelihood estimation problem into a regression problem when dealing with a Gaussian distribution with a fixed variance.

[0138] The embodiment of the present application may also adopt a deterministic strategy model π φ Therefore, the embodiment of the present application can obtain the following policy optimization regularization term:

[0139]

[0140] in, represents π without gradient backpropagation φ Using a deterministic strategy also simplifies the training process because there is no need to learn V separately θ Function. R2(π φ ) combined with the standard policy improvement goals, the embodiments of the present application can update the policy by maximizing the following goals:

[0141]

[0142] Among them, λ∈(0,1) is a hyperparameter used to balance these two items. In addition, the embodiment of the present application can also adjust the Q in the first item in each small batch. θ Normalization is performed to maintain a balanced scale across tasks.

[0143] Step S203: guiding the agent based on the constrained training strategy.

[0144] In summary, it is proved in theory and experiment that the embodiment of the present application can suppress out-of-distribution actions while correcting out-of-distribution states, unifying out-of-distribution state correction and out-of-distribution action suppression in offline reinforcement learning. The embodiment of the present application also implements value-based out-of-distribution state correction, avoiding the need to model complex distributions, and significantly improving performance compared to traditional out-of-distribution state correction methods.

[0145] The embodiment of the present application was applied in an offline reinforcement learning benchmark test, achieving excellent performance while maintaining consistent hyperparameters without additional tuning. In addition, thanks to its ability to correct out-of-distribution states, the embodiment of the present application exhibits greater robustness in dealing with environmental disturbances.

[0146] According to the guiding method of the intelligent agent proposed in the embodiment of the present application, each original state in the target data set can be disturbed to obtain the state after the disturbance, and the training strategy can be constrained to guide the intelligent agent to start from the state after the disturbance, and transfer the constraint-based training strategy to the high-value distribution state in the value-based state transfer distribution, and formulate a simple and effective strategy regularizer for offline reinforcement learning, unify the out-of-distribution state correction and out-of-distribution action suppression, realize the value-based out-of-distribution state correction, and correct the intelligent agent from the out-of-distribution state to the high-value distribution state. In this way, the technical problems in the related technology that the computational efficiency is low, the simplicity of the algorithm is poor, and the quality of the strategy is easily affected when the data set contains a large number of suboptimal states are solved.

[0147] Next, the guiding device of the intelligent body proposed in the embodiment of the present application is described with reference to the accompanying drawings.

[0148] Figure 3 It is a block diagram of a guiding device of an intelligent body according to an embodiment of the present application.

[0149] like Figure 3 As shown, the guiding device 10 of the intelligent agent includes: a disturbance module 100, a constraint module 200 and a guiding module 300.

[0150] Specifically, the perturbation module 100 is used to add Gaussian noise to each original state in the target data set to obtain a perturbed state.

[0151] The constraint module 200 is used to constrain the initial training strategy using at least one constraint condition to obtain an actual training strategy, wherein the transition distribution induced by the actual training strategy on the perturbed state is aligned with the value-based state transition distribution in the original state.

[0152] The guiding module 300 is used to guide the intelligent agent to transfer to a high-value distribution state in the value-based state transition distribution based on the actual training strategy starting from the perturbed state.

[0153] Optionally, in one embodiment of the present application, the guiding device 10 of the intelligent agent further includes: a construction module and a calculation module.

[0154] Among them, the construction module is used to construct a value-based state transfer distribution optimization problem based on the target data set.

[0155] The calculation module is used to calculate the analytical solution of the expected state transfer distribution of the state transfer distribution optimization problem.

[0156] Optionally, in one embodiment of the present application, the expression of the state transition distribution optimization problem is:

[0157]

[0158] Among them, α is a hyperparameter for balancing the two terms, is the target data set, s is the state in the target data set, V is the value function, D KL is the KL divergence, N(·|s) is the state transition distribution in the target data set, and N * (·|s) is the value-based state transition distribution.

[0159] Optionally, in one embodiment of the present application, the constraint module 200 includes: a constraint unit, an optimization unit and a first calculation unit.

[0160] Among them, the constraint unit is used to use KL divergence to constrain the initial training strategy.

[0161] The optimization unit is used to optimize the initial training strategy using the analytical solution and the empirical normalization term, and convert the optimized initial training strategy into the Qinsheng inequality.

[0162] The first computing unit is used to solve the Jhinsen inequality to obtain an actual training strategy that satisfies a preset stability condition.

[0163] Optionally, in one embodiment of the present application, the expression of the actual training strategy is:

[0164]

[0165] Among them, exp(αV(s)) is the normalization term of experience, π is the initial training strategy, s′ is the state in the high value distribution, is the out-of-distribution state after disturbance, is Gaussian noise, σ 2 is the variance of Gaussian noise, and M is the state transition dynamics function.

[0166] Optionally, in one embodiment of the present application, the constraint module 200 includes: a training unit, a second computing unit and a third computing unit.

[0167] The training unit is used to train the dynamics model using the target data set to obtain a trained deterministic dynamics model.

[0168] The second computing unit is used to obtain a corresponding policy network based on the target data set and the policy evaluation loss function.

[0169] The third computing unit is used to obtain an actual training strategy based on the deterministic dynamics model, the strategy network and the initial training strategy.

[0170] Optionally, in one embodiment of the present application, the actual training strategy expression is:

[0171]

[0172] Among them, π φ is the policy network, M ω is a deterministic kinetic model, V θ (s) is a function of the state policy network without gradient backpropagation.

[0173] It should be noted that the above explanation of the embodiment of the guiding method of the intelligent body is also applicable to the guiding device of the intelligent body of this embodiment, and will not be repeated here.

[0174] According to the guiding device of the intelligent agent proposed in the embodiment of the present application, each original state in the target data set can be disturbed to obtain the state after the disturbance, and the training strategy can be constrained to guide the intelligent agent to start from the state after the disturbance, and transfer the constraint-based training strategy to the high-value distribution state in the value-based state transfer distribution, so as to formulate a simple and effective strategy regularizer for offline reinforcement learning, unify the out-of-distribution state correction and out-of-distribution action suppression, realize the value-based out-of-distribution state correction, and correct the intelligent agent from the out-of-distribution state to the high-value distribution state. In this way, the technical problems in the related technology that the computational efficiency is low, the simplicity of the algorithm is poor, and the quality of the strategy is easily affected when the data set contains a large number of suboptimal states are solved.

[0175] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0176] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0177] When the processor 402 executes the program, the guiding method of the intelligent agent provided in the above embodiment is implemented.

[0178] Furthermore, the electronic device further comprises:

[0179] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0180] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0181] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0182] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0183] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0184] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0185] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned agent booting method is implemented.

[0186] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the agent guiding method provided by an embodiment of the present invention.

[0187] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0188] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0189] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0190] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0191] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0192] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0193] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0194] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for guiding an intelligent agent, characterized in that: The following steps are involved: Add Gaussian noise to each original state in the target data set to obtain the perturbed state; Constraining the initial training policy using at least one constraint condition to obtain an actual training policy, wherein the transition distribution induced by the actual training policy on the perturbed state is aligned with the value-based state transition distribution in the original state; Starting from the perturbed state, the agent is guided to transfer to a high-value in-distribution state in the value-based state transition distribution based on the actual training policy.

2. The method according to claim 1, characterized in that Before constraining the initial training strategy using at least one constraint condition, the method further includes: Based on the target data set, construct a value-based state transfer distribution optimization problem; An analytical solution to the expected state transition distribution of the state transition distribution optimization problem is calculated.

3. The method according to claim 2, characterized in that The expression of the state transfer distribution optimization problem is: Among them, α is a hyperparameter for balancing the two terms, is the target data set, s is the state in the target data set, V is the value function, D KL is the KL divergence, N(·|s) is the state transition distribution in the target data set, N * (·|s) is the value-based state transition distribution.

4. The method according to claim 2, characterized in that: The using at least one constraint condition to constrain the initial training strategy to obtain the actual training strategy includes: Using KL divergence to constrain the initial training strategy; Optimizing the initial training strategy using the analytical solution and the empirical normalization term, and converting the optimized initial training strategy into a Qinsheng inequality; Solving the Jhinsen inequality obtains the actual training strategy that satisfies the preset stability condition.

5. The method according to claim 4, characterized in that The expression of the actual training strategy is: Where exp(αV(s)) is the normalization term of experience, π is the initial training strategy, s′ is the state in the high value distribution, is the out-of-distribution state after disturbance, is Gaussian noise, σ 2 is the variance of the Gaussian noise, and M is the state transition dynamics function.

6. The method according to claim 1, characterized in that The using at least one constraint condition to constrain the initial training strategy to obtain the actual training strategy includes: Using the target data set to train a kinetic model to obtain a trained deterministic kinetic model; Obtaining a corresponding policy network based on the target data set and the policy evaluation loss function; The actual training strategy is obtained based on the deterministic dynamics model, the strategy network and the initial training strategy.

7. The method according to claim 6, characterized in that The actual training strategy expression is: Among them, π φ is the policy network, M ω is the deterministic kinetic model, V θ (s) is a function of the state policy network without gradient backpropagation.

8. A guiding device for an intelligent agent, characterized in that: include: The perturbation module is used to add Gaussian noise to each original state in the target data set to obtain the perturbed state; A constraint module, configured to constrain an initial training strategy using at least one constraint condition to obtain an actual training strategy, wherein a transition distribution induced by the actual training strategy on the perturbed state is aligned with a value-based state transition distribution in the original state; A guiding module is used to guide the agent to transfer to a high-value distribution state in the value-based state transition distribution based on the actual training strategy starting from the perturbed state.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the guiding method of an intelligent agent as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the guiding method of an intelligent agent as described in any one of claims 1 to 7.

Images